This paper presents a system, where clocked comparators consuming only CV2 energy directly derive classification decisions from analog sensor signals, thereby replacing instrumentation amplifiers, ADCs, and digital MACs, as typically required. A machine-learning algorithm for training the classifier is presented, which enables circuit non-idealities as well as severe energy/area scaling in analog circuits to be overcome. Furthermore, a noise model of the system is presented and experimentally verified, providing a means to predict and optimize classification error probability in a given application. The noise model shows that superior noise efficiency is achieved by the comparator-based system compared with a system based on linear low-noise amplifiers. A prototype in 130-nm CMOS performs image recognition of handwritten numerical digits, by taking raw analog pixels as the inputs. Due to pin limitations on the chip, the images with 28 × 28 = 784 pixels are resized and downsampled to give 47 pixel features, yielding an accuracy of 90% for an ideal ten-way classification system (MATLAB simulated). The prototype comparator-based system achieves equivalent performance with a total energy of 543 pJ per ten-way classification at a rate up to 1.3 M images per second, representing 33× lower energy than an ADC/digital-MAC system.