How face detection works, and what really hides a face

Cameras have drawn boxes around faces since the mid-2000s, and photo apps now blur strangers automatically. Both rest on face detection — finding that a face is there — which is a different, easier and less controversial problem than recognition. This guide explains how detectors work, why they miss what they miss, and what kind of blur genuinely protects someone; the face blur tool runs a detector on your device.

Detection versus recognition

Detection answers "where are the faces?" and returns boxes. Recognition answers "whose face is this?" and matches against a database of known people. Detection is what a camera uses to focus, a photo app uses to blur, and an anonymisation tool needs; it stores nothing about identity. Recognition is the technology behind unlocking a phone and the one regulated in many jurisdictions. Everything on this site is detection only.

How detectors find faces

The 2001 Viola–Jones detector scanned an image with a cascade of simple rectangular contrast tests — eyes darker than cheeks, the bridge of the nose lighter than the eyes — and was fast enough for cameras, at the cost of only finding frontal faces. Modern detectors are small convolutional neural networks trained on hundreds of thousands of labelled faces: they look at the image at several scales, propose boxes, score each for face-likeness, and merge overlapping proposals (non-maximum suppression). UltraFace, the model used here, is a 1.2 MB network that takes a 320×240 input and returns boxes and confidences in a few hundred milliseconds on a laptop — small enough to download once and run in a browser. Larger models (RetinaFace, MTCNN) also return landmarks — eyes, nose, mouth — and handle harder cases at a higher cost.

Why detectors miss faces

• Size. At 320 pixels across, a face in a crowd photo is a few pixels wide — below the detector's training range. Cropping to a region and re-running finds them.
• Angle. Profiles and faces looking down score low; training data is mostly frontal and three-quarter.
• Occlusion. Masks, hands, sunglasses and hair remove the features the network keys on.
• Light and blur. Backlit or motion-blurred faces lose the contrast patterns.
• Bias. Detectors trained on unbalanced datasets perform worse on some skin tones and ages; this is documented and a reason to check results by eye rather than trust the count.

Any anonymisation tool therefore needs a manual fallback — drag a box over the one it missed — and a human check.

Blur, pixelate or box: what actually anonymises

A light Gaussian blur is reversible in principle: blur is a linear operation, and deconvolution plus a guess at the kernel recovers a surprising amount; researchers have also re-identified blurred and pixelated faces by matching against candidate photos processed the same way. What works is destroying information, not smearing it: pixelation at large blocks (so a face is a handful of squares), a solid box, or a blur so heavy the result is a featureless blob — and, crucially, doing it to the original before anything is published, because a lightly blurred version already online is the version that gets attacked. Context leaks too: clothing, location and the other people in the frame identify someone without the face; What your photos reveal about you covers the GPS data in the file, and the EXIF remover strips it.

Doing it without uploading the photo

The photos people anonymise are the ones they least want uploaded — children, patients, bystanders, a workplace — and until recently detection required a server. A 1 MB detector running on-device through a WebAssembly runtime changes that: the photo stays in the browser, detection takes a moment, and the blur or box is applied locally. The same on-device pattern powers the background remover and How background removal works — and why hair is hard explains the segmentation model behind it.