How background removal works — and why hair is hard

"Remove background" is the most-used button in photo editing, and the least understood. It looks like the software knows what a person is; what it actually produces is a mask — a grey image saying how much of each pixel belongs to the subject — and everything else follows from that. This guide explains the mask, the network that draws it, the file format that stores transparency, and the failure modes you'll meet, so that the background remover and the background changer stop being magic and start being predictable.

It's a mask, not magic

A cutout is the original photo plus a second, single-channel image the same size: white where the subject is, black where the background is, grey along the edge. Multiply the photo's opacity by that mask and the background disappears. Every background remover — Photoshop, a phone app, this site — ends at the same place; they differ only in how the mask is drawn. Twenty years ago you drew it yourself with a pen tool or a green screen. Now a network draws it, and your job is judging the edge.

How a neural network finds the subject

The task is called salient object segmentation: given a photo, label the pixels of the thing the photo is about. Networks such as U²-Net are trained on tens of thousands of photos paired with hand-drawn masks; they learn that a person-shaped region of skin tones surrounded by different texture is probably the subject, and produce a probability per pixel. Because the model sees a downscaled copy (typically 320 pixels square), fine detail below that resolution is guessed rather than seen — which is why the mask is upscaled with smoothing and why edges are soft. The model here is the "lite" variant at about 4.6 MB, small enough to download once and run on a phone; larger models are sharper and slower, and none of them think.

Transparency: PNG, alpha and the halo problem

JPG has no transparency, so a cutout must be saved as PNG (or WebP), whose fourth channel — alpha — stores the mask. Two things go wrong at the edge. A hard mask gives a jagged outline; a soft one keeps a rim of the old background's colour, the "halo" you see when a cutout made against green is placed on white. Feathering the edge (the softness slider) trades one for the other. Placing the subject on a background close in tone to the original hides halos; placing it on the opposite tone reveals them. How to reduce image file size without wrecking it explains why a transparent PNG is larger than the JPG it came from.

Where it fails and what to do

• Hair, fur, lace, smoke: thinner than the model's resolution. Expect a soft blob; soften the edge and choose a forgiving background, or use a tool with dedicated hair matting.
• Glass and transparency: the model labels the glass, not what's seen through it. Products in glass need manual work.
• Subject matches background: a white shirt on a white wall gives the network nothing to separate. Reshoot with contrast if you can.
• Several subjects: it picks what it thinks is salient — often all of them, sometimes one. Crop to the subject first.
• Tiny subjects in big frames: at 320 pixels the subject is a few pixels wide. Crop, then cut out.

For flat, single-colour backgrounds (product shots on white, screenshots), the colour-key mode is sharper than any network because it isn't guessing — it removes a colour.

Why it can run in your browser now

Until recently this required a server with a GPU, which is why background-removal sites upload your photos and charge per image. Two things changed: models got small (a few megabytes) and browsers gained a fast way to run them (WebAssembly, and the ONNX Runtime built on it). The trade is speed — a few seconds on a laptop, longer on an old phone — for privacy and cost: the photo never leaves the device, and nobody pays for a GPU. That is the design behind every AI tool on this site.