How do 3D photos work? Depth maps, mattes and two planes, explained

A 3D photo is a single 2D image that moves as if it had depth. Software estimates how far every pixel is from the camera (a depth map), separates the subject from the background, fills in what the subject was hiding, then shifts the layers by different amounts as you move — parallax — so your eyes read depth.

“3D photo” means three different things: a stereo pair with one image per eye, the phone features Facebook and Apple ship, and the depth-parallax scene this article is about — which is what Facebook 3D Photos, iOS Spatial Scenes and Gifsy all do underneath. Everything below is the exact pipeline Gifsy runs, with the models named and the numbers as they are in the code. For using interactive 3D photos on a website — which platform, which photos, what it costs — start with the complete guide; to put one on a page right now, the embed guide.

Drag the scene above. The viewer is drawing an image, a depth map, a mask and a backdrop — no AI is running in your browser right now.

The pipeline in one line

1. Estimate a depth map from the photo.
2. Cut a subject matte — which pixels are the thing in front.
3. Refine the depth with the matte so edges snap to the picture and the halo goes.
4. Paint a backdrop with the subject removed, so moving it reveals scenery rather than a smear.
5. Render two displaced planes — backdrop and subject — plus a contact shadow, in WebGL.
6. Move the camera with the pointer or a drag, inside a cone the photo can honestly support.

Step 1 — The depth map: how software guesses distance from one photo

Monocular depth estimation is a neural network that looks at a single image and predicts, for every pixel, how far away it is. It learned the cues a person uses — size, occlusion, perspective, texture gradient, haze — from millions of images with known depth. The output is relative depth: near versus far, not metres.

Gifsy runs Depth Anything V2 Small (24.8 M parameters, Apache-2.0, the only V2 size under a permissive licence) as an fp16 ONNX model in the browser. It is a vision transformer with 14-pixel patches, so the photo is resized to a long edge of 518 px — or 770 px on a machine with 8 GB and 8 cores or more, because the later edge refine has more real detail to work with — with both sides rounded to a multiple of 14. The raw output is normalised to 0–1 using the 0.5th and 99.5th percentiles rather than the true min and max, so a few outlier pixels can't flatten the whole map. On Gifsy, brighter means closer.

The network ships in two parts, and both run on your device: the encoder — 44 MB, cached after the first run — and the small head that turns the encoder's output into depth.

Step 2 — The subject matte: deciding what's in front

A depth map alone tilts like a sheet. What makes a scene read as 3D is a subject moving against its background, so a second model, ISNet (from the DIS segmentation work, via the open-source @imgly/background-removal library), produces a per-pixel alpha of the subject — entirely in the browser.

The raw alpha is cleaned: values below about 9% are dropped as fringe, islands smaller than 5% of the main region are erased (a crumb, a blade of grass), and the edge gets a one-pixel feather. The matte stays full-frame and registered to the image and the depth map — never cropped — because the layers must line up. If ISNet finds nothing to isolate, the scene falls back to a single displaced plane; the app says so: “Depth-only view — segmentation couldn't separate a subject.” That is the honest reason some photos look flat.

Why the raw depth map has a halo, and how the matte fixes it

Depth models bleed a soft halo of “near” into the background pixels hugging a subject, which renders as a fuzzy cardboard cut-out. Gifsy refines the map in three passes, all on the CPU, once per generation. A guided filter uses the photo's own luminance as its guide, so depth edges snap onto real image edges (radius 2% of the short side, clamped to 3–16 px). Then the subject plus a contamination band around it is marked unknown and the background depth is extrapolated inward through that band with a push-pull pyramid; the subject's own relief is kept inside the alpha and lifted forward by a fixed 0.06 so it always leads the backdrop. Finally a smootherstep contrast curve (strength 0.35) steepens the mid-range, where real photos bunch their depth, so near, mid and far separate without more displacement — displacement is what causes artefacts.

Step 3 — The backdrop: painting the subject out

When the subject slides, the pixels it used to cover become visible — disocclusion. Without a fill you see a smeared ghost of the subject where it used to be. In the live preview, Gifsy fills the hole with a push-pull blur from the surrounding background, which is fast but leaves a flat smear in the gap parallax opens beside the subject — worst on landscapes. At publish, once, on the server, LaMa(“large mask inpainting”, with Fourier convolutions) paints the subject out properly at a fixed 512×512, letterboxed to keep the aspect ratio, and the fill is mapped back to full resolution and baked into the scene as its own file. The viewer never runs it.

Step 4 — Two planes, not a mesh

The scene is two plane meshes of 150×150 segments in Three.js. The backdrop plane carries the inpainted fill, overscanned by 8% so a slide never reveals a gutter. The subject plane carries the photo multiplied by the matte's alpha. Each vertex is pushed toward the camera by the refined depth in a custom shader. A third, thin mesh is the contact shadow: a soft black copy of the silhouette just in front of the backdrop that lags the subject, so a gap opens under it as it moves — the strongest floating cue there is.

Why not a mesh? One photo contains no hidden surfaces. Build a mesh from a single depth map and every silhouette becomes a torn skirt of stretched triangles; Facebook's 2018 3D Photos hit exactly this and had a neural network hallucinate the gaps. A mesh you can spin all the way round is a different product — see how the tools compare. Why not one plane? Depth without separation reads as a sheet tilting, not a subject standing in front of a background. And a single shifted plane stretches the texture into a horizontal smear at the depth step; Gifsy slides the subject plane almost rigidly (15% internal relief) so its feathered edge translates instead of stretching.

The camera is placed at exactly the distance that makes the plane cover the viewport, with 8% bleed at rest to hide the sway, easing in slightly on hover. Cut-out scenes use a contain fit instead so feet and edges aren't cropped.

Step 5 — The orbit, and why it stops at 20°

Two motion modes. Hover parallax: the pointer position becomes a depth-weighted offset, with the subject responding about three times as strongly as the backdrop (0.34 versus 0.10) around a pivot depth of 0.4. Spin: a drag steers a real camera arc around the displaced relief — actual perspective and self-occlusion, not a shader slide.

The arc is capped at ±0.35 radians (about 20°) sideways and ±0.22 (about 13°) vertically. A single-image depth map is a one-sided height-field, a bas-relief: it has no back and no sides, so past a shallow cone the stretched silhouette edges smear. We set the cone from frames, not taste — at ±32°/±19° a frame-filling cartoon smeared its outer third, so the limit came in until a held drag stayed inside the honest zone. It sat at ±0.40 (about 23°) until September 2026; we took it to 0.35 after printed labels on product photos stretched visibly at the extremes. One more number learned the hard way: the camera radius is 0.85× the cover distance, because at the old 23° cone a plane tilted to the corner foreshortened to about 0.896 of its width, and at 0.9 the viewport saw past the plane's corner at every extreme.

What 3D photos can't do

• Show a back or sides — it is a height-field, so no 360°.
• Orbit past about ±20° sideways or ±13° up and down without the silhouette stretching.
• Handle frame-filling subjects: nothing behind them to parallax against, so the edges tear on the first drag. A subject cut off by the frame edge has no matte there at all.
• Separate a subject from a busy or low-contrast background — the matte and the inpainter both struggle, and the scene falls back to depth-only.
• Do much with fewer than about 640 px on the long edge; the app warns below that.
• Cope well with hair, glass, water, motion blur and low light — Meta's 2019 list of hard cases still holds.

3D photo vs 3D model vs 3D video vs spatial photo

Video wins where a page can't run an iframe — social feeds, email. Most 3D photo tools export an MP4; the comparison sorts them by that. For the interactive row, the embed guide.

What leaves your browser (and what doesn't)

Your photo stays in the browser. The depth model runs entirely on your device; only if its second half can't load there does the browser fall back to sending the intermediate activations — not the photo — to our server for that step. When you publish, the finished scene — the processed image, its depth map, the subject mask and the generated backdrop — is uploaded so it can be embedded, and at that point your image and mask are also sent once to the inpainting step. Published scenes are public. The full account is the privacy policy.

Frequently asked questions

Make a 3D photo from your own image — the viewer needs no AI, so the scene at the top of this page is exactly what your visitors would get. The older, hand-layered version of this effect is covered in the 2.5D parallax effect.

Sources: Depth Anything V2 (arXiv 2406.09414; github.com/DepthAnything/Depth-Anything-V2); ISNet / DIS (arXiv 2203.03041; github.com/xuebinqin/DIS); LaMa (arXiv 2109.07161; github.com/advimman/lama; the Carve/LaMa-ONNX export); He, Sun & Tang, “Guided Image Filtering”; Three.js; onnxruntime-web. Prior art: Alan Zucconi's “Inside Facebook 3D Photos” and Meta's 2019 “3D photos: how they work”.