Google Photos’ Auto Frame fixes your bad angles after the fact

We’ve all been there. You snap what you think is the perfect shot, only to review it later and realize the angle is slightly off. Maybe the camera was a hair too high, or you caught too much of one side of your face. Or it’s that selfie with the great smile ruined by a wide-angle lens making your nose look like a different continent. Classic photo editing tools can’t fix this. Crop all you want, zoom in till it’s pixelated — the underlying perspective is still locked in. The parallax is fixed, and what’s outside the frame stays invisible.

Google’s new Auto frame feature in Google Photos aims to solve exactly this problem. Instead of treating your photo as a flat image, it interprets it as a 3D scene frozen in time. Then it moves the virtual camera around inside that scene, generating whatever was previously hidden behind the foreground. It’s now rolling out as part of the Auto frame feature, and honestly, it’s the kind of thing that sounds like magic until you see the demos.

The approach breaks down into two stages: first, the system estimates the 3D structure of the scene and the original camera position. Then it uses generative AI to fill in the visual gaps that appear when you shift the viewpoint. What sets this apart from other generative editing tools is the decoupling of 3D estimation from image formation. You get full control over both camera intrinsics (like focal length) and extrinsics (position and orientation).

For the 3D estimation, Google uses an internal model tuned specifically for reconstructing human bodies and faces. This is smart — generic depth estimation tends to mangle faces, and nobody wants their portrait looking like a melted wax figure. The model spits out a 3D point for every pixel in the original image, plus an approximation of the original focal length. Then classical 3D rendering takes over to produce what the image would look like from the new camera position.

Of course, moving the virtual camera reveals the dirty secret of 3D reconstruction: holes. You can’t render what was never captured. The point map is always incomplete from a new angle. To fix this, Google trained a latent diffusion model on pairs of images with known camera parameters. It learns to reconstruct the second image from the re-rendered first one. At inference time, it uses classifier guidance with regional scaling to fill those gaps convincingly.

I’ve seen similar ideas bounce around research papers for years, but actually shipping this in a consumer product is another beast entirely. The fact that it runs on-device (or at least within Google Photos’ infrastructure) without requiring you to manually specify camera parameters is impressive. It just works — you open a photo, hit Auto frame, and it suggests a better composition.

Is it perfect? Probably not. Generative inpainting still has its quirks, and I’d expect some artifacts in complex scenes with lots of fine detail. But for the use case of fixing that slightly-off selfie or group shot, this could be genuinely useful. It’s a far cry from the days of just cropping and hoping for the best.