Vibe Coding XR: When Gemini Speaks Fluent Spatial

I’ve been watching the “vibe coding” trend with a mix of amusement and genuine interest. The idea that you can just describe what you want and have an LLM spit out working code is powerful, but it’s mostly been stuck in 2D land. XR prototyping has remained this painful exercise in stitching together perception pipelines, game engine quirks, and sensor data. Until now.

Google Research just dropped something called Vibe Coding XR, and honestly, it’s the first time I’ve felt genuinely excited about AI-assisted spatial computing. The workflow pairs Gemini Canvas with the open-source XR Blocks framework to turn natural language prompts into interactive, physics-aware WebXR applications for Android XR. And it does this in under 60 seconds.

How it works

The flow is refreshingly simple. You open the XR Blocks Gem in Chrome on an Android XR headset like the Galaxy XR, type or speak something like “Create a beautiful dandelion,” and Gemini handles the rest. It configures the scene, sets up perception and interaction logic, and builds a working XR app. A pinch gesture at the “Enter XR” button and you’re inside your creation—in this case, an animated dandelion that scatters when you interact with it.

If you don’t have a headset handy, the desktop Chrome simulator lets you test the same experience before deploying. Some features like depth sensing and hand tracking obviously shine better on actual hardware, but the simulator is good enough for rapid iteration.

The tech under the hood

The real magic is in how Gemini uses multi-step planning with specialized system prompts and curated code templates from XR Blocks. It’s not just generating random WebXR code—it’s learning from actual working examples and applying spatial logic automatically. The team claims this handles everything from physics to hand interactions without the user needing to understand any of it.

I’ve seen this approach tried before with varying degrees of success. What makes this different is the tight integration with a framework that’s already designed for spatial computing, rather than trying to bolt XR onto a general-purpose code generator.

Where it falls short

Let’s be real for a second. This is rapid prototyping, not production-ready application development. The demos shown are impressive for what they are—single-scene experiences with basic interactions. If you’re building anything with complex multiplayer logic, custom shaders, or intricate state management, you’re still going to need a proper development pipeline.

Also, the quality of the output depends heavily on how well Gemini understands the XR Blocks API. The team has clearly done a lot of work on the prompt engineering side, but I suspect edge cases will still produce janky results. The “Share” button that creates a public link is a nice touch, though.

The bigger picture

What interests me most is the direction this signals. We’re moving toward a world where describing an experience is enough to generate a functional prototype. For experienced developers, this means being able to test UI concepts, 3D interactions, and spatial visualizations in minutes instead of days. For educators and designers who don’t want to learn Unity or Unreal, it opens up XR as a medium for experimentation.

The team is presenting this at ACM CHI 2026, and the live demo at the Google Booth should be worth seeing. But even without attending, you can try it right now if you have an Android XR headset.

Vibe coding was already changing how we think about software creation. Making it spatial was the obvious next step. I just didn’t expect it to land this cleanly.