9. AI for 3D, design, and games

Why this matters¶

The first wave of generative AI was 2D: text, images, audio. The second wave is spatial: 3D meshes, point clouds, Gaussian splats, full scenes, characters, animations, levels, interfaces. Spatial creative work — game design, product design, architecture, museum exhibits, AR/VR — is being transformed at the same pace text was transformed in 2022.

This week we look at where AI fits into 3D and design pipelines, and what its strengths and weaknesses are when shape and space — not just colour and word — are the medium.

Three kinds of “3D AI”¶

You will meet three quite different things:

1. 3D capture from real scenes¶

The classic problem: take a few photos or a video of an object or a place; produce a digital 3D model.

• NeRFs (Neural Radiance Fields) — train a small neural network on multi-view photos to render any new view. Pioneered in 2020.
• Gaussian splats (3D Gaussian Splatting) — a 2023 technique that represents scenes as millions of tiny 3D Gaussians. Renders fast, looks photorealistic. Now the dominant capture technology.
• Photogrammetry + AI denoising — a more traditional pipeline, accelerated and improved by AI.

Tools: Luma AI, Polycam, Postshot, Scaniverse.

What works well in 2026: outdoor scenes, statues, rooms, products. What does not: reflective surfaces, hair, dynamic scenes, transparent objects.

2. Generative 3D from scratch¶

Text-to-3D and image-to-3D pipelines. Tools: Meshy, Tripo3D, Rodin, and many research models on Hugging Face.

What works well: cartoonish objects, props, mid-poly assets for games. What does not: clean topology (game-ready quad meshes), rigged characters, large coherent scenes.

3. Generative texture and material AI¶

Even when the geometry is hand-made, AI can produce:

• Textures — diffuse, normal, roughness maps from a text prompt or photo.
• PBR materials — for game engines.
• Lighting and HDRIs — environment maps for realistic rendering.

Tools: Polycam Textures, Substance 3D (with AI features), Layer for game art.

AI in design (2D and UX)¶

Outside of game and 3D contexts, “design” is now an AI-saturated discipline:

• Layout drafts — Figma’s AI features, Galileo, Recraft.
• Icon generation — described in words, produced in vector.
• UI text — microcopy, error messages, onboarding flows.
• Brand boards — Midjourney moodboards, Recraft brand kits.
• Photo retouching — Photoshop’s generative fill is now a standard tool.

The shift is from “AI generates a finished asset” to “AI is in every menu”. A design student in 2026 should expect every tool in their stack — Figma, Adobe Suite, Sketch, Canva — to have AI built in. The interesting questions are no longer about whether to use AI but about which decisions to keep human.

AI in game development¶

Game development is a useful microcosm because it touches all of the above plus interactive behaviour.

A modern indie pipeline in 2026 might use:

• Concept art — generated with image models (chapter 5).
• 2D sprites and tilesets — generated and refined with Scenario or similar.
• 3D assets — generated meshes, hand-retopologised, textured with AI material tools.
• Voice acting — synthetic, often with permission to use a specific voice actor’s clone.
• Background music and ambient stems — generated.
• Sound effects — generated.
• Procedural levels — sometimes AI-assisted, more often classical procedural generation with AI tweaks.
• In-game NPC dialogue — increasingly powered by small local LLMs.
• Playtesting — automated, with AI agents exploring builds.
• Localisation — LLM-translated, human-reviewed.

You can ship a real, polished game in 2026 with two people and a year of work. The bottleneck has shifted from production to taste.

Where 3D AI still struggles¶

• Clean topology. Generated meshes are almost always “potato-mesh” — fine for distant props, useless for animated characters without manual retopology.
• Multi-object scenes. A single object: easy. A whole room with proper relationships: hard.
• Articulated motion. Rigged characters animating well are not yet a solved problem.
• Editability. Once generated, a 3D model is harder to edit precisely than a 2D image. Stretching a leg, tweaking a curve, fixing an intersection — these are still mostly manual.

The trajectory is clear, but the gap to professional-grade 3D production is wider than the gap to professional-grade 2D was in 2022.

This week’s lab: Reflect, Explore, Create¶

Reflect (≈ 30 min, in lab + your weekly log)¶

Pick one prompt and write 150–300 words in your weekly log:

1. What does it mean to “design” a thing when the AI can generate 100 variations in 60 seconds? Where does design judgement now sit?
2. A small game studio with five people in 2018 made one game in two years. The same studio with five people in 2026 can make six games in two years. What changes culturally?
3. Which parts of 3D / design work do you most want to stay human, and why?

Explore (≈ 20 min, in lab) — feel the limits first¶

Before committing to a Create path below, spend 20 minutes deliberately breaking one 3D AI tool of your choice. Try at least three inputs likely to fail:

• a reflective surface (a mirror, a glossy car);
• a thin structure (a chair, a bicycle, hair);
• an unusual perspective (a top-down photo, fish-eye, low light).

Capture three failure screenshots and one short note for each. Your Create output below will be better for it.

Create (≈ 60 min, in lab + carry-over to your portfolio) — pick one path¶

Path A — Gaussian splatting (no code).

1. Take 50–100 photos of an object (a plant, a sculpture, a piece of furniture) from many angles.
2. Upload to Luma AI or Polycam and let it process the splat.
3. Take three screenshots from three angles. Identify three failures of the capture and explain what caused them.
4. Commit the screenshots and a one-paragraph caption.

Path B — Text-to-3D + scene.

1. Generate a 3D asset in Meshy or similar.
2. Import into Blender (free).
3. Place it in a small scene with one light and one camera.
4. Render a single still. Document the steps and commit the render.

Path C — Game asset pipeline (advanced).

1. Generate four matching 2D sprites for a tiny game character (idle, walk, jump, action).
2. Generate one tileset for a small environment.
3. Generate one music loop and three sound effects.
4. (Optional) Combine in a small playable scene using Phaser or PICO-8 with help from an AI coding assistant (chapter 8).
5. Commit the assets (and the scene, if you built it) to your portfolio.

Going further¶

• Kerbl, Kopanas, Leimkühler, Drettakis, 3D Gaussian Splatting for Real-Time Radiance Field Rendering Kerbl et al., 2023 — the original 2023 SIGGRAPH paper.
• Polycam Academy Polycam, 2024 — quick, video-based introductions to 3D capture.
• Blender Blender Foundation, 2024 — the free, open-source 3D suite, with a growing ecosystem of AI-assisted add-ons.