8. 3D, XR and AI

This chapter works at the practice layer of the five layers. Everything the course has generated so far has been flat: a page of text, a still image, a waveform, a sequence of frames. This week the output has extent in three dimensions, and in the second half of the chapter so does the person looking at it.

The first wave of generative AI was two-dimensional, covering text, images, and audio. The second wave is spatial: meshes, point clouds, Gaussian splats, whole scenes, characters, animations, levels, and interfaces. Spatial creative work, which is to say game design, product design, architecture, museum exhibits, and anything built for a headset, is being reshaped at something like the pace text was reshaped in 2022.

Two properties make this different from every medium so far, and both are constraints rather than features. The first is space: a generated object has to hold together from every angle, not only from the one the camera happened to be at, so there is no equivalent of choosing the good frame. The second is interaction: somebody moves through the result and decides for themselves what to look at, so a failure cannot be hidden by framing, pacing, or a cut.

The aim of the chapter is to leave you able to say which of three quite different technologies you are using when someone says “3D AI”. It is also to show what a headset adds that a screen does not, and to help you plan a piece of spatial work in which the generated parts are the ones generation is good at.

Three kinds of “3D AI”¶

Three quite different technologies travel under the same label, and confusing them is the single most common source of disappointment in this area. One reconstructs something that exists, one invents something that does not, and one decorates geometry that somebody else made.

Figure: Capture, generation, and texture all travel under the label of 3D AI, and only the first one is answerable to something outside itself.

Capture from real scenes¶

The classic problem: take photographs or a video of an object or a place, and produce something you can look at from angles the camera never occupied.

Photogrammetry is the traditional answer. Features are matched across overlapping photographs, camera positions are solved, and a mesh is triangulated from the result. It is decades old, it produces ordinary geometry, and machine learning has entered it mostly as denoising, hole filling, and feature matching rather than as a replacement.

Neural radiance fields, usually called NeRFs, arrived in 2020 and changed the framing Mildenhall et al., 2020. Instead of building a surface, a small neural network is trained on the photographs until it can predict the colour and density of any point in space seen from any direction. Asking for a new view is then a rendering operation rather than a lookup. The results were startling and both training and rendering were slow.

Gaussian splatting, published in 2023, keeps the idea of optimising a scene against photographs but drops the neural network Kerbl et al., 2023. A scene is represented as a very large number of small three-dimensional Gaussians, each with a position, a size, an orientation, a colour, and an opacity, and the whole set is optimised until re-rendering the scene matches the input photographs. Because splats can be sorted and drawn directly by a graphics card, the result renders at interactive rates, which is what moved the technique out of the papers and into phone apps within about a year. By 2026 it is the default for capture.

Watch on YouTube

Keep one distinction firmly in mind, because most of the practical trouble in this area comes from losing it. A splat or a NeRF is a rendering, not geometry. It can look photographic from any angle and still contain no surface you can stand on, collide with, cut, or deform. Converting one into a mesh is possible and it is lossy, and it usually throws away exactly the fuzzy detail that made the capture look good. If your destination is a viewer, a splat is excellent. If your destination is a game engine, budget for the conversion.

Figure: A splat is a cloud of overlapping blobs that renders convincingly from any angle and holds no surface, so converting it into a mesh of triangles throws away part of what made it look good.

Capture guides typically ask for on the order of a hundred overlapping photographs of a small object, or a slow video pass that gives the same coverage Polycam, 2024. What works well in 2026 is roughly what worked well for photogrammetry: outdoor scenes, statues, rooms, furniture, products. What still fails is reflective surfaces, transparent objects, fine structures such as hair and foliage, and anything that moves while you are capturing it. The reasons are not arbitrary, and this week’s lab is built around finding them out by hand.

Generation from scratch¶

Text-to-3D and image-to-3D take a sentence or a picture and return an object. Most current pipelines work in two stages: an image model generates several views of the same imagined thing, constrained to agree with each other, and a second stage lifts those views into geometry. That two-stage shape explains both the quality and the failure modes, because the object is only ever as consistent as the views it was reconstructed from.

What works well is props: cartoonish objects, background furniture, stylised weapons and tools, mid-detail assets that will be seen from a few metres away. What does not work is clean topology, rigged characters, and large coherent scenes. Topology here means the layout of the edges in a mesh, and a rig is the skeleton that lets a character be posed and animated. Ask for a room and you will typically get a room-shaped object rather than a room.

Texture and material generation¶

The third kind is the least discussed and the most quietly useful, because it slots into an existing pipeline without asking anybody to change how they work. Even where the geometry is entirely hand-made, a model can produce the surface: colour maps, normal maps and roughness maps, full physically-based material sets for a game engine, and environment maps for lighting a scene. Somebody who already models can adopt this on a Tuesday and keep every other habit.

All three produce something that looks plausible, and only the first is answerable to anything outside itself. A capture can be checked against the object it came from, and if the two disagree the capture is wrong, which is the same distinction chapter 7 drew between an analysis and a generation. A generated asset has nothing behind it to be wrong about, which is a liberty when you are inventing a prop and a problem when somebody takes your model of a building for a survey of it.

Virtual, augmented and extended reality¶

The vocabulary is used loosely in public and it is worth fixing once. Virtual reality (VR) replaces what you see and hear with a computed environment, so the room you are physically standing in disappears. Augmented reality (AR) leaves the world in place and adds graphics registered to it, so a virtual object appears to sit on your actual table. Mixed reality (MR) is the case where the added material is aware of the room, can be hidden behind real furniture, and can rest on real surfaces, which is a matter of degree rather than a separate technology. Extended reality (XR) is the umbrella term covering all three, and it is the right word when the distinction does not matter or when one device does several of them, which by 2026 most of them do.

Presence and embodiment¶

Two things change when a display is strapped to your head, and neither is resolution.

The first is presence, the sensation of being in the place the system is showing you. Presence is not one thing, and the most useful account splits it in two Slater, 2009. Place illusion is the feeling of being there, and it is produced by valid sensorimotor contingencies. You turn your head and the view updates the way it would in the world, you lean and the parallax is right, you crouch and you see under the table. Plausibility illusion is the feeling that what is happening is really happening, and it comes from events that refer to you personally, from the environment responding when you act on it, and from things behaving as things behave.

The split matters practically because the two have different causes and can be broken independently. Place illusion depends on the display and the tracking, not on the graphics being realistic, which is why a cartoon world with correct low-latency head tracking gives a far stronger sense of being somewhere than a photorealistic world that lags. Plausibility depends on the content, which is to say on your work. Generated material rarely damages the first, and it is very good at damaging the second.

The second change is embodiment. In a headset you have a body in the scene, and it is not quite the body you have outside it. You have a height, a reach, hands that may or may not be your own, and sometimes a full avatar. The immediate consequence is that scale stops being an aesthetic setting and becomes a bodily fact. An object ten per cent too large is a slightly odd picture on a screen and an unmistakably wrong object in a headset, because you are standing next to it and your body is the ruler. Door heights, seat heights, table heights, and step depths are the ones people notice first, and generated assets frequently arrive with no meaningful scale at all.

Embodiment also changes what an interface is. On a screen the interface is a pointer and a keyboard, and a designer arranges a rectangle. In XR the interface is where you look, where you reach, and where you walk. Design becomes a kind of choreography: what is within arm’s reach, what requires you to turn around, what you have to cross the room for. Chapter 12 takes up embodiment as a theoretical matter; here it is a set of measurements.

Where generated material enters¶

Generated material enters an XR pipeline at three points, and they carry very different risk.

Assets are individual objects placed in a scene somebody authored: a captured splat of a real sculpture, a generated prop, a texture set. This is the most common entry point and the safest, because everything around the generated object is under your control and can compensate for it.

Environments are the surroundings themselves: a captured room, a captured outdoor place, or a generated environment map providing the light and the horizon. Captured environments work well in 2026 and are a quick route to a convincing scene. Fully generated environments are still weak, for the same reason generated rooms are weak, which is that the models are far better at surfaces than at the relationships between objects.

Characters and behaviour are where a scene becomes an encounter. It brings together an avatar with a generated appearance, a synthesised voice from chapter 6, dialogue driven by a language model from chapter 4, and a control loop of the kind chapter 11 describes. This is where plausibility illusion is won and lost. A character that responds to what you actually did supports it strongly, and a character that responds a beat too late, or plausibly but to something else, breaks it completely.

One constraint governs all three. Everything in a headset lives inside a real-time budget, and a frame that arrives late is not merely a worse image, it is discomfort. That budget is why generated content in XR is in 2026 almost always produced in advance and baked into the scene rather than generated live. Anything running inside the loop must be small enough for the device, or hidden behind an interaction long enough to cover a round trip to a server.

Starting on a phone¶

You do not need a headset to work on any of this, and phone AR is the accessible entry point. A current phone has a camera, motion tracking, and often a depth sensor, which is enough to place a virtual object in the room you are actually standing in. Two routes matter for this course. A capture app will show a splat back to you in place, which is the fastest way to feel the difference between looking at a capture and standing in one. And WebXR, an open browser standard, runs the same page on a phone and in a headset browser, asking the device for whatever immersion it can provide. A scene you build once is viewable on whatever hardware the room contains. WebXR is the reason a small XR piece is a realistic 45 minutes of lab work rather than a term project.

What XR adds over video¶

It is worth being precise about what XR adds over the video of chapter 7, because it is easy to describe it as video that surrounds you, and that is the one thing it is not. A film decides for you where to look and when, and almost all of the craft of the medium lives in that decision. An XR scene gives the decision away. You gain agency, scale, and presence, and you lose framing, pacing, and the cut, which means the editing skills of the previous chapter do not transfer because there is nothing to edit. What replaces them is staging: putting things where a person will find them, and giving them a reason to turn around.

AI in design (2D and UX)¶

Outside 3D and games, design has become an AI-saturated discipline faster than almost any other, and mostly without a public argument about it.

The typical uses by 2026 are:

• layout drafts generated from a description of the content;
• icons and vector artwork, described in words and produced as editable curves;
• interface text, meaning microcopy, error messages, and onboarding flows;
• brand boards and moodboards, assembled from a few reference images;
• photographic retouching, where generative fill has become an ordinary part of the toolbar rather than a novelty.

The shift worth naming is that AI stopped being a place you go and became a menu item everywhere. A design student in 2026 should expect every tool in their stack to have generation built into it, which means the interesting question is no longer whether to use it but which decisions to keep human.

That question has a fairly precise answer, and it is the one the Reflect exercise below is about. When a hundred variations cost a minute, the variations are no longer the scarce resource, and the scarce resource becomes the criterion by which you choose among them. Design judgement moves out of production and concentrates at the two ends: specifying what the artefact is for, and selecting against that specification. Both of those get harder, not easier, when the supply of candidates is unlimited, because a plausible option you have no reason to reject is the most expensive kind of option there is.

AI in game development¶

Game development is a useful microcosm because it touches everything above and adds interactive behaviour on top. A small studio pipeline in 2026 might use generation at almost every stage.

• Concept art, generated with the image models of chapter 5 and used to align a team on a look before anybody models anything.
• Sprites and tilesets, generated and then hand-corrected for consistency across a set.
• 3D assets, generated, retopologised by hand where they need to deform, and textured with material tools.
• Voice, synthesised, and in professional work under an explicit licence from the actor whose voice it is, on the terms chapter 6 and chapter 3 set out.
• Music and ambience, generated as loops and stems that a composer arranges.
• Sound effects, generated, which is the least contentious use in the list because most of them were library assets already.
• Levels, still mostly classical procedural generation with generated decoration, because a level has to be playable and a plausible-looking level frequently is not.
• Dialogue for non-player characters, increasingly driven by small language models running locally so that the studio pays no per-word cost and the game works offline.
• Playtesting, partly automated, with software agents exploring builds to find places a player can fall out of the world.
• Localisation, machine-translated and human-reviewed, which was true before this wave and is more true now.

What has genuinely changed is the cost of assets. A small team can now populate a game with art it could not previously have afforded to commission, and studios describe the asset stage compressing by something in the region of an order of magnitude. Treat any specific multiple you see quoted with suspicion, because nobody counts the same way twice and the people quoting are usually selling something; the direction is well attested and the size is not.

The bottleneck has moved from production to taste: when the fiftieth prop is cheap, the difference between studios is what they choose to make and what they are willing to cut, and neither of those has become easier.

Chapter 7 introduced world models, which predict the next frame conditioned on an action and therefore behave like a playable environment rather than a clip Bruce et al., 2024. For games this points at content generated as it is played rather than authored in advance, and what stands in the way is the thing a game needs most, which is a rule that holds. An environment that invents a plausible next moment is not the same as an environment where the door you locked is still locked when you come back.

Where 3D AI still struggles¶

Clean topology. Generated meshes are almost always an undifferentiated field of triangles. That is fine for a distant prop and useless for anything that has to deform. A mesh bends along its edges, and a mesh whose edges do not follow the anatomy folds into creases at the elbow and flattens at the shoulder. Character work needs edge loops running around the joints, and putting them there is retopology, a skilled job that automatic tools have assisted for years and that generative models had not displaced by 2026.

Multi-object scenes. One object is easy and a room with correct relationships between its contents is hard. This is the same weakness as the counting failures in chapter 5: these models are strong at surface and texture and weak at discrete structure.

Articulated motion. Rigging a generated character and getting it to animate convincingly is not solved, and the two failures compound, because a bad rig on bad topology produces motion that is wrong in ways nobody can localise.

Editability. Once generated, a 3D model is much harder to change precisely than a 2D image. Stretching one leg, adjusting a curve, or fixing a self-intersection are manual operations, and there is no layer stack or history to reach back into.

Scale and units. Generated assets often arrive with no meaningful real-world size. On a screen that is a nuisance, and in a headset, for the reasons above, it is the difference between a scene that reads as a place and one that reads as a diorama.

The trajectory is clear, and the gap to professional-grade 3D production is wider than the gap to professional-grade 2D was in 2022. Plan around the gap rather than waiting for it to close during your semester.

This week’s lab: Explore, Reflect, Create¶

Three movements: break a capture tool on purpose, argue about where judgement sits now that variations are free, and make one spatial artefact.

A3 is due this week. Multimodal mini-piece was set in week 6 and is handed in now, so keep time clear for it before you start capturing; the path you take below is a portfolio piece and does not replace the assignment. The project proposal is set next week, so it is worth noticing this week which of these paths you would want to build on, and whether it would work as a performance or as an installation.

Explore (about 30 min)¶

Break it deliberately. You will learn more about capture from three failures than from one success, and the failures are predictable enough to plan.

1. Pick one capture or text-to-3D tool and stay with it, so that the differences you find are differences in the subject rather than in the software.
2. Capture or generate three subjects chosen to fail, one from each category:
   • a reflective subject, such as a mirror, a chrome kettle, a glossy car, or a window;
   • a thin subject, such as a bicycle, a wire chair, a plant with fine leaves, or hair;
   • a moving subject, such as a person who cannot hold still, foliage in wind, or a pet.
3. For each one, find the angle from which the result is worst and take a screenshot there. Three screenshots.
4. Write one sentence per failure naming the cause, not the symptom. “It looks smeared” is a symptom. “The surface colour changed with my viewing angle, so no two photographs agreed about what was there” is a cause.
5. Then predict, before you look it up, which of your three would be hardest to fix and why.

Optional. Compare captures side by side in the Splat viewer.

Reflect (about 15 min)¶

Work in pairs. This is a discussion, not a writing block.

1. Show each other your three failure screenshots without saying what the subject was. Let your partner guess the cause first, then compare it against the sentence you wrote. Where you disagreed is the interesting part.
2. Discuss: what does it mean to design something when a tool can generate a hundred variations in a minute? Where does design judgement actually sit now, and what is the scarce resource? Push each other for a concrete answer rather than “it depends”.
3. Discuss: which parts of 3D and design work do you most want to stay human, and why? Distinguish the parts you want to keep because they carry the meaning from the parts you want to keep because you enjoy them, since those are different arguments and both are legitimate.
4. Close the round by each stating aloud which Create path you are taking and what the finished artefact will be.

Create (about 45 min)¶

Pick one path. All four end with something committed to your portfolio, with a caption.

Path A: capture a Gaussian splat.

1. Photograph an object from many angles, on the order of a hundred overlapping frames, or take a slow video pass giving the same coverage Polycam, 2024. Choose something that is not on your Explore failure list.
2. Process the capture in a capture app and let it build the splat.
3. Take three screenshots from three genuinely different angles, including one from below or behind.
4. Find three failures in the result and say what caused each. This is the Explore method applied to something you were trying to make work.
5. Commit the screenshots and a one-paragraph caption.

Path B: a generated asset in a scene.

1. Generate a 3D asset from a text prompt in a text-to-3D service.
2. Import it into an open-source 3D suite Blender Foundation, 2024.
3. Place it in a small scene with one light and one camera, and set its real-world scale against something whose size you know.
4. Fix at least one thing by hand: the scale, the origin point, a hole, a stray floating fragment, or the material.
5. Render one still. Commit the render and a caption naming what you fixed and what you left.

Path C: a small XR scene.

1. Build a WebXR page containing one generated or captured asset and one generated ambient sound bed from chapter 6. An assistant from chapter 9 will write most of the page; it is short. Serve it over HTTPS, for instance from your repository’s pages site, because a page opened from disk cannot enter immersive mode.
2. Set the scale so the object is right for somebody standing next to it, and place it where a person has to turn or take a step to see it properly.
3. View it in a headset if one is available, and on a phone in AR if not. Both count.
4. Record a screen capture, or photograph somebody using it.
5. Write two sentences: one on place illusion and one on plausibility illusion, saying which held and which broke and what broke it.
6. Commit the page, the recording, and the captions.

Path D: the game-asset pipeline.

1. Generate four matching sprites for a small character: idle, walk, jump, and one action. Consistency across the four is the hard part.
2. Generate one tileset for a small environment in the same style.
3. Generate one music loop and three sound effects using chapter 6.
4. Optional, if you have time: assemble a playable scene in a lightweight framework with help from a coding assistant, as chapter 9 describes.
5. Commit the assets, the scene if you built one, and a caption saying which parts you corrected by hand.

At home, write this week’s entry in your practice log using the practice log template.

A critical look: does VR make you feel present?¶

The claim. Virtual reality is “the empathy machine”. The phrase comes from a 2015 TED talk by the film-maker Chris Milk Milk, 2015, and it has been repeated ever since by humanitarian campaigns, museums, and training providers. The argument is that because a headset puts you there, in a refugee camp or a rising sea or somebody else’s body, it produces understanding that a film cannot, and therefore changes what people think and do.

The evidence. The core of the claim is not marketing, and there is a serious research literature under it. Presence has been decomposed into two measurable illusions Slater, 2009. Place illusion follows from valid sensorimotor contingencies, which is to say from the system responding to your movements the way the world would, and it is manipulated by changing tracking, latency, and field of view. Plausibility illusion follows from events that refer to you and from things behaving credibly, and it is manipulated by changing what happens rather than how it is displayed. The important finding is behavioural rather than introspective: when both illusions hold, people act as though the situation were real. They flinch, they keep away from an edge, they respect an avatar’s personal space, and they show physiological stress responses in a virtual height scenario while reporting quite clearly that they know there is a floor. Acting as if is a much stronger result than saying it feels realistic, and it is what separates presence research from enthusiasm.

The method. These are controlled laboratory studies, and reading them well means asking three questions. What was manipulated? Latency, field of view, avatar appearance, and whether an event responded to the participant are different interventions with different implications. What was measured? Presence questionnaires are self-report, and a participant in a study about VR knows what the study is about. The results that carry weight are the ones with a behavioural or physiological measure attached, such as heart rate, skin conductance, postural sway, or the distance people keep from each other. And against what baseline? A study showing that a headset beats a desktop monitor has demonstrated something about displays. A study showing that an interactive headset scene beats a matched 360-degree video has demonstrated something about interaction, which is a far more interesting claim and a much rarer design. Samples in this field are usually small, so treat any single striking result as a starting point.

The limits. Three things separate the evidence from the claim, and none of them is a shortcoming a better headset would fix.

First, presence is not empathy. What the research measures is “I am in this place and this is happening to me”. Empathy is understanding another person’s situation and caring about it, and nothing in a presence questionnaire establishes that the second follows from the first. The step is assumed rather than demonstrated, and it is the whole of the claim.

Second, the effects that have been found fade. Where perspective-taking in VR has shifted attitudes, the shifts are usually measured immediately afterwards and are much weaker or absent at follow-up. The honest summary in 2026 is that VR reliably produces a strong immediate experience, and that durable attitude change is far less well supported than the promotional literature implies.

Third, generated environments introduce plausibility failures that authored ones did not have. Assets sit at the wrong scale, objects float or intersect, captured geometry dissolves when you walk behind it, and characters respond a beat too late or to the wrong thing. Place illusion survives all of this, because it rests on tracking and the tracking does not care what you put in the scene. Plausibility does not survive it. The rule for your own work follows directly: the more of a scene you generate, the more attention you owe to the half of presence that generation is best at breaking.

None of this makes VR ineffective. It is a reason to be specific rather than to avoid the medium. A headset is very good at putting somebody somewhere, which is worth a great deal for training, rehearsal, spatial understanding, and art. Whether being somewhere changes what a person believes about the people who live there is a separate question with a weaker answer. Who is being asked to feel for whom, and who benefits from the asking, is the question chapter 3 raises and no display technology answers.