Augmented and Virtual Reality: Immersive Technology

The Pikachu on Your Study Table

Point your phone camera at your study table and open an AR app — say something like Pokemon GO or Google Lens. A small cartoon creature appears to be sitting right there, next to your pencil box, casting a faint shadow, staying put even as you tilt and move the phone around it. Your table has not changed. There is no creature on it. But your screen insists there is, and the illusion is convincing enough that you can "walk around" the table and see the creature from a different angle, exactly the way a real object would look.

Now imagine a completely different experience: you put on a bulky headset that covers your eyes entirely. The room around you — your table, your pencil box, the wall behind you — disappears from view. Instead you see a lunar surface, or a cricket stadium, or a chemistry lab, rendered so convincingly that turning your head reveals more of that world, not more of your bedroom. You have not been teleported anywhere. You are still standing in your room. But your brain, cut off from any visual evidence of your real surroundings, accepts the substitute.

These are two genuinely different tricks, solving two genuinely different problems: one adds digital content onto the real world you can still see, and the other replaces the real world with a digital one. The first is Augmented Reality (AR); the second is Virtual Reality (VR). This chapter is about how both of these tricks actually work — not just what to call them, but the cameras, sensors, geometry, and code that make the illusion hold together frame after frame, sixty times a second, without falling apart.

A Working Definition, Built From the Two Scenes Above

From the Pikachu example: Augmented Reality (AR) is a system that captures your view of the real world (usually through a camera) and overlays computer-generated content onto it in real time, so that the digital content appears to occupy a specific place in the real environment — not just floating on top of the screen like a sticker, but locked to a real location, correctly sized and correctly oriented as you move.

From the headset example: Virtual Reality (VR) is a system that replaces your entire visual (and often audio) input with a fully computer-generated environment, and continuously updates what you see as you move your head or body, so that your brain is tricked into feeling physically present inside that generated environment.

Notice the shared ingredient in both definitions: real-time response to your movement. A static 3D image of a dinosaur on a poster is not AR. A 360-degree photo that never changes no matter which way you turn your head is not VR — it is closer to a panorama. What makes both AR and VR "immersive technology" rather than just "graphics on a screen" is that the system is constantly re-measuring where you are and where you're looking, and re-rendering to match, fast enough that the delay is imperceptible. Everything else in this chapter is really about how that re-measuring and re-rendering is done.

The Reality-Virtuality Continuum

In 1994, researchers Paul Milgram and Fumio Kishino pointed out that "real" and "virtual" are not two isolated boxes — they are the two ends of a single spectrum, and most interesting systems sit somewhere in between. This idea is called the Reality-Virtuality Continuum, and it is the correct mental model to use instead of memorising AR/VR/MR as three unrelated buzzwords.

At the far left is the unmodified real environment. At the far right is a fully virtual environment (VR). In between sit two "mixed" cases: Augmented Reality, where a mostly-real scene has virtual objects added to it, and Augmented Virtuality, where a mostly-virtual scene has real elements pulled into it — for example, a television weather forecast where the presenter is a real, camera-captured person standing in front of a green screen, composited live into an entirely computer-generated studio. Both of these mixed cases, together, are what the industry calls Mixed Reality (MR) — any system where real and virtual objects coexist and can interact in real time. Microsoft HoloLens, which lets you pin an interactive 3D hologram to your actual desk and walk around it, is a well-known MR device. So "AR", "VR", and "MR" are not three competing technologies — MR is the umbrella term, and AR and VR are specific points near its two edges.

How VR Fools Your Brain: Two Eyes, One Illusion

Try this right now: hold one finger up about 15 cm in front of your nose. Close your left eye, and note where the finger appears to line up against something in the background — the edge of a door, say. Now switch: open your left eye and close your right eye. The finger appears to have jumped sideways relative to that same background object, even though your finger and the door didn't move — your eye did. That sideways "jump" is called binocular disparity, and it is one of the main cues your brain uses to judge distance: nearby objects show a large jump between your two eyes' views, and far-away objects show almost none.

A VR headset manufactures this same cue artificially. It does not have one screen — functionally, it renders two slightly different images of the same 3D scene, one for your left eye and one for your right eye, computed from two virtual cameras placed the same distance apart as your actual eyes (roughly 6.3 cm for an adult, called the interpupillary distance). Each eye sees only its own image through a separate lens. Your brain fuses the two slightly-offset images and reconstructs depth from the disparity between them — exactly the depth cue your finger experiment just demonstrated. This is called stereoscopic rendering, and it is the first of three ingredients that make VR convincing.

The second ingredient is head tracking: sensors (accelerometers and gyroscopes, plus in most modern headsets, small cameras that watch fixed points in your real room) continuously measure how your head is rotating and, in better headsets, how it is physically moving through space. The third ingredient is that the rendered image must update to match that measurement fast enough that you never notice a lag. Get all three right, and your brain accepts the substitute world as if it were real. Get the third one wrong, and you get a very unpleasant reminder that it isn't — which is worth a worked example.

Worked Example: Why VR Needs a High, Steady Frame Rate

A "frame" is one complete rendered image. The frame rate, measured in Hz (frames per second), tells you how many separate images the headset draws every second. The time budget available to render each single frame is simply:

frame_time (ms) = 1000 / frame_rate (Hz)

At a standard video frame rate of 30 Hz: frame_time = 1000 / 30 ≈ 33.3 ms per frame. At a common comfortable VR frame rate of 90 Hz: frame_time = 1000 / 90 ≈ 11.1 ms per frame.

Why does VR insist on the harder number (90 Hz, 11.1 ms) instead of the easier one (30 Hz)? Because of a quantity called motion-to-photon latency: the time between you physically turning your head and the correspondingly updated image actually reaching your eyes. A widely used comfort guideline in the VR industry is to keep this latency under roughly 20 milliseconds. At 33.3 ms per frame, the image you see is, on average, tens of milliseconds behind your actual head position — your inner ear (which senses rotation instantly, with no lag at all) reports one head orientation, while your eyes report a slightly older one. This mismatch between your balance system and your vision is the direct physical cause of simulator sickness — the nausea some people feel in VR. At 90 Hz and short, consistent frame times, the visual lag stays small enough that the two senses agree closely enough to not conflict. This is also why ordinary 30 fps video, which feels perfectly smooth on a flat screen, would feel sickening in a headset: on a flat screen, your head isn't the thing driving what you see, so there's no mismatch to detect.

Degrees of Freedom: Looking Around vs. Moving Around

Not all head tracking is equal. Describing an object's position and orientation in 3D space takes exactly six independent numbers: three for rotation — pitch (nodding "yes"), yaw (shaking "no"), and roll (tilting your head toward a shoulder) — and three for translation — moving along the x, y, and z axes (left-right, up-down, forward-back). These six numbers together are called six degrees of freedom (6DoF).

A cheap cardboard-and-lens VR viewer, which only has a gyroscope, can measure the three rotation numbers but none of the translation numbers. This is called 3DoF. In such a headset, turning your head correctly looks around the virtual world — but if you physically lean sideways or duck, the image does not shift to match, because the device has no way of knowing you moved. A modern standalone headset like a Meta Quest uses small outward-facing cameras to continuously spot and re-recognise fixed features in your real room (corners, furniture edges, wall textures) and calculates your position from how those features shift — this is the same core idea used in AR world-tracking, and it is what upgrades the headset to full 6DoF: now leaning sideways to peek around a virtual pillar actually works, because the system is tracking translation, not just rotation.

How AR Actually Works: The Camera-to-Screen Pipeline

AR looks like magic from the outside, but it runs the same five-stage loop, over and over, roughly 30 to 60 times every second:

Stage 1 grabs the current camera image. Stage 2 searches that image for something trackable — either a deliberately printed pattern (a marker) or naturally occurring visual detail like corners and edges (a feature). Stage 3 is the mathematically hard part: using what was detected, the system works out the device's pose — its position and orientation relative to the real-world object — which, as you now know from the previous section, is exactly six numbers (6DoF). Stage 4 uses that pose to render the virtual object from the correct viewpoint and correct scale, and layers it on top of the camera image. Stage 5 shows the result. Then the loop repeats with a brand-new camera frame. It is this constant repetition — recomputing the pose fresh, every single frame — that keeps the virtual Pikachu glued to your table even as your hand shakes slightly or you walk around it. If the app only computed the pose once and reused it, the creature would drift out of place within a second.

Marker-Based AR: The Math Behind "Sticking" a Virtual Object in Place

Stage 3 above — estimating pose — sounds abstract, but for the simplest case, marker-based AR, it comes down to a piece of geometry you already know: similar triangles. A printed marker of known real-world size appears smaller in the camera image the farther away it is, and by exactly how much smaller depends on distance and on a property of the camera called its focal length (roughly, how "zoomed in" the lens is, expressed in pixels once the camera is calibrated). The relationship is:

distance = (real_width * focal_length_px) / apparent_width_px

Suppose a printed AR marker is actually 10 cm wide, the phone's camera has a calibrated focal length of 700 pixels, and in the current video frame the marker measures 140 pixels wide. Here it is as code:

def estimate_distance(real_width_cm, focal_length_px, pixel_width_px):
    return (real_width_cm * focal_length_px) / pixel_width_px

marker_real_width = 10      # cm, the physical marker's printed size
focal_length_px   = 700     # calibrated once for this camera
pixel_width_now   = 140     # measured in the current camera frame

distance_cm = estimate_distance(marker_real_width, focal_length_px, pixel_width_now)
print(distance_cm)

Tracing it by hand: 10 * 700 = 7000, then 7000 / 140 = 50.0. The program prints 50.0 — the marker, and therefore the phone, is 50 cm apart. This is a genuinely simplified version of the actual technique (called the pinhole camera model) used inside real AR toolkits such as ARKit and ARCore, extended with full 3D rotation matrices to also recover the marker's tilt, not just its distance — but the core idea is exactly this calculation: closer things look bigger, so working backward from apparent size in pixels recovers real-world distance.

The AR engine then uses that distance to decide how big to draw the virtual object, so that it keeps looking like it belongs at that spot rather than floating at a fixed screen size:

def render_scale(distance_cm, k=6000):
    return k / distance_cm

print(render_scale(50))    # frame where marker is 50 cm away
print(render_scale(100))   # frame where marker is 100 cm away

Tracing: render_scale(50) computes 6000 / 50 = 120.0 (draw the object 120 pixels tall). If you then move the phone back until the marker is twice as far away, render_scale(100) computes 6000 / 100 = 60.0 — exactly half as tall. This matches how objects behave in real life: move twice as far from something and it looks half as big. Because the AR app recomputes this every single frame using the pipeline from the previous section, the virtual object shrinks and grows in step with your movement, which is precisely what makes it feel "attached" to a fixed point in the real world instead of floating on the glass of your screen.

Markerless AR and SLAM

Printed markers are reliable but inconvenient — you need to place a special pattern in the scene first. Most modern AR apps, including Pokemon GO placing a creature on your bare floor or the IKEA Place app previewing a sofa in your actual living room, use markerless AR instead. Rather than looking for one designed pattern, the software continuously extracts dozens of naturally distinctive points from the live video — a corner of a rug, the edge of a skirting board, a scuff mark on the floor — and tracks how those points shift, frame to frame, as the camera moves. By comparing how much each point shifted, the system solves for both the camera's own motion through space and a rough 3D map of the points around it, simultaneously. This technique has a name you'll see again in robotics: SLAM — Simultaneous Localization and Mapping. Google Maps' Live View walking-navigation feature, which overlays direction arrows onto the street ahead of you, layers a similar visual-matching idea on top of your phone's GPS to sharpen exactly where you're facing on a real street.

It's worth being precise about what counts as "full" AR here. A Snapchat or Instagram filter that sticks dog ears to your face is technically augmenting the real world with virtual content — but it typically only needs to track a face (a much smaller, more predictable tracking problem, solved by locating landmark points like the corners of your eyes and mouth), not the entire surrounding room. Placing a life-size virtual sofa believably on your living-room floor is the harder markerless-AR-with-SLAM problem, because the system must recover the geometry of an entire unknown room, including where the floor plane actually is, not just where a face is.

A Trick Older Than You Think

It's tempting to assume AR and VR are inventions of the last decade, but the core idea — synchronising an artificial sensory experience with your body's movement to produce an illusion — is nearly a century old. In 1929, engineer Edwin Link built the Link Trainer, one of the first flight simulators: a mock cockpit mounted on mechanical bellows that pitched, rolled, and yawed in sync with a trainee pilot's instrument readings, entirely without electronics, to train pilots without risking a real aircraft. In 1962, inventor Morton Heilig patented the Sensorama, an arcade-style cabinet that combined a stereoscopic 3D display, stereo sound, seat vibration, wind, and even smell to simulate riding a motorcycle through a city — a strikingly early, and strikingly multi-sensory, attempt at immersion. What changed only recently is not the idea but the engineering: small, high-resolution screens; motion sensors cheap enough to put in every phone; and enough processing power in a lightweight, battery-powered device to run the pose-estimation math from the previous sections at 90 times a second. The concept is old; the pocket-sized hardware to do it well is new.

Common Misconceptions, Corrected

Misconception: "AR is just putting a sticker or filter over a video." A sticker fixed to a corner of your screen, or a filter that ignores camera movement entirely, is not genuine AR — it fails the definition from earlier in this chapter, because it does not stay registered to a real-world location as you move. Real AR must continuously re-estimate pose (Stage 3 of the pipeline) every frame specifically so the virtual content tracks a real position, correctly scaled, from any viewing angle — which is exactly why the distance and scale calculations in this chapter matter, and why simply pasting an image over a video feed would look wrong the instant the camera moved.

Misconception: "A VR headset projects a real 3D hologram into the room." Nothing physical appears anywhere. A VR headset shows two flat, ordinary 2D images — one per eye — on small screens a few centimetres from your eyeballs. There is no light bouncing off any object in 3D space; the entire sense of depth is manufactured by binocular disparity between those two 2D images, exactly as demonstrated in the finger-and-door experiment earlier. The "3D" is a perceptual illusion constructed entirely inside your visual system, not a physical property of anything in the room.

Misconception: "AR, VR, and MR are three separate, unrelated technologies you must memorise independently." As the continuum diagram showed, they are points on one spectrum defined by a single variable — how much of what you perceive is computer-generated versus real — and Mixed Reality is simply the umbrella name for any point in between the two extremes where real and virtual content coexist and interact.

Where This Fits: CBSE Syllabus and India

In the CBSE Computer Science / Artificial Intelligence curriculum, AR and VR are studied under the "Emerging Technologies" umbrella alongside AI, Machine Learning, IoT, and Robotics. Typical exam questions ask you to define and correctly differentiate AR from VR, explain what makes a system "immersive" (real-time tracking and response to movement, as established at the start of this chapter), and identify or justify real-world applications — so being able to classify an unfamiliar example on the continuum, the way you practiced above, is directly exam-relevant, not just conceptual decoration.

You can find every point on this continuum in everyday Indian use already. Indian edtech platforms such as Practically build AR/VR-based virtual science labs mapped to the CBSE syllabus, letting students "perform" experiments — like a titration, or wiring a circuit — that a school lab might not have the equipment or safety margin for. Google Maps' Live View AR walking directions are used daily in Indian cities to disambiguate confusing junctions. Pokemon GO, built by the US company Niantic, remains widely played in Indian cities and parks. IKEA Place, from the Swedish furniture retailer IKEA, lets shoppers preview furniture at true scale in their own homes before buying. And flight simulators — direct engineering descendants of the 1929 Link Trainer — remain a core, and far cheaper, part of how pilots are trained before ever touching a real aircraft's controls, in India as everywhere else.

Check Your Understanding

1. Explain, in one or two sentences, the single most important difference between how AR and VR treat the real world around you.
2. Classify each of the following as AR, VR, Augmented Virtuality, or MR, and justify your answer using the continuum: (a) an Instagram filter that adds animated ears to your face; (b) a fully computer-generated cricket stadium explored through a Meta Quest headset; (c) a televised weather forecast where a real presenter stands in front of a virtual studio set; (d) a Microsoft HoloLens hologram of a 3D molecule pinned to your real desk that you can walk around.
3. A different AR marker is 8 cm wide. In the current camera frame it measures 160 pixels wide, and the phone's calibrated focal length is 700 pixels. Using the distance formula from this chapter, calculate how far the marker is from the camera. Show your working.
4. A budget headset advertises a horizontal field of view of 90°, while an unaided pair of human eyes together covers roughly 200°. What percentage of the natural human field of view does this headset cover?
5. Using render_scale(distance_cm, k=6000) defined earlier in this chapter, what value is printed by print(render_scale(175))? Round to two decimal places and show your calculation.
6. A friend says, "My cardboard VR viewer is broken — when I lean sideways to peek around a virtual box, the view doesn't change." Using the vocabulary from the Degrees of Freedom section, explain why this is very likely normal behaviour, not a fault.

Answers: (1) AR keeps the real world visible and adds digital content on top of it; VR replaces the real world entirely with a computer-generated one. (2) (a) AR — a real face with virtual content added and tracked to it; (b) VR — entirely computer-generated environment; (c) Augmented Virtuality — a mostly virtual scene with a real element (the presenter) composited in; (d) MR — real and virtual objects coexisting and interactable in the same space. (3) distance = (8 × 700) / 160 = 5600 / 160 = 35 cm. (4) 90 / 200 = 0.45 = 45%. (5) render_scale(175) = 6000 / 175 ≈ 34.29. (6) A cardboard viewer typically only has a gyroscope, giving 3DoF (rotation only, no translation tracking); it has no camera-based positional tracking, so it cannot detect sideways leaning at all — this is an expected hardware limitation, not damage.

Summary

• AR overlays computer-generated content onto a still-visible real world; VR replaces the real world entirely with a computer-generated one. Both require real-time tracking of your movement to count as genuinely immersive.
• The Reality-Virtuality Continuum (Milgram & Kishino, 1994) places Real Environment and Virtual Environment at two ends of one spectrum, with Augmented Reality and Augmented Virtuality in between; Mixed Reality is the umbrella term for that entire in-between region.
• VR creates depth using stereoscopic rendering (two eye images exploiting binocular disparity), combined with head tracking and low-latency rendering; motion-to-photon latency above roughly 20 ms is a major cause of simulator sickness, which is why VR targets high, steady frame rates like 90 Hz (11.1 ms per frame) rather than 30 Hz (33.3 ms per frame).
• Six degrees of freedom (6DoF) — three rotational (pitch, yaw, roll) plus three translational (x, y, z) — describe full head/device tracking; cheap 3DoF devices track rotation only.
• AR runs a five-stage loop every frame: capture, detect features/markers, estimate 6DoF pose, render and composite, display — repeated continuously so virtual objects stay registered to the real world as you move.
• Marker-based AR estimates distance using similar triangles (distance = real_width × focal_length / apparent_pixel_width) and rescales the virtual object each frame to match, which is what makes it look physically "attached" rather than pasted on.
• Markerless AR uses SLAM (Simultaneous Localization and Mapping) to track naturally occurring visual features instead of a printed marker, enabling apps like IKEA Place or Pokemon GO to place objects on ordinary, unmarked surfaces.
• Immersive technology predates digital computers in concept — the 1929 Link Trainer and 1962 Sensorama both synchronised artificial sensory input with the user's actions — what changed recently is the hardware, not the core idea.