Week 11: 3D Cameras

• Today's Lab: Camera Cloning
  • [ ] How are 3D vertices transformed into positions on a 2D screen?
  • [ ] What is the difference between a perspective and orthographic projection?
  • [ ] What do we need to know to describe a camera shot in a 3D game?
  • [ ] What are some challenges of positioning cameras in 3D spaces?
  • [ ] What are some additional challenges introduced by interactivity?
  • [ ] In your own words, explain how one or more fixed camera angles could work in a game and what information they need.
  • [ ] In your own words, explain how first-person cameras work and what information they need.
    • [ ] How is our geometry stuff from last week relevant here?
  • [ ] In your own words, explain how orbit cameras work and what information they need.
  • [ ] How is engine3d different from engine2d?
  • [ ] How is engine3d and the camera3d example different from our collision3d starter?

Pairs of teams should get together and discuss what you've been up to with your 3D game. What have you implemented so far? What are the roadblocks you're up against? Can you figure out together a way to solve them or work around them by tweaking the design?

What has teamwork been like? Can you think of ways to improve it or make it more equitable?

Last week the notes included this little buried treasure:

The extra cool thing about matrices is that they can be inverted. We can compute a local-to-world transform as above, but we can also convert a world-to-local transform by inverting each matrix along the way from the world frame of reference to the object's frame. This is important when we need to, say, put a coffee cup in a character's hand; but it's also important for viewing the scene in the first place! If we have a transform that positions a virtual camera in the world somewhere, and we know the world positions of all the objects in the scene, then we also need a way to transform those world positions into the camera's frame of reference—this is just the inverse of the camera's transformation. From there we can apply perspective projection and map the 3D world onto a 2D viewport.

(If you want a refresher on matrix transformations, check out this nicely illustrated article.)

After we apply the inverse camera transformation to every vertex in the scene (on the GPU, natch) we know where every object will be with respect to the camera—but we don't yet know what the camera will "see". Real digital cameras have a rectangular image sensor in their body which measures light (focused by the camera lens) from the scene. You can imagine that there is a four-sided pyramid shape projecting from the camera out into the world, and anything contained within the planes of the pyramid is in principle viewable by the camera. Computers don't like infinity, so we add a far plane to that pyramid; computers also don't like infinitely small points, so we put a near plane in too, where the camera's image sensor would be. What the camera sees—and what we'll eventually map onto the viewport—is the visible portion of the scene from the perspective of that near plane, normalized to fit into that pyramidal shape (the camera frustum). We call this final coordinate space clip space. By playing with the relative distances between the left and right planes (or top and bottom planes, or near and far planes) we can achieve many effects simulating camera field of view and other properties. (Santell's site has some good visualizations of this as well; here are a couple more.)

This frustum shape is what gives us the sense of perspective we need to make a scene feel 3D: farther objects are smaller (the plane they're on has to be shrunken to map onto the near plane) and nearer objects are larger (their size on their plane is closer to the size they'll be when projected onto the near plane). Since scene vertices are all defined in terms of homogeneous coordinates, we can apply a transformation which scales vertices' homogeneous w coordinate depending on the distance from the camera (their z coordinate!), and then divide out that w coordinate when returning to 3D coordinates to achieve sizes varying with distance. In the special case where our far plane is just the same size as our near plane, we have what's called an orthographic projection (parallel lines stay parallel).

In some sense this is where we have the key payoff of using homogeneous coordinates for everything: translations, scaling, and rotations all use one kind of matrix, which means that the camera projection code can uniformly (homogeneously) transform any object's location and size in space.

To sum up, object vertices go through a chain of transformations up to the point where they're drawn onto the screen:

Model space ★ model matrix ⏩ world space ★ view matrix ⏩ view space ★ projection matrix ⏩ clip space

So that's 3D graphics programming—at least, that's how we get vertices from the world to the screen. Somehow we define a camera transformation (a matrix, or an eye position/at direction/up direction, or a position and a quaternion) and parameters like frustum plane positions (maybe determined via field of view variables), and we get a way to go from world triangles to screen pixels. But how do we decide where to put the camera and what to point it at? Especially in an interactive setting, we might want the player to move the camera around, or have the camera follow the player character through space; we might have certain aspects of our game level that are meant to be viewed up close and others that are never meant to be near the camera, or viewed from behind.

In today's lecture we'll outline a couple types of cameras and how to implement them.

For this week's starter we'll generalize the code from last week into a crate called engine3d. Set up your workspace like so:

• A threed folder, with

  • A Cargo.toml like this one:

    [workspace]
    members = ["roguelike", "triangle", "triangle-tex", "number-guessing-game", "extreme-number-guess-challenge", "interactive-drawing", "scene2d", "scene2d-offscreen", "run-wasm", "engine2d", "test2d"]
    exclude = ["target", "content"]
    resolver = "2"
    [profile.dev.package.backtrace]
    opt-level = 3
    [profile.dev.package.image]
    opt-level = 3
    [profile.dev.package.png]
    opt-level = 3
    [profile.dev.package.adler]
    opt-level = 3
    [profile.dev.package.miniz_oxide]
    opt-level = 3
    [profile.release]
    debug=true
    

  • A content folder, with the contents of content.zip
  • An engine3d folder, with the contents of engine3d.zip

  • A camera3d folder, with a Cargo.toml like this:

    [package]
    name = "camera3d"
    version = "0.1.0"
    authors = ["Joseph C. Osborn <joseph.osborn@pomona.edu>"]
    edition = "2018"
    
    # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
    
    [dependencies]
    frenderer = {git="https://github.com/JoeOsborn/frenderer.git"}
    

    • And its src/main.rs:

      #![allow(dead_code)]
      
      use frenderer::animation::{AnimationSettings, AnimationState};
      use frenderer::assets::AnimRef;
      use frenderer::camera::Camera;
      use frenderer::types::*;
      use frenderer::{Engine, Key, MousePos, Result, WindowSettings};
      use std::rc::Rc;
      
      const DT: f64 = 1.0 / 60.0;
      
      struct Player {
          trf: Similarity3,
          model: Rc<frenderer::renderer::skinned::Model>,
          animation: AnimRef,
          state: AnimationState,
      }
      impl Player {
          fn tick_animation(&mut self) {
              self.state.tick(DT);
          }
      }
      struct World {
          camera: Camera,
          camera_control: OrbitCamera,
          player: Player,
          floor: Thing,
      }
      struct Thing {
          trf: Similarity3,
          model: Rc<frenderer::renderer::textured::Model>,
      }
      impl frenderer::World for World {
          fn update(&mut self, input: &frenderer::Input, _assets: &mut frenderer::assets::Assets) {
              // WS
              self.player.trf.prepend_translation(Vec3::new(
                  0.0,
                  0.0,
                  (input.key_axis(Key::S, Key::W) * 100.0 * DT as f32) / self.player.trf.scale,
              ));
              // AD to turn, not strafe
              self.player.trf.prepend_rotation(Rotor3::from_rotation_xz(
                  input.key_axis(Key::A, Key::D) * (PI / 4.0) * DT as f32,
              ));
              self.player.tick_animation();
              self.camera_control.update(input, &self.player);
              self.camera_control.update_camera(&mut self.camera);
          }
          fn render(
              &mut self,
              _a: &mut frenderer::assets::Assets,
              rs: &mut frenderer::renderer::RenderState,
          ) {
              rs.set_camera(self.camera);
              rs.render_skinned(
                  self.player.model.clone(),
                  self.player.animation,
                  self.player.state,
                  self.player.trf,
                  0,
              );
              rs.render_textured(self.floor.model.clone(), self.floor.trf, 0);
          }
      }
      fn main() -> Result<()> {
          frenderer::color_eyre::install()?;
      
          let mut engine: Engine = Engine::new(WindowSettings::default(), DT);
      
          let camera = Camera::look_at(
              Vec3::new(0., 0., 100.),
              Vec3::new(0., 0., 0.),
              Vec3::new(0., 1., 0.),
          );
      
          let floor_tex = engine.load_texture(std::path::Path::new("content/cube-diffuse.jpg"))?;
          let floor_meshes = engine.load_textured(std::path::Path::new("content/floor.obj"))?;
          let floor = engine.create_textured_model(floor_meshes, vec![floor_tex]);
          let tex = engine.load_texture(std::path::Path::new("content/robot.png"))?;
          let meshes = engine.load_skinned(
              std::path::Path::new("content/characterSmall.fbx"),
              &["RootNode", "Root"],
          )?;
          let animation = engine.load_anim(
              std::path::Path::new("content/anim/run.fbx"),
              meshes[0],
              AnimationSettings { looping: true },
              "Root|Run",
          )?;
          assert_eq!(meshes.len(), 1);
          let model = engine.create_skinned_model(meshes, vec![tex]);
          let world = World {
              camera,
              camera_control: OrbitCamera::new(),
              player: Player {
                  trf: Similarity3::new(
                      Vec3::new(0.0, -15.0, -10.0),
                      Rotor3::from_rotation_xz(0.01),
                      0.1,
                  ),
                  model,
                  animation,
                  state: AnimationState { t: 0.0 },
              },
              floor: Thing {
                  trf: Similarity3::new(Vec3::new(0.0, -25.0, 0.0), Rotor3::identity(), 10.0),
                  model: floor,
              },
          };
          engine.play(world)
      }
      
      pub struct FPCamera {
          pub pitch: f32,
          player_pos: Vec3,
          player_rot: Rotor3,
      }
      impl FPCamera {
          fn new() -> Self {
              Self {
                  pitch: 0.0,
                  player_pos: Vec3::new(0.0, 0.0, 0.0),
                  player_rot: Rotor3::identity(),
              }
          }
          fn update(&mut self, input: &frenderer::Input, player: &Player) {
              let MousePos { y: dy, .. } = input.mouse_delta();
              self.pitch += DT as f32 * dy as f32 / 10.0;
              // Make sure pitch isn't directly up or down (that would put
              // `eye` and `at` at the same z, which is Bad)
              self.pitch = self.pitch.clamp(-PI / 2.0 + 0.001, PI / 2.0 - 0.001);
              self.player_pos = player.trf.translation;
              self.player_rot = player.trf.rotation;
          }
          fn update_camera(&self, c: &mut Camera) {
              // The camera's position is offset from the player's position.
              let eye = self.player_pos
              // So, <0, 25, 2> in the player's local frame will need
              // to be rotated into world coordinates. Multiply by the player's rotation:
                  + self.player_rot * Vec3::new(0.0, 25.0, 2.0);
      
              // Next is the trickiest part of the code.
              // We want to rotate the camera around the way the player is
              // facing, then rotate it more to pitch is up or down.
      
              // We need to turn this rotation into a target vector (at) by
              // picking a point a bit "in front of" the eye point with
              // respect to our rotation.  This means composing two
              // rotations (player and camera) and rotating the unit forward
              // vector around by that composed rotation, then adding that
              // to the camera's position to get the target point.
              // So, we're adding a position and an offset to obtain a new position.
              let at = eye + self.player_rot * Rotor3::from_rotation_yz(self.pitch) * Vec3::unit_z();
              *c = Camera::look_at(eye, at, Vec3::unit_y());
          }
      }
      
      pub struct OrbitCamera {
          pub pitch: f32,
          pub yaw: f32,
          pub distance: f32,
          player_pos: Vec3,
          player_rot: Rotor3,
      }
      impl OrbitCamera {
          fn new() -> Self {
              Self {
                  pitch: 0.0,
                  yaw: 0.0,
                  distance: 50.0,
                  player_pos: Vec3::zero(),
                  player_rot: Rotor3::identity(),
              }
          }
          fn update(&mut self, events: &frenderer::Input, player: &Player) {
              let MousePos { x: dx, y: dy } = events.mouse_delta();
              self.pitch += (DT * dy) as f32 / 10.0;
              self.pitch = self.pitch.clamp(-PI / 4.0, PI / 4.0);
      
              self.yaw += (DT * dx) as f32 / 10.0;
              self.yaw = self.yaw.clamp(-PI / 4.0, PI / 4.0);
              self.distance += events.key_axis(Key::Up, Key::Down) * 5.0 * DT as f32;
              self.player_pos = player.trf.translation;
              self.player_rot = player.trf.rotation;
              // TODO: when player moves, slightly move yaw towards zero
          }
          fn update_camera(&self, c: &mut Camera) {
              // The camera should point at the player (you could transform
              // this point to make it point at the player's head or center,
              // or at point in front of the player somewhere, instead of
              // their feet)
              let at = self.player_pos;
              // And rotated around the player's position and offset backwards
              let camera_rot = self.player_rot * Rotor3::from_euler_angles(0.0, self.pitch, self.yaw);
              let offset = camera_rot * Vec3::new(0.0, 0.0, -self.distance);
              let eye = self.player_pos + offset;
              // To be fancy, we'd want to make the camera's eye an object
              // in the world whose rotation is locked to point towards the
              // player, and whose distance from the player is locked, and
              // so on---so we'd have player OR camera movements apply
              // accelerations to the camera which could be "beaten" by
              // collision.
              *c = Camera::look_at(eye, at, Vec3::unit_y());
          }
      }
      

Let's dig in to a few aspects of camera3d's main.rs. First let's just note that there is a Camera trait which describes methods that must be implemented by a game camera for the camera3d program. We'll get into the details on that later, but there are two implementors: FPCamera and OrbitCamera.

We'll jump to main next. You'll notice it's a very short function that just calls engine3d::run with some type parameters, a content path, and a window builder with parameters we control.

The next thing to read up on is these two types we pass into run: GameData and Game. GameData is just a struct we define with four model references in it; it serves a role similar to what engine2d called Rules. In a more complex game it would have level data and the like. Game, on the other hand (in this case parameterized further by a Camera type), has our dynamic game state: the marbles, the walls, the player, the camera, and collision contact Vecs. It also implements the trait engine3d::Game which means it must define start (which returns a Game and GameData), render (which issues rendering commands to the engine), and update (which updates game state). This is just a fancier version of what engine2d did.

Looking at the Player struct, it has a position (in its collision shape), a rotation, and linear and angular velocity. Its integrate method is about what we'd expect, but its render method is new. Instead of returning an InstanceRaw like we did last time, it receives an &mut InstanceGroups with a render method that takes a model and an InstanceRaw. engine3d::render::InstanceGroups is a new struct that gathers up instanced drawing commands issued for different models (either one instance at a time or in a larger batch of several at once) so that it can efficiently make batched drawing calls without knowing in advance all the models that might be drawn. Note the difference between Marbles and Player here.

Input handling is also different now: instead of using winit_input_helper, engine3d::events::Events provides queries for whether a key is pressed or where the mouse position is. This should help resolve issues we faced with unreliable key_pressed and key_released events.

Finally, the OrbitCamera and FPCamera structs are two example cameras. Change which one the game uses by modifying the call to engine3d::run at the end of the file.

Today's lab will have a reverse-engineering component and a relatively small programming component. We'll do it in triples.

First, work with two partners to find three game cameras that are interesting and that are different from the provided basic FPCamera and OrbitCamera examples. They should be games you can play and experiment with. They don't have to be 3D games but at least one should be.

Second, describe in fine detail how each these cameras work with respect to the camera's position and orientation, the character's movement, the player's control over the camera, and the larger environment they're in. This description should be sufficient to produce an implementation of the camera for someone who hasn't played the game you're talking about. Take note of what input data are necessary for these cameras and how they depend on context.

Finally, pick one of these camera types and implement it in the camera3d starter code. If it requires collision detection or ray casting, implement that too! Basic ray-plane and ray-sphere collision code is in chapter 5 of Real-Time Collision Detection and you can find code online as well. Efficiency isn't too much of a concern at this point. Be sure that you make necessary changes both to how the player moves and to how the camera moves! Player character, camera, and controls aren't really fully separable notions, so don't forget that they all contribute to a gameplay experience!