# Lecture 1: Images and image filtering

CS5670: Computer Vision Noah Snavely Lecture 15: Structure from motion Readings Szeliski, Chapter 7.1 7.4

Structure from motion Given many images, how can we a) figure out where they were all taken from? b) build a 3D model of the scene?

This is (roughly) the structure from motion problem Structure from motion Reconstruction (side)

Input: images with points in correspondence (ui,j,vi,j) Output structure: 3D location xi for each point pi motion: camera parameters Rj , tj possibly Kj Objective function: minimize reprojection error

(top) pi,j = Input

Camera calibration & triangulation Suppose we know 3D points And have matches between these points and an image How can we compute the camera parameters?

Suppose we have know camera parameters, each of which observes a point How can we compute the 3D location of that point? Structure from motion

SfM solves both of these problems at once A kind of chicken-and-egg problem (but solvable) Photo Tourism

First step: how to get correspondence? Feature detection and matching Feature detection Detect features using SIFT [Lowe, IJCV 2004]

Feature detection Detect features using SIFT [Lowe, IJCV 2004] Feature matching Match features between each pair of images

Feature matching Refine matching using RANSAC to estimate fundamental matrix between each pair Correspondence estimation

Link up pairwise matches to form connected components of matches across several images Image 1 Image 2

Image 3 Image 4 Image connectivity graph

(graph layout produced using the Graphviz toolkit: http://www.graphviz.org/) Structure from motion X4 X1

X3 g(R, T, X) X2

X5 minimize X7

non-linear least squares X6 p1,1 p1,3

p1,2 Camera 1 R1,t1

Camera 3 Camera 2 R2,t2 R3,t3

Problem size What are the variables? How many variables per camera? How many variables per point? Trevi Fountain collection

466 input photos + > 100,000 3D points = very large optimization problem Structure from motion Minimize sum of squared reprojection errors:

predicted image location observed image location

indicator variable: is point i visible in image j ? Minimizing this function is called bundle adjustment

Optimized using non-linear least squares, Levenberg-Marquardt e.g. Is SfM always uniquely solvable?

Is SfM always uniquely solvable? No Incremental structure from motion

Incremental structure from motion Incremental structure from motion Incremental structure from motion

Photo Explorer Demo Questions?

SfM Failure cases Necker reversal Structure from Motion Failure cases Repetitive structures

SfM applications

3D modeling Surveying Robot navigation and mapmaking Visual effects (Match moving) https://www.youtube.com/watch?v=RdYWp70P_kY