Lecture 1: Images and image filtering

Lecture 1: Images and image filtering

CS5670: Computer Vision Noah Snavely Lecture 15: Structure from motion Readings Szeliski, Chapter 7.1 7.4

Structure from motion Given many images, how can we a) figure out where they were all taken from? b) build a 3D model of the scene?

This is (roughly) the structure from motion problem Structure from motion Reconstruction (side)

Input: images with points in correspondence (ui,j,vi,j) Output structure: 3D location xi for each point pi motion: camera parameters Rj , tj possibly Kj Objective function: minimize reprojection error

(top) pi,j = Input

Camera calibration & triangulation Suppose we know 3D points And have matches between these points and an image How can we compute the camera parameters?

Suppose we have know camera parameters, each of which observes a point How can we compute the 3D location of that point? Structure from motion

SfM solves both of these problems at once A kind of chicken-and-egg problem (but solvable) Photo Tourism

First step: how to get correspondence? Feature detection and matching Feature detection Detect features using SIFT [Lowe, IJCV 2004]

Feature detection Detect features using SIFT [Lowe, IJCV 2004] Feature matching Match features between each pair of images

Feature matching Refine matching using RANSAC to estimate fundamental matrix between each pair Correspondence estimation

Link up pairwise matches to form connected components of matches across several images Image 1 Image 2

Image 3 Image 4 Image connectivity graph

(graph layout produced using the Graphviz toolkit: http://www.graphviz.org/) Structure from motion X4 X1

X3 g(R, T, X) X2

X5 minimize X7

non-linear least squares X6 p1,1 p1,3

p1,2 Camera 1 R1,t1

Camera 3 Camera 2 R2,t2 R3,t3

Problem size What are the variables? How many variables per camera? How many variables per point? Trevi Fountain collection

466 input photos + > 100,000 3D points = very large optimization problem Structure from motion Minimize sum of squared reprojection errors:

predicted image location observed image location

indicator variable: is point i visible in image j ? Minimizing this function is called bundle adjustment

Optimized using non-linear least squares, Levenberg-Marquardt e.g. Is SfM always uniquely solvable?

Is SfM always uniquely solvable? No Incremental structure from motion

Incremental structure from motion Incremental structure from motion Incremental structure from motion

Photo Explorer Demo Questions?

SfM Failure cases Necker reversal Structure from Motion Failure cases Repetitive structures

SfM applications

3D modeling Surveying Robot navigation and mapmaking Visual effects (Match moving) https://www.youtube.com/watch?v=RdYWp70P_kY

Applications Photosynth Applications Hyperlapse https://www.youtube.com/watch?v=SOpwHaQnRSY

Recently Viewed Presentations

  • Chapter 3 Section 5 - Mr. Kay's Class Website

    Chapter 3 Section 5 - Mr. Kay's Class Website

    Repealing the Stamp Act. Delegates from the 9 colonies got together and held the Stamp Act Congress in 1765. Sent a reply on the Stamp Act to congress. It was repealed in 1766. Parliament was angry that the colonists challenged...
  • Ma famille - MME. PASYK'S FABULOUS FRENCH

    Ma famille - MME. PASYK'S FABULOUS FRENCH

    Ma famille. By: Karolina. Ma mère. Voici ma mère. Elle s'appelleIzabela. Elle aime le vacances. Elle dètestel'hiver. Elle favoritism couleurc'est bleu. Ma père. Voici ma père. Il s'appelle Richard. Il aime de la musique. Il dèteste le chat.
  • Presentation to the Portfolio Committee on Sport and

    Presentation to the Portfolio Committee on Sport and

    45 % female participation. 251 sports specific and general workshops, 9 781 participants, 35% female participation. On average 35 000 to 40 000 learners per week, approximately 48% females. Work in about 140 schools PE Workshops Built 36 facilities, 14...
  • Short Story Unit - Ms. Pacheco's Website

    Short Story Unit - Ms. Pacheco's Website

    Short Story Unit. What is a short story? Have you ever read a short story? If so, which one? What are the parts of a short story? Short stories usually lack detail. Short stories are brief. Short stories use P.A.C.T.S...
  • Think like an historian:  H- Historical Context. Explain

    Think like an historian: H- Historical Context. Explain

    The work is an exception to some of the Egyptian art norms. Showing a normal body with sagging chest is unusual in Egyptian culture art. The work is naturalistic with individualized, defined features where as the art of the pharaoh...
  • Webinar Host - nQuery Sample Size Software

    Webinar Host - nQuery Sample Size Software

    Sample Size Re-estimation (SSR) Will focus here on specific adaptive design of SSR. Adaptive Trial focused on higher sample size if needed. Obvious adaption target due to intrinsic SSD uncertainty. Note that more suited to knowable/short follow-up. Could also adaptively...
  • Data Mining: Concepts and Techniques (3rd ed.)  Chapter

    Data Mining: Concepts and Techniques (3rd ed.) Chapter

    Much information (in multiple relations) is needed to judge whether two tuples are similar A user may not be able to provide a good training set It is much easier for a user to specify an attribute as a hint,...
  • Organic Chemistry

    Organic Chemistry

    MEMBERS SHOW A GRADATION OF PHYSICAL PROPERTIES. As successive members of a series differ by a -CH. 2. group, they have successively longer chains. There is a gradual trend in the change of physical properties due to this change in...