Spring 2017 CSCI 5980/8980
Multiview 3D Geometry in Computer Vision
Tue/Thr 4:00pm5:15pm @ Rapson Hall 43

Description
Multiple cameras are continually capturing our daily events involving social and physical interactions in a form of first person camera (e.g., google glass), cellphone camera, and surveillance camera. Multiview geometry is a core branch in computer vision that studies the 3D spatial relationship between cameras and scenes. This technology is used to localize and plan robots, reconstruct a city, e.g. Rome, from internet photos, and understand human behaviors using bodyworn cameras. In this course, we will focus on 1) fundamentals of projective camera geometry; 2) implementation of 3D reconstruction algorithm; and 3) research paper review. The desired outcome of the course is for each student to have his/her own 3D reconstruction algorithm called "structure from motion''. This will cover core mathematics of camera multiview geometry including perspective projection, epipolar geometry, point triangulation, camera resectioning, and bundle adjustment. This geometric concept will be then, in parallel, implemented to directly apply to domain specific research such as robot localization.

Information
Instructor: Hyun Soo Park (hspark at umn dot edu)
Syllabus: pdf
Moodle: https://ay16.moodle.umn.edu/course/view.php?id=13616
Office hour: Tue/Thr 2:00pm4:00pm (Keller Hall 5225E)
Prerequisite: Linear Algebra
Textbook: Not required but the following book will be frequently referred:
"Multiple View Geometry in Computer Vision", R. Hartley and A. Zisserman

Topic
Single view 
Camera model
Camera projection matrix
Projective line
Single view metrology
Image transformation
Estimation I (Linear algebra)
Camera calibration
Rotation representation
Where am I? (vanishing lines)
Where am I? (pnp)

Multiview 
Epipolar geometry
Fundamental matrix
Triangulation and stereo
Feature and matching
Estimation II (robust modeling)
Estimation III (nonlinear optimization)
Bundle adjustment I (geometric error)
Bundle adjustment II (spare structure and analytic jacobian)
Bundle adjustment III (implementation)


Schedule
The course will not directly follow the textbook but students may want to read the related book chapter for deeper understanding.
c: Paper committee
Date 
Topic 
Slide 
Book ch. 
Homework 
Paper pres. 
Jan 17 
Computer Vision Introduction 
pdf  


Jan 18 
Camera Model 
pdf 
Ch. 6   
Jan 24 
Camera Projection Matrix 
pdf 
Ch. 6, 7 
HW #1 out  
Jan 26 
Projective Line I 
pdf 
Ch. 2   
Jan 31 
Projective Line II 
pdf 
Ch. 2   
Feb 2 
Where am I? (Vanishing Point) / Single View Metrology 
pdf  Ch. 8 
HW #1 due  
Feb 7 
Image Transformation 
pdf  Ch. 2 
HW #2 out  
Feb 9 
Linear Least Squares / Homography Computation 
pdf 
Ch. 2, A4, A5   
Feb 14 
Fun with Homography / Camera Calibration 
pdf  Ch. A4, A5, 4 
 
Feb 16 
Camera Calibration (checkerboard pattern) MATLAB calibration toolbox 
pdf  
HW #2 due  
Feb 21 
Where am I? (Homography) / Tour into Photo 
pdf  
HW #3 out  
Feb 23 
Rotation Representation 
pdf  
 
Feb 28 
Rotation Representation / Epipolar Geometry 
pdf  
 Paper selection due 
Mar 2 
Rotation Representation / Epipolar Geometry 
pdf  
 
Mar 7 
Epipolar Geometry 
pdf  
 
Mar 9 
Fundamental Matrix Computation / Where am I? (Relative Pose) 
pdf  
HW #3 due  
Mar 14 & 16 
Spring Break 
 
 
Mar 21 
No class 
 
HW #4 out  
Mar 23 
Where am I? (Relative Pose) / RANSAC 
pdf  
 
Mar 28 
RANSAC 
pdf  
 
Mar 30 
Triangulation and Stereo 
pdf  
 
Apr 4 
Stereo and PnP 
pdf  
HW #4 due and HW #5 out  Tong Ke pdf c: Cheng Peng 
Apr 6 
Recitation (Shan Su) 
pdf  
 
Apr 11 
PnP and Nonlinear Estimation 
pdf  
 Cheng Peng pdf c: Jiawei Mo 
Apr 13 
Nonlinear Estimation 
pdf  
HW #5 due  Jingbin Yang pdf c: Sanfer D'souza 
Apr 18 
Jacobian 
pdf  
HW #6 out, Data  Tien Do pdf c: Aarti Sundararjan 
Apr 20 
Jacobian and Bundle Adjustment 
pdf  
 Anushree Jagrawal pdf c: Tong Ke 
Apr 25 
Bundle Adjustment 
pdf  
 Jiawei Mo pdf c: Tien Do 
Apr 27 
HW #6 review 
pdf  
 Aarti Sundararjan pdf c: Jingbin Yang 
May 2 
TomasiKanade Factorization 
pdf  
 Sanfer D'souza pdf c: Anushree Jagrawal 
May 4 
HW #6 review and Future Computer Vision 
pdf  
HW #6 due  

MATLAB Code
Example codes can be found here.

Paper reading
Format: 20 min presentation and 15+ min Q&A.
Presenter: defending the paper.
Committee: attacking the paper.
Recommended paper 



Tomasi and Kanade, Shape and Motion from Image Streams under Orthography: a Factoriaztion Method, IJCV, 1992




Reid and Zisserman, Goaldirected Video Metrology, ECCV, 1996




Zhang, A Flexible New Technique for Camera Calibration, PAMI, 2000




Nister, An Efficient Solution to the FivePoint Relative Pose Problem, PAMI, 2004




Criminisi, Reid, and Zisserman, Single View Metrology, IJCV, 2000




Xiao and Furukawa, Reconstructing the World’s Museum, IJCV, 2014




Izadi et al., KinectFusion: Realtime 3D Reconstruction and Interaction
Using a Moving Depth Camera, UIST, 2011





Homework

Grading policy
90%: 6 programming assignments (15% each)
10%: paper presentation
Late submission: 20% off from each extra late day
For 8980: scalable SfM implementation running +30 images

Scholastic Misconduct
Scholastic misconduct is broadly defined as "any act that violates the right of another student in academic work or that involves misrepresentation of your own work. Scholastic dishonesty includes, (but is not necessarily limited to): cheating on assignments or examinations; plagiarizing, which means misrepresenting as your own work any part of work done by another; submitting the same paper, or substantially similar papers, to meet the requirements of more than one course without the approval and consent of all instructors concerned; depriving another student of necessary course materials; or interfering with another student's work."
