Tue/Thr 4:00pm-5:15pm @ Rapson Hall 43

Description

Multiple cameras are continually capturing our daily events involving social and physical interactions in a form of first person camera (e.g., google glass), cellphone camera, and surveillance camera. Multiview geometry is a core branch in computer vision that studies the 3D spatial relationship between cameras and scenes. This technology is used to localize and plan robots, reconstruct a city, e.g. Rome, from internet photos, and understand human behaviors using body-worn cameras. In this course, we will focus on 1) fundamentals of projective camera geometry; 2) implementation of 3D reconstruction algorithm; and 3) research paper review. The desired outcome of the course is for each student to have his/her own 3D reconstruction algorithm called "structure from motion''. This will cover core mathematics of camera multiview geometry including perspective projection, epipolar geometry, point triangulation, camera resectioning, and bundle adjustment. This geometric concept will be then, in parallel, implemented to directly apply to domain specific research such as robot localization.

Information

Instructor: Hyun Soo Park (hspark at umn dot edu)
Syllabus: pdf
Moodle: https://ay16.moodle.umn.edu/course/view.php?id=13616
Office hour: Tue/Thr 2:00pm-4:00pm (Keller Hall 5-225E)
Prerequisite: Linear Algebra
Textbook: Not required but the following book will be frequently referred:
                   "Multiple View Geometry in Computer Vision", R. Hartley and A. Zisserman
                  

Topic

Single view

Camera model Camera projection matrix Projective line Single view metrology Image transformation Estimation I (Linear algebra) Camera calibration Rotation representation Where am I? (vanishing lines) Where am I? (pnp)

Multiview

Epipolar geometry Fundamental matrix Triangulation and stereo Feature and matching Estimation II (robust modeling) Estimation III (nonlinear optimization) Bundle adjustment I (geometric error) Bundle adjustment II (spare structure and analytic jacobian) Bundle adjustment III (implementation)

Schedule

The course will not directly follow the textbook but students may want to read the related book chapter for deeper understanding.

c: Paper committee

Date	Topic	Slide	Book ch.	Homework	Paper pres.
Jan 17	Computer Vision Introduction	pdf
Jan 18	Camera Model	pdf	Ch. 6
Jan 24	Camera Projection Matrix	pdf	Ch. 6, 7	HW #1 out
Jan 26	Projective Line I	pdf	Ch. 2
Jan 31	Projective Line II	pdf	Ch. 2
Feb 2	Where am I? (Vanishing Point) / Single View Metrology	pdf	Ch. 8	HW #1 due
Feb 7	Image Transformation	pdf	Ch. 2	HW #2 out
Feb 9	Linear Least Squares / Homography Computation	pdf	Ch. 2, A4, A5
Feb 14	Fun with Homography / Camera Calibration	pdf	Ch. A4, A5, 4
Feb 16	Camera Calibration (checkerboard pattern) MATLAB calibration toolbox	pdf		HW #2 due
Feb 21	Where am I? (Homography) / Tour into Photo	pdf		HW #3 out
Feb 23	Rotation Representation	pdf
Feb 28	Rotation Representation / Epipolar Geometry	pdf			Paper selection due
Mar 2	Rotation Representation / Epipolar Geometry	pdf
Mar 7	Epipolar Geometry	pdf
Mar 9	Fundamental Matrix Computation / Where am I? (Relative Pose)	pdf		HW #3 due
Mar 14 & 16	Spring Break
Mar 21	No class			HW #4 out
Mar 23	Where am I? (Relative Pose) / RANSAC	pdf
Mar 28	RANSAC	pdf
Mar 30	Triangulation and Stereo	pdf
Apr 4	Stereo and PnP	pdf		HW #4 due and HW #5 out	Tong Ke pdf c: Cheng Peng
Apr 6	Recitation (Shan Su)	pdf
Apr 11	PnP and Nonlinear Estimation	pdf			Cheng Peng pdf c: Jiawei Mo
Apr 13	Nonlinear Estimation	pdf		HW #5 due	Jingbin Yang pdf c: Sanfer D'souza
Apr 18	Jacobian	pdf		HW #6 out, Data	Tien Do pdf c: Aarti Sundararjan
Apr 20	Jacobian and Bundle Adjustment	pdf			Anushree Jagrawal pdf c: Tong Ke
Apr 25	Bundle Adjustment	pdf			Jiawei Mo pdf c: Tien Do
Apr 27	HW #6 review	pdf			Aarti Sundararjan pdf c: Jingbin Yang
May 2	Tomasi-Kanade Factorization	pdf			Sanfer D'souza pdf c: Anushree Jagrawal
May 4	HW #6 review and Future Computer Vision	pdf		HW #6 due

MATLAB Code

Example codes can be found here.

Paper reading

Format: 20 min presentation and 15+ min Q&A.
Presenter: defending the paper.
Committee: attacking the paper.

Recommended paper
Tomasi and Kanade, Shape and Motion from Image Streams under Orthography: a Factoriaztion Method, IJCV, 1992
Reid and Zisserman, Goal-directed Video Metrology, ECCV, 1996
Zhang, A Flexible New Technique for Camera Calibration, PAMI, 2000
Nister, An Efficient Solution to the Five-Point Relative Pose Problem, PAMI, 2004
Criminisi, Reid, and Zisserman, Single View Metrology, IJCV, 2000
Xiao and Furukawa, Reconstructing the World’s Museum, IJCV, 2014
Izadi et al., KinectFusion: Real-time 3D Reconstruction and Interaction Using a Moving Depth Camera, UIST, 2011

Homework

Due date	Topic	Desciption
Feb 2	Building a camera obscura and Dolly Zoom
Feb 16	Creating a 360 panoramic image
Mar 2	Virtual tour into your photos
Apr 4	Structure from Motion I (Fundamental matrix / RANSAC / Camera Pose)
Apr 18	Structure from Motion II (Triangulation / PnP)
May 4	Structure from Motion III (Bundle adjustment)

Grading policy

90%: 6 programming assignments (15% each)
10%: paper presentation
Late submission: 20% off from each extra late day
For 8980: scalable SfM implementation running +30 images

Scholastic Misconduct

Scholastic misconduct is broadly defined as "any act that violates the right of another student in academic work or that involves misrepresentation of your own work. Scholastic dishonesty includes, (but is not necessarily limited to): cheating on assignments or examinations; plagiarizing, which means misrepresenting as your own work any part of work done by another; submitting the same paper, or substantially similar papers, to meet the requirements of more than one course without the approval and consent of all instructors concerned; depriving another student of necessary course materials; or interfering with another student's work."