**Assignment 4: SfM Reconstruction**
Student name: Abhishek Pavani
(##) LATE DAYS : 1
(#) **RUN CODE**
(##) To run code for Q1 and see the 3D reconstruction, run
`python main.py --question='q1'`
(##) To run code for Q2 and see incremental SfM, run
`python main.py --question='q2' --show_plot1=True --show_plot2=True --show_plot3=True`
(#) Q1 Baseline Reconstruction
(##) RESULTS
| Reconstructed view 1| Reconstructed view 2 |
| ----------- | ----------- | --- |
|||
Rotation
R = $\begin{bmatrix}
0.99112584 & 0.0115062 & 0.13242799 \\
0.02494454 & 0.96244867 & -0.27031524 \\
-0.13056544 & 0.27121977 & 0.95362073
\end{bmatrix}$
Translation
t = $\begin{bmatrix}
0.02041602 \\
0.17785712 \\
0.98384452
\end{bmatrix}$
(##) BRIEF EXPLANATION OF IMPLEMENTATION
1. Given the points, first step is to normalize them
2. Normalize these points using the transforamtion matrix below
T= $\begin{bmatrix}
s & 0 & -s*x_{0} \\
0 & s & -s*y_{0} \\
0 & 0 & 1
\end{bmatrix}$
3. Using these normalized coordinates, construct a matrix to recover the fundamental matrix. This can be done using the epipolar constraint $x'^{T} F x =0$
4. Using the above equation we can reduce it to the form Af = 0, where f contains all elements of the fundamental matrix and A is constructed as follows
A = $\begin{bmatrix}
x_{1}*x_{2} & x_{2}*y_{1} & x_{2} & y_{2}*x_{1} & y_{1}*y_{2} & y_{2} & x_{1} & y_{1} & 1
\end{bmatrix}$
Each point-point correspondence will give one such equation and we need 8 such equations to solve for all elements of F
Note: here I used all correspondences as suggested in the piazza post instead of using only 8 points.
5. Now perform SVD to recover all elements of F
6. Once we get F, we first unnormilize the F matrix using $F_{unnormalized} = T'^{T} F_{normalized} T$ . After this, we set the last element to one and also enforce rank 2 constraint. We can do this by setting $\sigma_{3} = 0$
7. After we enforce rank 2 constraint, we again compute F by multiplying u, s and v
8. Once you compute F using 8 point algorithm, E can be estimated using the following expression, $E = K'^{T} F K$
9. This E can then be decomposed using FVD to get R and T. The decomposition gives us $P_{2}$ with 4 ambiguities listed as follows
10. We then traingulate these points in 3D and then choose a camera for which all depth is positive and reprojection error is minimum
11. Using the inlier points and the chosen P2, we again reconstruct to get the final reconstruction which can be seen in the results section.
(#) Q2 Incremental SfM
(##) RESULTS
| using cameras 1 and 2| After adding camera 3 | After adding camera 4|
| ----------- | ----------- | --- |------------|
||||
(##) Camera 3
Rotation
R = $\begin{bmatrix}
1.956716e-04 & 1.64307947e-06 & 3.50304655e-05 \\
-1.168836e-06 & 1.98351631e-04 & 2.96112889e-06 \\
-1.59933161e-04 & 5.69207317e-06 & 2.05634027e-04
\end{bmatrix}$
Translation
t = $\begin{bmatrix}
-1.04219233e-06 \\
-1.02793358e-06\\
1.96908585e-03
\end{bmatrix}$
(##) Camera 4
Rotation
R = $\begin{bmatrix}
-2.31056438e-04 & -2.78899116e-06 & 2.71722445e-05 \\
8.92325482e-07 & -2.90867884e-04 & -5.9521303e-06 \\
2.10351243e-04 & 4.30808792e-06 & -3.43513724e-04
\end{bmatrix}$
Translation
t = $\begin{bmatrix}
7.07784881e-07 \\
1.67087700e-06 \\
-2.019497961e-03
\end{bmatrix}$
(##) BRIEF EXPLANATION OF IMPLEMENTATION
1. For camera pair 12, choose cam 1 and 2 and follow all the steps that were done previously.
2. Then choose camera pair12 and camera pair 23. For both pairs, cam2 is common. So choose points which are common in cam2 for pairs 12 and 23, and find their corresponding 3D point. Then use PnP to solve for the camera projection matrix $P_{3}$. Use this camera projection $P_{3}$, and use $P_{2}$ to triangulate all points in cameras 2 and 3. This gives us additional triangulated points.
3. Do the same thing for $P_{4}$ and get additional points.
4. Use multiple pair of images while solving for PnP. This gives us better reconstruction results and less error in 3D points
(#) Q3 Reconstruct your own scene
(##) RESULTS
(##) 1. Nike Air Shoe
For this reconstruction I used over 300 images, few of which are shown below
(##) Input images
| Input views| Input views |
| ----------- | ----------- | --- |
|||
|||
|||
(##) Output with extrinsics
(##) 2. Lemoncake
For this reconstruction I used over 250 images, few of which are shown below
(##) Input images
| Input views| Input views |
| ----------- | ----------- | --- |
|||
|||
|||
(##) 3. A city in switzerland
For this reconstruction I used over 1000 images, few of which are shown below
(##) Input images
| Input views| Input views |
| ----------- | ----------- | --- |
|||
|||
|||
(##) Output with extrinsics
(#) Q3(B) Stress test the hyperparameters of COLMAP
(##) 1. What happens if we reduce number of input images?
When you reduce the number of images, we get lower quality reconstruction since the number of correspondences between images are reduced and hence the triangulated 3D points are less in number. Additionally, if overlap between images is not sufficient then reconstruction looks broken. To ensure accurate reconstruction we need to capture images from all sides of the object with sufficient overlap. This is evident from the reconstruction above of Nike shoe and Lemon cake.
(###) Reconstruction of a city with less images in the middle leading to poor quality reconstructions in the area
(##) 2. Under what scenario and conditions does the reconstruction pipeline breaks?
Reconstruction pipeline breaks when it is asked to reconstruct thin objects. It also breaks when asked to reconstruct transparent surfaces (eg. glass water). The pipeline breaks when asked to reconstruct extreme dark color like black (eg human hair). As seen from the gif below. I reconstructed an entire city and wherever there were thin surfaces like electric poles, they would not get reconstructed properly. The glass windows of the buildings showed incorrect reconstruction because of depth ambiguity associated with transparent surfaces. Moving objects like cars and people arent also reconstructed properly because they are changing their position in time.
(###) Dense Reconstruction of a Mumbai city using drone images
(#) **RUN CODE**
(##) To run code for Q1 and see the 3D reconstruction, run
`python main.py --question='q1'`
(##) To run code for Q2 and see incremental SfM, run
`python main.py --question='q2' --show_plot1=True --show_plot2=True --show_plot3=True`
(#) Q1 Baseline Reconstruction
(##) RESULTS
| Reconstructed view 1| Reconstructed view 2 |
| ----------- | ----------- | --- |
|
|
|
Rotation
R = $\begin{bmatrix}
0.99112584 & 0.0115062 & 0.13242799 \\
0.02494454 & 0.96244867 & -0.27031524 \\
-0.13056544 & 0.27121977 & 0.95362073
\end{bmatrix}$
Translation
t = $\begin{bmatrix}
0.02041602 \\
0.17785712 \\
0.98384452
\end{bmatrix}$
(##) BRIEF EXPLANATION OF IMPLEMENTATION
1. Given the points, first step is to normalize them
2. Normalize these points using the transforamtion matrix below
T= $\begin{bmatrix}
s & 0 & -s*x_{0} \\
0 & s & -s*y_{0} \\
0 & 0 & 1
\end{bmatrix}$
3. Using these normalized coordinates, construct a matrix to recover the fundamental matrix. This can be done using the epipolar constraint $x'^{T} F x =0$
4. Using the above equation we can reduce it to the form Af = 0, where f contains all elements of the fundamental matrix and A is constructed as follows
A = $\begin{bmatrix}
x_{1}*x_{2} & x_{2}*y_{1} & x_{2} & y_{2}*x_{1} & y_{1}*y_{2} & y_{2} & x_{1} & y_{1} & 1
\end{bmatrix}$
Each point-point correspondence will give one such equation and we need 8 such equations to solve for all elements of F
Note: here I used all correspondences as suggested in the piazza post instead of using only 8 points.
5. Now perform SVD to recover all elements of F
6. Once we get F, we first unnormilize the F matrix using $F_{unnormalized} = T'^{T} F_{normalized} T$ . After this, we set the last element to one and also enforce rank 2 constraint. We can do this by setting $\sigma_{3} = 0$
7. After we enforce rank 2 constraint, we again compute F by multiplying u, s and v
8. Once you compute F using 8 point algorithm, E can be estimated using the following expression, $E = K'^{T} F K$
9. This E can then be decomposed using FVD to get R and T. The decomposition gives us $P_{2}$ with 4 ambiguities listed as follows
10. We then traingulate these points in 3D and then choose a camera for which all depth is positive and reprojection error is minimum
11. Using the inlier points and the chosen P2, we again reconstruct to get the final reconstruction which can be seen in the results section.
(#) Q2 Incremental SfM
(##) RESULTS
| using cameras 1 and 2| After adding camera 3 | After adding camera 4|
| ----------- | ----------- | --- |------------|
|
|
|
|
(##) Camera 3
Rotation
R = $\begin{bmatrix}
1.956716e-04 & 1.64307947e-06 & 3.50304655e-05 \\
-1.168836e-06 & 1.98351631e-04 & 2.96112889e-06 \\
-1.59933161e-04 & 5.69207317e-06 & 2.05634027e-04
\end{bmatrix}$
Translation
t = $\begin{bmatrix}
-1.04219233e-06 \\
-1.02793358e-06\\
1.96908585e-03
\end{bmatrix}$
(##) Camera 4
Rotation
R = $\begin{bmatrix}
-2.31056438e-04 & -2.78899116e-06 & 2.71722445e-05 \\
8.92325482e-07 & -2.90867884e-04 & -5.9521303e-06 \\
2.10351243e-04 & 4.30808792e-06 & -3.43513724e-04
\end{bmatrix}$
Translation
t = $\begin{bmatrix}
7.07784881e-07 \\
1.67087700e-06 \\
-2.019497961e-03
\end{bmatrix}$
(##) BRIEF EXPLANATION OF IMPLEMENTATION
1. For camera pair 12, choose cam 1 and 2 and follow all the steps that were done previously.
2. Then choose camera pair12 and camera pair 23. For both pairs, cam2 is common. So choose points which are common in cam2 for pairs 12 and 23, and find their corresponding 3D point. Then use PnP to solve for the camera projection matrix $P_{3}$. Use this camera projection $P_{3}$, and use $P_{2}$ to triangulate all points in cameras 2 and 3. This gives us additional triangulated points.
3. Do the same thing for $P_{4}$ and get additional points.
4. Use multiple pair of images while solving for PnP. This gives us better reconstruction results and less error in 3D points
(#) Q3 Reconstruct your own scene
(##) RESULTS
(##) 1. Nike Air Shoe
For this reconstruction I used over 300 images, few of which are shown below
(##) Input images
| Input views| Input views |
| ----------- | ----------- | --- |
|
|
|
|
|
|
|
|
|
(##) Output with extrinsics
(##) 2. Lemoncake
For this reconstruction I used over 250 images, few of which are shown below
(##) Input images
| Input views| Input views |
| ----------- | ----------- | --- |
|
|
|
|
|
|
|
|
|
(##) 3. A city in switzerland
For this reconstruction I used over 1000 images, few of which are shown below
(##) Input images
| Input views| Input views |
| ----------- | ----------- | --- |
|
|
|
|
|
|
|
|
|
(##) Output with extrinsics
(#) Q3(B) Stress test the hyperparameters of COLMAP
(##) 1. What happens if we reduce number of input images?
When you reduce the number of images, we get lower quality reconstruction since the number of correspondences between images are reduced and hence the triangulated 3D points are less in number. Additionally, if overlap between images is not sufficient then reconstruction looks broken. To ensure accurate reconstruction we need to capture images from all sides of the object with sufficient overlap. This is evident from the reconstruction above of Nike shoe and Lemon cake.
(###) Reconstruction of a city with less images in the middle leading to poor quality reconstructions in the area
(##) 2. Under what scenario and conditions does the reconstruction pipeline breaks?
Reconstruction pipeline breaks when it is asked to reconstruct thin objects. It also breaks when asked to reconstruct transparent surfaces (eg. glass water). The pipeline breaks when asked to reconstruct extreme dark color like black (eg human hair). As seen from the gif below. I reconstructed an entire city and wherever there were thin surfaces like electric poles, they would not get reconstructed properly. The glass windows of the buildings showed incorrect reconstruction because of depth ambiguity associated with transparent surfaces. Moving objects like cars and people arent also reconstructed properly because they are changing their position in time.
(###) Dense Reconstruction of a Mumbai city using drone images
