**16-825 : Differential Volume Rendering and Neural Radiance Fields**
Student name: Abhishek Pavani
(#) 1. Differential Volume Rendering
In the emission-absorption (EA) model, volumes are described by their appearance (e.g. emission) and geometry (absorption) at every point in 3D space. For part 1 of the project, I have implemented a Differentiable Renderer for EA volumes, which is used in parts 2 and 3. Differentiable renderers are extremely useful for 3D learning problems --- one reason is because they allow you to optimize scene parameters (i.e. perform inverse rendering) from image supervision only!
(##) 1.3. Ray Sampling
| Visualization of Grid | Visualization of Rays |
|-----------------------|------------------------|
| | |
(##) 1.4. Point Sampling
| Points from camera 0 |
|-----------------------|
| |
(##) 1.5. Volume Rendering
| Depth Visualization Box SDF |
|-----------------------|
| |
| |
(#) 2. Optimizing a basic implicit volume
(##) 2.1 and 2.2 Random ray sampling and loss functions
Box center: (0.25, 0.25, 0.00)
Box side lengths: (2.00, 1.50, 1.50)
(##) 2.3 Image to point cloud
| Optimized volume |
|-----------------------|
| |
(##) 3. Optimizing a Neural Radiance Field (NeRF)
| Epoch 10 | Epoch 50 | Epoch 200 | Epoch 250 |
|-----------------------|------------------------|------------------------|---|
| | | ||
(#) NeRF Extras
(##) 4.1 View Dependence
As seen from the results published by the original authors and as mentioned in the paper,
the addition of view dependence helps model specular effects, especially as seen from the "ship in the water" scene published by the original authors.
This increases the training time and generalization quality decreases. As NeRFs become more view-dependent,
they may become more specialized to the specific viewing angles they were trained on, and may not generalize well to other viewpoints.
| Epoch 10 | Epoch 50 | Epoch 200 | Epoch 250 |
|-----------------------|------------------------|------------------------|---|
| | | ||
(##) 4.3 High Resolution Imagery
| Epoch 10 | Epoch 50 | Epoch 200 |
|-----------------------|------------------------|------------------------|
| | ||
Final Result: Epoch 250
I realized that a lot of intricate high frequency details were missing from this radiance field.
So, I changed hyper parameters to recover these. In particular I changed the following hyper-parameters
n_harmonic_functions_xyz: 8
n_harmonic_functions_dir: 2
n_hidden_neurons_xyz: 200
n_hidden_neurons_dir: 200
n_layers_xyz: 6
append_xyz: [3]
n_pts_per_ray: 128
This config helped me recover the following details at epoch 250
I could have recovered more fine grained details in the model by increasing the number of neurons, layers and sampling more points along the ray. But I ran out of GPU memory.
(#) 1. Differential Volume Rendering
In the emission-absorption (EA) model, volumes are described by their appearance (e.g. emission) and geometry (absorption) at every point in 3D space. For part 1 of the project, I have implemented a Differentiable Renderer for EA volumes, which is used in parts 2 and 3. Differentiable renderers are extremely useful for 3D learning problems --- one reason is because they allow you to optimize scene parameters (i.e. perform inverse rendering) from image supervision only!
(##) 1.3. Ray Sampling
| Visualization of Grid | Visualization of Rays |
|-----------------------|------------------------|
|
|
|
(##) 1.4. Point Sampling
| Points from camera 0 |
|-----------------------|
|
|
(##) 1.5. Volume Rendering
| Depth Visualization Box SDF |
|-----------------------|
|
|
|
|
(#) 2. Optimizing a basic implicit volume
(##) 2.1 and 2.2 Random ray sampling and loss functions
Box center: (0.25, 0.25, 0.00)
Box side lengths: (2.00, 1.50, 1.50)
(##) 2.3 Image to point cloud
| Optimized volume |
|-----------------------|
|
|
(##) 3. Optimizing a Neural Radiance Field (NeRF)
| Epoch 10 | Epoch 50 | Epoch 200 | Epoch 250 |
|-----------------------|------------------------|------------------------|---|
|
|
| 
|
|
(#) NeRF Extras
(##) 4.1 View Dependence
As seen from the results published by the original authors and as mentioned in the paper,
the addition of view dependence helps model specular effects, especially as seen from the "ship in the water" scene published by the original authors.
This increases the training time and generalization quality decreases. As NeRFs become more view-dependent,
they may become more specialized to the specific viewing angles they were trained on, and may not generalize well to other viewpoints.
| Epoch 10 | Epoch 50 | Epoch 200 | Epoch 250 |
|-----------------------|------------------------|------------------------|---|
|
|
|
|
Final Result: Epoch 250
I realized that a lot of intricate high frequency details were missing from this radiance field.
So, I changed hyper parameters to recover these. In particular I changed the following hyper-parameters
n_harmonic_functions_xyz: 8
n_harmonic_functions_dir: 2
n_hidden_neurons_xyz: 200
n_hidden_neurons_dir: 200
n_layers_xyz: 6
append_xyz: [3]
n_pts_per_ray: 128
This config helped me recover the following details at epoch 250
I could have recovered more fine grained details in the model by increasing the number of neurons, layers and sampling more points along the ray. But I ran out of GPU memory.