Structure from Motion

In this project, I have implemented algorithms for doing structure from motion. The project begins with reconstruction from triangulation using 2 input images(two view reconstruction) and then goes onto explore incremental sfm(a technique used by most modern SfM pipelines) and finally I explored COLMAP (an off the shelf application to do SfM)

Details of the project can be found here

Neural Radiance Fields

In this project, I have implemented differentiable volume rendering. I used those concepts to perform novel view synthesis using Neural Radiance Fields (NeRFs)

Details of the project can be found here

Neural Surfaces

In this project, I have implemented sphere tracing and MLP to extract neural surfaces. There are two main papers I have approached here 1. VolSDF 2. NeUS. I have also explored neural relighting and have implemented the phong reflection model.

Details of the project can be found here

Epipolar Geometry and 3D reconstruction

In this project, I have implemented epipolar geometry constraints and understood its importance in generating 3D structure from 2D images. I have also implemented bundle adjustment which is used in SfM pipelines to obtain accurate camera poses

Details of the project can be found here

Point Cloud Classification and Segmentation

In this project, I have implemented the pointNet Architecture to classify and segment point clouds.

Details of the project can be found here

Single View Reconstruction

In this project, I used concepts like vanishing points and perspective n-point to reconstruct 3D of a scene given a single view. Unlike SfM from triangulation which requires multiple views, this project aims to use the geometry behind planes and reconstruct 3D.

Details of the project can be found here

Image Rectification

In this project, I used the geometric principles from perspective projection and implemented different types of image rectification(affine and metric). Implementation of image rectification improved my understanding of lines and points at infinity, vanishing points.

Details of the project can be found here

Recovering Pose from Panorama style videos

This is a course project done in the course 16-822: Geometry Based Methods in Vision. It is difficult to recover camera pose and structure of a scene from a video that has rotation dominant motion. In this project I have explored concepts like spherical epipolar geometry, feature tracking using Lucas Kanade Optical flow pyramids, GRIC score computation(geometric robust information criterion) and rotation averaging. We wrote a detailed paper which outlines our approach and hyperparameters we used. We empirically validate our approach using real world data and have made our repository public

Details of the project can be found here

Planar Homography

In this project, I implemented 2D planar homography. Given enough annotations, we can transform any source plane to fit the target(destination) plane. I generated a simple augmented reality output by overlaying a movie on top of a book.

Details of the project can be found in the homography section of this page

Photometric Stereo

In this project, I reconstructed 3D surface, given a source image and multiple lighting conditions using physics based methods. I implemented the n dot l lighting model and understood concepts like albedo and surface normals

Pytorch 3D

In this project, I worked on the basics of using the pytorch3D API. I worked on rendering point clouds, implicit, parametric surfaces. I also implement sampling point clouds from meshes. More details on the project can be found here

Image Registration

In this project, I have written algorithms to align the channels of a RGB image. I used similarity metrics like SSD(sum of squared differences), NCC(Normalized Cross Correlation, ZNCC(Zero Normalized Cross Correlation). I tested my algorithms on the Prokudin-Gorskii photo collection. I also tested the algorithm on imagery taken from a hubble telescope. More details about the project can be found here

Defect Identification using Deep Networks

The goal of this project was to identify defects in a wind turbine blade using object detection methods. For this project, I used Faster R-CNN network with a ResNet architecture to detect 9 different types of defects in images. The images of these wind turbine blades were captured from a drone. To make the processing of these images faster, I used a multi-scale pyramid network, to cater for the varying size of defects since the distance of the captured images may vary.

As a result of this implementation we got the defect identification time from 1 day to 15 mins. This helped us deliver results faster to our clients.

Semantic Segmentation in high resolution aerial imagery

In this project, I worked on generating semantic segmentation masks for buildings in high resolution aerial imagery. This is useful for creating maps from captured drone imagery.

To create a dataset, I used high resolution 4K aerial imagery with 4cm GSD and created ground truth segmentation masks using a GIS software like QGIS. I then preprocessed these images and fed this into a UNet architecture and treated it as a supervised learning problem.

Understanding and dispensing ingredients for smart robotic cooking

In this project, we have developed an end-end robotic system capable of making food recipes from scratch from a single click. The system is capable of identifying and validating ingredients kept in container, dispense a wide variety of ingredients including solids, liquids and powders.

More details about the project can be found here

Lego Kitting

In this project, we worked on developing a robot that could sort different types of lego blocks based on color. We used image centroid and image moments to identify the camera blocks and then generated transforms from the camera frame to the robot end-effector frame to control the robot inorder to pick up the block of choice. The robot stops once it has sorted out all the blocks based on color.

More details about the project can be found in this report.

A video link to our system in action can be found here

E-VTOL UAV with swappable payloads

The goal of this project was to create an e-VTOL UAV with swappable payloads. I designed and fabricated this VTOL from scratch. The aircraft can stay afloat for 1.5 hrs while carrying 1kg payload onboard. It is controlled by ardupilot firmware and uses li-ion batteries to power the aircraft. The aircraft was made to survey/map large areas without landing.

Image Registration

In this project, I have used supervised learning approaches to reconstruct 3D of an object using a single image. I explored different network architectures to output different 3D representation. I have done an empirical study to see which 3D representation is kind of useful when inferring 3D objects from single images.

More details about the project can be found here

Real Time Controllable Motion Transition for Characters

I implemented this paper from scratch during my internship at Apple. The idea is to use a neural network to interpolate between 2 animation keyframes using a generative model.

The input data here are .bvh files, from which we extract the joint position and rotations for each frame. We feed the pose information to the encoder of the conditional-VAE to generate a latent space. We then sample from this latent space and pass it through a decoder which has multiple expert networks. The gating scheme allows for the network to choose the right expert from the decoder.

The VAE network is trained using multiple loss terms like reconstruction loss, KL Divergence, Bone Length Loss, Footskating Loss, L1 Rotation loss. These loss functions constrain the generated pose to some acceptable human pose.

Next we train a transition sampler, which takes in the current frame and the target frame and outputs the next frame in the sequence. The network architecture used for this has been mentioned in the paper.

GANs and Diffusion Model

In this project, I implemented 2 different GAN network architectures, DCGAN and CycleGAN. I tested my implementation over 2 different datasets and have reported my findings. I also implemented the DDPM model from scratch to recover images from noise. More details about the project can be found here

Gradient Domain Fusion

In this project, I explored gradient domain fusion techniques. In particular, I implemented poisson blending, to blend 2 images. The goal is to seamlessly blend an object or texture from a source image into a target image, without creating noticeable seams. The approach is to preserve the gradient of the source region while ignoring the overall intensity, using a least squares problem to solve for new intensity values within the source region. The method presented is one example of a more general set of gradient-domain processing techniques, with applications including blending, tone-mapping, and non-photorealistic rendering. The project involves implementing the Poisson blending technique and potentially exploring additional techniques as well. The reason why “Poisson Blending” produces a more realistic composition than simply pasting two similarly colored images together is that our visual system is more responsive to changes in contrast rather than variations in intensity. More details about the project can be found here

Neural Style Transfer

In this project, I explored neural style transfer. The idea is to have a style image and a content image and then we try to copy the style from the style image while keeping the content in the content image the same. More details about the project can be found here