Computer Vision News - October 2022

9 Intro to NeRF Let’s start by importing some basic tools and packages. # Download sample data used in the official tiny_nerf example import os import torch import torch.optim as optim import torch.utils.data as datatorch import numpy as np import matplotlib.pyplot as plt # Setup GPU usage device = torch.device("cuda" if torch.cuda.is_available() else "cpu") Nowwe candownload an example dataset used to trainNeRF.We used this link https:// people.eecs.berkeley.edu/~bmild/nerf/tiny_nerf_data.npz , and moved the download- ed file to the current directory. # Load input images, poses, and intrinsics data = np.load("tiny_nerf_data.npz") # Images images = data["images"] # Camera extrinsics (poses) tform_cam2world = data["poses"] tform_cam2world = torch.from_numpy(tform_cam2world).to(device) # Focal length (intrinsics) focal_length = data["focal"] focal_length = torch.from_numpy(focal_length).to(device) # Height and width of each image height, width = images.shape[1:3] # Near and far clipping thresholds for depth values. near_thresh = 2. far_thresh = 6. # Hold one image out (for test). testimg, testpose = images[101], tform_cam2world[101] testimg = torch.from_numpy(np.expand_dims(testimg, axis=0)).to(devi- ce) testpose = testpose.expand([1,4,4]) # Map images to device images = torch.from_numpy(images[:100, ..., :3]).to(device) plt.imshow(testimg[0].detach().cpu().numpy()) plt.show() Now we can install the repo through pip and import the model. The NeRF model, as de- scribed in the original paper, is just a multilayer perceptron that takes encoded 3D points and directions (x, d) as inputs and returns RGB values (c, s ). Quite a simple architecture that is helped by other components: a positional encoder to extract high frequency variations, a differentiable volume renderer to convert the RGBA output points in 3D (c, s ) to an ima�- ge, a stratified sampling , and a hierarchical volume sampling . These elements are already integrated in this package, so we can just easily import the full model, after downloading nerf-pytorch through pip.