zakialawi02 / helen Goto Github PK

Helen is a fully differentiable dense SLAM framework. It provides a repository of differentiable building blocks for a dense SLAM system, such as differentiable nonlienar least squares solvers, differentiable ICP (iterative closest point) techniques, differentiable raycastring modules, and differentiable mapping / fusion blocks. One can use these blocks to construct SLAM systems that allow gradients to flow all the way from the outputs of the system (map, trajectory) to the inputs (raw color / depth images, parameters, calibration)

SLAM is technological mapping method that allows robots and other autonomous vehicles to build a map and localize itself on that map at the same time. using a wide range of algorithm, computation and other sensory data, SLAM software system allow a robot or other vehicle like a drone or self-driving-car to ploat a course through an unfamiliar environment while simultaneously identifying its own location within that environment

• point clouds living room
• RGBDimages living room
• helen pointfusion
• helen pointclouds

the beginnings of differentiable visual odometry can be traced back to seminal Lucas-Kanade iterative matching algorithm. apply the Lucas-Kanade algorithm to perform real-time dense visual odometry. Their system is differentiable, and has been extensively used for self-superviesd depth and motion estimation. coupled with the success of spatial transformer networks (STN), they can be used to perform dense visual odometry.

hoewever, extending differentiability beyodn the two view case (frame-frame alingment) is not starightforward. global consistency necessitates fusing measuerements from live frames into a global model (model-frame alignment) which is not trivially differentiable

some approaches hava recently propesed to learn the optimization of nonlinera least squares objective functions. This is motivated by the fact that similiar cost functions have similar cost functions have similar loss landscape, and learning methods can help converge faster, or potientially to better

in BA-Net, the authors learn to predict the damping coefficient of the levenberg-marquardt optimizer, while in LS-Net, the authors entirely replace the levenberg marquad optimizer by an LSTM network. that predicts update steps. In GN-Net, a differentiable version of the Gauss-newton loss is used to show better bustness to wheather conditions.

nodes in red represent variables, nodes in blue reresents operations on variables. Edges represent adata flow. this graph computes the function 3(xy + z). dashed lines indicate (loca, i.e, per-node) gredients in the backward pass.

for the requirements you can see on requirements.txt

install from github as pip package

pip install git+https://github.com/slowy/helen

to verify helen has success been built, test on python interperter and check by

import helen as helen_slam
print(helen_slam.__version__)

rgbd_images = RGBDimages(color, depths, intrunsics)
slam = PointFusion()
pointclouds, recorvered_poses = slam(rgbd_images)
pointclouds.poltly(0).show()