Skip to content

Commit 8b59dff

Browse files
UnderscoreAsteriskalalek
authored andcommitted
Merge pull request #2161 from UnderscoreAsterisk:dynafu
DynamicFusion Implementation * Add new nodes from points * Addition of new node in the field * Warp nodes demo * Add newline before { * Remove 3rd party flann dependency * Regularisation Heirarchy * Correct node radius * Change default growth rate * New node position = centroid of the neighbourhood * Enlarge nodes while paused * Dynafu TSDF * TSDF demo * Avoid double calc and adjust initial tsdf weight * Fix bug that caused some voxels to disappear * getNodePos helper * Remove USE_INTRINSIC check * Correct RT avg calculation and remove redundant knn calc * Slight perf improvements * Use LinearIndex * Debug calculations * set all nodes' RT * Various bug fixes * Separate camera and warpfield * Add dynafu documentation * Adhere to coding style * Add dynafu unit test * update demo * Merge pull request #2133 from savuor:kinfu_truncate KinectFusion: several updates (#2133) * truncateThreshold added * demo updated for RealSense * first Kinect2, then RealSense * more distance-based parameters * Remove trailing whitespaces * Replace bool vector with array * create findNeighbours in WarpField * Maintain nodesPos matrix in WarpField * Fix warnings on Windows build * Remove cameraPose from WarpField * Use AutoBuffer * Marching Cubes * Fix MC * Split mesh vertices & edges * Change Mat types in MC * OpenGL rendering * Check for HAVE_OPENGL * Error handling in case HAVE_OPENGL is not defined * Replace Mat_ with std::vector inside marchCubes * Parallelise marching cubes * Fix warpfield and estimate depth inside DynaFuImpl::updateT() * Linearise depth and use points/normals from MC * Don't test dynafu without OpenGL support * Analytical calculation of Jacobian matrices * Add details about rotation and differentiate graph terms in J_r * Use derivative of T^-1 from the tutorial * Remove L2 norm from reg term * Use chain rule to differentiate data term * Markdown * Fix markdown * Replace MD file by HTML * Change the data term expression * Calculate J_d using vector identities * Rasterize vertex and normals * Apply warpfield before rendering * Add CV_UNUSED for normImage to fix warning * Render float image instead of uint8 * Implement ICP data term and other changes: 1. add option to applyWarp to normals 2. add option to `fetchPointNormals` to return points in voxel coordinates 3. Fix: Use voxel coordinates to update WarpField * Fix non-OpenGL build * Intialise newly discovered node transforms with DQB * Fix data term * Change data term normal and add kinfu-like distance/angle checks * Implement regularisation * Fix warnings * Credit authors of DQB and table for MC * cast size_t to int to supress warning * Correct regularisation and add normal equation set up * Delete html * Dynafu unit test
1 parent 6a0e9fd commit 8b59dff

16 files changed

+4382
-0
lines changed

modules/rgbd/doc/dynafu_ICP.ipynb

Lines changed: 419 additions & 0 deletions
Large diffs are not rendered by default.

modules/rgbd/doc/rgbd.bib

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,3 +16,10 @@ @inproceedings{kinectfusion
1616
chapter = {},
1717
isbn = {978-1-4503-0716-1},
1818
}
19+
@inproceedings{dynamicfusion,
20+
author = {Newcombe, Richard A. and Fox, Dieter and Seitz, Steven M.},
21+
title = {DynamicFusion: Reconstruction and Tracking of Non-Rigid Scenes in Real-Time},
22+
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
23+
month = {June},
24+
year = {2015}
25+
}

modules/rgbd/include/opencv2/rgbd.hpp

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
#include "opencv2/rgbd/linemod.hpp"
1313
#include "opencv2/rgbd/depth.hpp"
1414
#include "opencv2/rgbd/kinfu.hpp"
15+
#include "opencv2/rgbd/dynafu.hpp"
1516

1617

1718
/** @defgroup rgbd RGB-Depth Processing
Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
// This file is part of OpenCV project.
2+
// It is subject to the license terms in the LICENSE file found in the top-level directory
3+
// of this distribution and at http://opencv.org/license.html
4+
5+
// This code is also subject to the license terms in the LICENSE_KinectFusion.md file found in this module's directory
6+
7+
#ifndef __OPENCV_RGBD_DYNAFU_HPP__
8+
#define __OPENCV_RGBD_DYNAFU_HPP__
9+
10+
#include "opencv2/core.hpp"
11+
#include "opencv2/core/affine.hpp"
12+
13+
namespace cv {
14+
namespace dynafu {
15+
16+
struct CV_EXPORTS_W Params
17+
{
18+
/** @brief Default parameters
19+
A set of parameters which provides better model quality, can be very slow.
20+
*/
21+
CV_WRAP static Ptr<Params> defaultParams();
22+
23+
/** @brief Coarse parameters
24+
A set of parameters which provides better speed, can fail to match frames
25+
in case of rapid sensor motion.
26+
*/
27+
CV_WRAP static Ptr<Params> coarseParams();
28+
29+
/** @brief frame size in pixels */
30+
CV_PROP_RW Size frameSize;
31+
32+
/** @brief camera intrinsics */
33+
CV_PROP Matx33f intr;
34+
35+
/** @brief pre-scale per 1 meter for input values
36+
37+
Typical values are:
38+
* 5000 per 1 meter for the 16-bit PNG files of TUM database
39+
* 1000 per 1 meter for Kinect 2 device
40+
* 1 per 1 meter for the 32-bit float images in the ROS bag files
41+
*/
42+
CV_PROP_RW float depthFactor;
43+
44+
/** @brief Depth sigma in meters for bilateral smooth */
45+
CV_PROP_RW float bilateral_sigma_depth;
46+
/** @brief Spatial sigma in pixels for bilateral smooth */
47+
CV_PROP_RW float bilateral_sigma_spatial;
48+
/** @brief Kernel size in pixels for bilateral smooth */
49+
CV_PROP_RW int bilateral_kernel_size;
50+
51+
/** @brief Number of pyramid levels for ICP */
52+
CV_PROP_RW int pyramidLevels;
53+
54+
/** @brief Resolution of voxel space
55+
56+
Number of voxels in each dimension.
57+
*/
58+
CV_PROP_RW Vec3i volumeDims;
59+
/** @brief Size of voxel in meters */
60+
CV_PROP_RW float voxelSize;
61+
62+
/** @brief Minimal camera movement in meters
63+
64+
Integrate new depth frame only if camera movement exceeds this value.
65+
*/
66+
CV_PROP_RW float tsdf_min_camera_movement;
67+
68+
/** @brief initial volume pose in meters */
69+
Affine3f volumePose;
70+
71+
/** @brief distance to truncate in meters
72+
73+
Distances to surface that exceed this value will be truncated to 1.0.
74+
*/
75+
CV_PROP_RW float tsdf_trunc_dist;
76+
77+
/** @brief max number of frames per voxel
78+
79+
Each voxel keeps running average of distances no longer than this value.
80+
*/
81+
CV_PROP_RW int tsdf_max_weight;
82+
83+
/** @brief A length of one raycast step
84+
85+
How much voxel sizes we skip each raycast step
86+
*/
87+
CV_PROP_RW float raycast_step_factor;
88+
89+
// gradient delta in voxel sizes
90+
// fixed at 1.0f
91+
// float gradient_delta_factor;
92+
93+
/** @brief light pose for rendering in meters */
94+
CV_PROP Vec3f lightPose;
95+
96+
/** @brief distance theshold for ICP in meters */
97+
CV_PROP_RW float icpDistThresh;
98+
/** angle threshold for ICP in radians */
99+
CV_PROP_RW float icpAngleThresh;
100+
/** number of ICP iterations for each pyramid level */
101+
CV_PROP std::vector<int> icpIterations;
102+
103+
/** @brief Threshold for depth truncation in meters
104+
105+
All depth values beyond this threshold will be set to zero
106+
*/
107+
CV_PROP_RW float truncateThreshold;
108+
};
109+
110+
/** @brief DynamicFusion implementation
111+
112+
This class implements a 3d reconstruction algorithm as described in @cite dynamicfusion.
113+
114+
It takes a sequence of depth images taken from depth sensor
115+
(or any depth images source such as stereo camera matching algorithm or even raymarching renderer).
116+
The output can be obtained as a vector of points and their normals
117+
or can be Phong-rendered from given camera pose.
118+
119+
It extends the KinectFusion algorithm to handle non-rigidly deforming scenes by maintaining a sparse
120+
set of nodes covering the geometry such that each node contains a warp to transform it from a canonical
121+
space to the live frame.
122+
123+
An internal representation of a model is a voxel cuboid that keeps TSDF values
124+
which are a sort of distances to the surface (for details read the @cite kinectfusion article about TSDF).
125+
There is no interface to that representation yet.
126+
127+
Note that DynamicFusion is based on the KinectFusion algorithm which is patented and its use may be
128+
restricted by the list of patents mentioned in README.md file in this module directory.
129+
130+
That's why you need to set the OPENCV_ENABLE_NONFREE option in CMake to use DynamicFusion.
131+
*/
132+
class CV_EXPORTS_W DynaFu
133+
{
134+
public:
135+
CV_WRAP static Ptr<DynaFu> create(const Ptr<Params>& _params);
136+
virtual ~DynaFu();
137+
138+
/** @brief Get current parameters */
139+
virtual const Params& getParams() const = 0;
140+
141+
/** @brief Renders a volume into an image
142+
143+
Renders a 0-surface of TSDF using Phong shading into a CV_8UC4 Mat.
144+
Light pose is fixed in DynaFu params.
145+
146+
@param image resulting image
147+
@param cameraPose pose of camera to render from. If empty then render from current pose
148+
which is a last frame camera pose.
149+
*/
150+
151+
CV_WRAP virtual void render(OutputArray image, const Matx44f& cameraPose = Matx44f::eye()) const = 0;
152+
153+
/** @brief Gets points and normals of current 3d mesh
154+
155+
The order of normals corresponds to order of points.
156+
The order of points is undefined.
157+
158+
@param points vector of points which are 4-float vectors
159+
@param normals vector of normals which are 4-float vectors
160+
*/
161+
CV_WRAP virtual void getCloud(OutputArray points, OutputArray normals) const = 0;
162+
163+
/** @brief Gets points of current 3d mesh
164+
165+
The order of points is undefined.
166+
167+
@param points vector of points which are 4-float vectors
168+
*/
169+
CV_WRAP virtual void getPoints(OutputArray points) const = 0;
170+
171+
/** @brief Calculates normals for given points
172+
@param points input vector of points which are 4-float vectors
173+
@param normals output vector of corresponding normals which are 4-float vectors
174+
*/
175+
CV_WRAP virtual void getNormals(InputArray points, OutputArray normals) const = 0;
176+
177+
/** @brief Resets the algorithm
178+
179+
Clears current model and resets a pose.
180+
*/
181+
CV_WRAP virtual void reset() = 0;
182+
183+
/** @brief Get current pose in voxel space */
184+
virtual const Affine3f getPose() const = 0;
185+
186+
/** @brief Process next depth frame
187+
188+
Integrates depth into voxel space with respect to its ICP-calculated pose.
189+
Input image is converted to CV_32F internally if has another type.
190+
191+
@param depth one-channel image which size and depth scale is described in algorithm's parameters
192+
@return true if succeded to align new frame with current scene, false if opposite
193+
*/
194+
CV_WRAP virtual bool update(InputArray depth) = 0;
195+
196+
virtual std::vector<Point3f> getNodesPos() const = 0;
197+
198+
virtual void marchCubes(OutputArray vertices, OutputArray edges) const = 0;
199+
200+
virtual void renderSurface(OutputArray depthImage, OutputArray vertImage, OutputArray normImage, bool warp=true) = 0;
201+
};
202+
203+
//! @}
204+
}
205+
}
206+
#endif

0 commit comments

Comments
 (0)