CAD model-based detection and tracking of large objects

Sun, 01 Jan 2017 00:00:00 +0000

Inspecting, assembling and servicing large industrial objects — gas turbines, generators, machine parts — requires knowing precisely where the object is and from where it is observed. In industry, nominal CAD models of these objects are almost always available, yet classical pipelines rely on pre-calibrated set-ups, hand-crafted features and exhaustive search, which break down for mobile devices, textureless metal surfaces, symmetric parts and partial views. The project asked how deep learning can use the CAD model as prior knowledge to estimate object and camera poses directly from images, so that 3D reconstruction, inspection and augmented-reality guidance become possible without manual calibration.

What we did

X-ray PoseNet for mobile X-ray devices. For industrial computed tomography with uncalibrated mobile X-ray systems, a network learns the mapping from X-ray projections — simulated from the object’s CAD model — to the 6-DoF pose of the device, with an additional reconstruction loss that improves the quality of the resulting 3D volume and generalises to unseen instances of the same object class ( ); the approach was also filed as a patent with Siemens ( ).
Object recognition and pose from depth images. A multi-task framework combines manifold learning of viewpoint descriptors (triplet loss) with direct pose regression, so poses can be read out directly or by descriptor matching; the learned descriptors were far more discriminative, with almost 30% higher relative pose accuracy than related work ( ; patents and ).
Camera relocalisation from a single image. Pixel-wise scene-coordinate regression with learned confidences discards erroneous correspondences before pose estimation and supports hypothesis scoring and refinement ( ), and an adversarial framework learns the joint distribution of images and camera poses so that a discriminator can refine the regressed pose ( ).

Our role

Scientific lead of the project as postdoc at TUM CAMP under Nassir Navab (PI), with Slobodan Ilic at Siemens Corporate Technology, and mentoring of Mai Bui’s PhD, which grew out of this collaboration.

Perspective

The work on ambiguity in pose estimation continued in the with Stanford University, which moved from single pose estimates to full distributions over camera poses (ECCV 2020).

CAD Models |

CAD model-based detection and tracking of large objects

What we did

Our role

Perspective