completed2017 – 2019Our role: Scientific Lead (Postdoc at TUM CAMP)

Siemens CAD

CAD model-based detection and tracking of large objects

Industry project with Siemens AG (Corporate Technology) on learning the pose of large industrial objects and of the camera or X-ray device observing them, using their CAD models as prior knowledge — from X-ray PoseNet for mobile X-ray inspection to multi-task pose estimation from depth images and image-based camera relocalisation.

Project teaser
Highlights and overlays are illustrative. Images: Siemens Pressebild, Wikimedia Commons, CC BY-SA 3.0.

Abstract

Inspecting, assembling and servicing large industrial objects — gas turbines, generators, machine parts — requires knowing precisely where the object is and from where it is observed. In industry, nominal CAD models of these objects are almost always available, yet classical pipelines rely on pre-calibrated set-ups, hand-crafted features and exhaustive search, which break down for mobile devices, textureless metal surfaces, symmetric parts and partial views. The project asked how deep learning can use the CAD model as prior knowledge to estimate object and camera poses directly from images, so that 3D reconstruction, inspection and augmented-reality guidance become possible without manual calibration.

What we did

  • X-ray PoseNet for mobile X-ray devices. For industrial computed tomography with uncalibrated mobile X-ray systems, a network learns the mapping from X-ray projections — simulated from the object’s CAD model — to the 6-DoF pose of the device, with an additional reconstruction loss that improves the quality of the resulting 3D volume and generalises to unseen instances of the same object class (WACV 2017); the approach was also filed as a patent with Siemens (2018).
  • Object recognition and pose from depth images. A multi-task framework combines manifold learning of viewpoint descriptors (triplet loss) with direct pose regression, so poses can be read out directly or by descriptor matching; the learned descriptors were far more discriminative, with almost 30% higher relative pose accuracy than related work (ICRA 2018; patents 2019 and 2022).
  • Camera relocalisation from a single image. Pixel-wise scene-coordinate regression with learned confidences discards erroneous correspondences before pose estimation and supports hypothesis scoring and refinement (2018), and an adversarial framework learns the joint distribution of images and camera poses so that a discriminator can refine the regressed pose (ICCV Workshops 2019).

Our role

Scientific lead of the project as postdoc at TUM CAMP under Nassir Navab (PI), with Slobodan Ilic at Siemens Corporate Technology, and mentoring of Mai Bui’s PhD, which grew out of this collaboration.

Perspective

The work on ambiguity in pose estimation continued in the BaCaTeC project with Stanford University, which moved from single pose estimates to full distributions over camera poses (ECCV 2020).

Partner institutions

Publications 8