How OpenDerm builds a 3D skin map
The robot records overlapping, high-resolution photographs of small regions of skin, camera poses estimated from the robot’s encoders, and laser range measurements. The reconstruction pipeline refines the camera poses, estimates the shape of the skin, and projects every photograph onto a single continuous, measurable surface.
The pipeline
Capture
Acquire overlapping photographs while the tilt axis and laser sensors control camera orientation and working distance.
STAGE 1→Match
Identify corresponding skin features across overlapping photographs — approximately 6,000 candidate features per frame.
STAGE 2→Optimize
Jointly refine all camera poses and thousands of three-dimensional landmarks, using the encoder-derived poses as initial estimates.
How the robot images the skin.
Before each image capture, the robot positions the camera at a known working distance from the skin and aligns it normal to the local skin surface.
Two laser displacement sensors mounted beside the camera measure its distance and orientation relative to the skin. Using these measurements, the Rx joint tilts the camera and the z-axis adjusts its height until the camera reaches the target working distance and its optical axis aligns with the local surface normal. Once aligned, the robot captures a sharp, consistent, and reproducible 32.5-megapixel macro photograph and saves it with a synchronized JSON metadata file containing the station index, encoder-derived camera pose, and both laser distance measurements. The image and metadata are then passed to the reconstruction pipeline.
The robot scans the body in a serpentine raster. Adjacent images overlap by approximately 75%, so most regions of skin appear in several frames from slightly different viewpoints. This overlap provides the correspondences needed for image registration and surface reconstruction.
Each high-resolution polarized frame captures individual lesions in fine detail — pigment patterns, border structure, and surface texture are all clearly visible. In this respect the images approach the quality of dermoscopic images, the magnified close-up views used to manually examine skin lesions in the clinic.
The challenges of reconstructing an accurate 3D surface.
The robot records an approximate camera pose for every frame, but those coordinates are not accurate enough to place high-magnification images directly onto a common surface. Four factors make the reconstruction substantially harder than conventional 2D image stitching.
Encoder-based estimates lack sufficient precision
The robot’s encoders provide millimeter-scale position estimates, while each image samples the skin at approximately 13 µm per pixel. In this scan, the optimized camera poses differed from the encoder-based estimates by a median of 3.5 mm and by up to 14.3 mm. Discrepancies of this magnitude prevent precise image alignment.
The surface moves during the scan
Breathing and small movements shift the skin during the scan. Images of the same region captured on neighboring passes may therefore show the skin displaced by several millimeters. The reconstruction must account for this motion rather than assume a perfectly rigid surface.
Flat stitching distorts measurements
Projecting images of a curved skin surface onto a flat 2D plane would distort distances and lesion dimensions, particularly in sloped regions. Measurements must therefore be made along the reconstructed 3D surface.
The narrow field of view limits depth information
The macro lens has a field of view of only about 8°, so neighboring images view the skin from nearly the same angle. This provides limited depth information: small camera translations can appear similar to rotations, causing image-only estimates to drift. The reconstruction therefore also uses encoder-derived poses and laser distance measurements.
Before and after pose optimization
The effect is visible when four images of the same mole are projected onto the reconstructed surface. With encoder-based estimates, repeated observations of the lesion are displaced by up to 3.7 mm. This displacement blurs the lesion boundary and prevents fine features such as pores and hairs from aligning.
Optimizing the camera poses aligns the photographs closely enough to preserve fine structures such as pores and hairs. This precision is necessary to measure submillimeter changes in a lesion over time.
Finding corresponding features across images.
At a distance, skin can appear nearly featureless. At 13 µm per pixel, however, each image contains abundant fine-scale structure: pores, hair follicles, freckles, and variations in pigmentation. The pipeline detects thousands of these local features in each frame and identifies corresponding features in overlapping images.
CLAHE
Pose 1
Pose 2
Left: 612 of 6,000 SIFT keypoints detected in a 14 mm, CLAHE-equalized crop centered on a mole. Center and right: corresponding skin regions from two camera poses. RANSAC retained 920 of 1,321 candidate matches; 40 spatially distributed correspondences are shown.
Restricting and verifying candidate matches
Because skin contains many similar, repeated patterns, descriptor similarity alone can produce incorrect matches. The pipeline uses the robot’s approximate camera poses to predict which frames overlap and the expected displacement of features between them. Candidate correspondences inconsistent with this prior geometry are rejected.
The remaining candidates are evaluated with RANSAC. An image pair is accepted only if at least 25 correspondences are consistent with a single geometric transformation.
Jointly refining camera poses and 3D landmarks.
Pairwise matches are linked across images to form tracks. If a pore in frame A matches one in frame B, and the same pore in frame B matches one in frame C, the three observations are combined into a single track representing the same skin feature. Each valid track contains at most one observation per frame; ambiguous chains are discarded.
Tracks connect the two stages of optimization. During triangulation, a track identifies the camera rays that should converge on the same 3D landmark. During pose refinement, that landmark is projected back into each image and compared with its observed pixel location.
The track assignments stay fixed while the algorithm alternates between two steps until convergence:
- Triangulate — hold the camera poses fixed. Each observation in a track, together with its frame's current camera pose, defines a ray into the scene. The solver finds the 3D point that passes as close as possible to all of these rays. This is how the track becomes a landmark.
- Refine poses — hold the landmarks fixed. For one frame at a time, the solver gathers every track observed in that frame. It projects each track's landmark back into the image, compares the predicted pixel with the measured pixel in the track, and adjusts the camera's three position and three rotation parameters to reduce those reprojection errors.
Pose refinement updates each camera pose using the collective reprojection error. For example, if many projected landmarks fall to the right of their measured keypoints, the optimizer adjusts the camera position and orientation until the full pattern aligns. Because several frames share the same landmarks, updating one camera must preserve consistency with the others.
The encoder-derived poses provide the initial estimates and remain as soft constraints throughout the optimization. After each round, the pipeline reprojects every observation, removes those with large errors or physically implausible geometry, and retains only tracks supported by at least two frames. Frames with fewer than 20 usable observations remain at their prior pose.
3D reconstruction
The interactive model below shows the reconstruction described on this page. Drag to rotate the model.
The completed reconstruction combines a global map of visible skin features with high-resolution imagery of individual lesions. The panels below move from the complete 3D reconstruction to the pigment network of a single mole.
Because registered scans share the same coordinate system, scans taken months apart can be compared directly.