Computer vision · research pipeline
Describing objects with multiple views and 3D context
A scene-understanding pipeline built around ARKitScenes: reconstruct a scene, segment objects in images, lift the masks into 3D, and compare ways of describing the resulting objects.
COLMAP supplies camera geometry and reconstruction, while SAM supplies image masks. The pipeline lifts and fuses observations into object-level 3D representations, then extracts image crops for captioning.
Three captioning paths explore what the model gains from additional context: a baseline, a multiview approach, and a version that incorporates spatial information. The evaluation scripts compare caption attributes and heuristic consistency signals; the recorded example is a limited experiment rather than a broad benchmark.
- Data download and preparation for ARKitSceneRefer scenes.
- Reconstruction, segmentation, 3D lifting, object fusion, and crop extraction.
- Baseline, multiview, and context-aware caption generation with a shared evaluation workflow.