← work

Computer vision · research pipeline

Describing objects with multiple views and 3D context

GitHub repository

A scene-understanding pipeline built around ARKitScenes: reconstruct a scene, segment objects in images, lift the masks into 3D, and compare ways of describing the resulting objects.

COLMAP supplies camera geometry and reconstruction, while SAM supplies image masks. The pipeline lifts and fuses observations into object-level 3D representations, then extracts image crops for captioning.

Three captioning paths explore what the model gains from additional context: a baseline, a multiview approach, and a version that incorporates spatial information. The evaluation scripts compare caption attributes and heuristic consistency signals; the recorded example is a limited experiment rather than a broad benchmark.