AEROSENSE
An end-to-end edge-AI XR telepresence system integrating Meta Quest 3, NVIDIA Jetson Orin NX, real-time computer vision, and spatial reconstruction for immersive remote scene understanding.
Overview
AeroSense is an end-to-end edge-AI XR telepresence system integrating a remote IP camera, NVIDIA Jetson Orin NX, real-time computer vision, Unity, and Meta Quest 3 for immersive remote scene understanding and visualization.
The system was developed during a summer internship (May – July 2026) and demonstrates how embedded perception, spatial computing, and XR can be combined to let users understand and interact with remote environments in real time.
Perception Pipeline
The remote perception pipeline is built around RTSP video acquisition and embedded inference, processing live camera data with:
- YOLOv8 for real-time object detection
- SAM2 for instance segmentation
- MiDaS for monocular depth estimation
These models run on the NVIDIA Jetson Orin NX, providing the raw perception data needed for spatial reconstruction.
Visualization Modes
The system supports dual visualization modes:
- Remote Camera Feed — A conventional camera view with AI overlays showing detected objects, segmentation masks, and depth information
- XR Passthrough Mode — Presents reconstructed remote objects inside the user's physical environment using Meta Quest 3's passthrough capabilities
Novel Wireframe View (NWV)
The Novel Wireframe View reconstructs detected objects as persistent world-space wireframe geometry from segmentation contours, estimated depth, calibrated camera pose, and remote camera orientation. This creates a spatial understanding of the remote environment that persists and updates in real time within the Meta Quest 3 headset.
Spatial Calibration & Interaction
Spatial calibration and coordinate transformations between the remote camera and Meta Quest coordinate systems maintain alignment between reconstructed geometry and remote camera movement.
Bidirectional XR interaction through head tracking and UDP communication allows natural headset movement to control a remote two-axis pan-tilt camera while synchronizing camera orientation with the visualization system.
AeroSenseNet
AeroSenseNet is a unified extension of YOLOv8-Seg that combines object detection, instance segmentation, and object-centric depth estimation within a single inference pipeline. This reduces the redundant multi-model computation of running YOLOv8, SAM2, and MiDaS independently.
The architecture was designed around modular perception and visualization components so the multi-model backend could be replaced without changing the Unity visualization, networking, or reconstruction pipeline.