AI-Enabled Multi-Modal Object Detection Platform

Project Overview

AEGIS is a multi-modal computer vision platform designed to process optical (RGB), Synthetic Aperture Radar (SAR), and thermal/infrared imagery through a centralized detection and analysis dashboard. It integrates multiple deep-learning models for object detection, classification, instance segmentation, and video tracking, enabling users to compare model outputs, monitor inference performance, and visualize detected objects through a unified interface.

The platform combines a React-based frontend with a FastAPI backend to support image and video processing, model selection, configurable detection settings, and real-time result visualization.

Technologies Used

• React 18 and Vite
• Python and FastAPI
• Ultralytics YOLO
• Grounding DINO
• Mask R-CNN and Segment Anything Model (SAM ViT-H)
• OpenCV and PyTorch
• REST APIs and real-time event streaming
• GPU acceleration support with CUDA
• Geospatial visualization and detection result overlays

Problem Statement

Object detection workflows often rely on separate tools for different sensor types and AI models. Comparing model performance, processing multiple image modalities, configuring detection classes, and tracking objects across video frames can become complex and time-consuming. A unified platform is needed to simplify model management, streamline inference workflows, and present detection results in an interactive dashboard.

Our Approach

Ingest → Preprocess → Detect → Compare → Track → Visualize

The platform accepts optical, SAR, and thermal imagery and routes inputs to the selected computer vision models. YOLO supports object detection, Grounding DINO enables text-prompted detection, and Mask R-CNN and SAM provide segmentation capabilities. The backend processes requests and returns results to the React dashboard, where users can inspect detections, adjust model-specific settings, compare outputs, and analyze video tracking results.

Core Functionality

• Multi-modal image processing for optical RGB, SAR, and thermal/IR data
• Multiple AI model integrations for object detection and segmentation
• Text-prompted object detection using Grounding DINO
• Bounding-box visualization, class labels, and confidence scores
• Video processing with object tracking across frames
• Model-specific configuration and detection-class selection
• Side-by-side comparison of model outputs
• Model registry and custom model-weight management
• Batch image processing and inference performance monitoring
• Interactive dashboard for reviewing detection results

Results & Impact

AEGIS provides a centralized environment for evaluating and visualizing AI-based object detection across multiple image modalities. By bringing model selection, inference, detection overlays, video tracking, and result comparison into one interface, the platform simplifies computer vision workflows and supports more efficient model evaluation.

Actual detection accuracy, inference latency, and throughput depend on the selected model, hardware, input data, and evaluation conditions. Quantitative performance improvements should be reported only after benchmarking.

Target Users

• Geospatial intelligence analysts
• Computer vision researchers and developers
• Aerial and drone imagery analysis teams
• Maritime and infrastructure monitoring teams
• Environmental monitoring and disaster response teams
• AI model evaluation and testing teams

Technical Challenges

Integrating multiple computer vision architectures with different input requirements, managing GPU and memory resources, maintaining consistent detection outputs, and tracking object identities across video frames present significant engineering challenges. Supporting different sensor modalities also requires careful preprocessing and model-specific configuration.

Operational Challenges

The platform must accommodate varying image resolutions, sensor characteristics, lighting conditions, and model capabilities while maintaining a clear, responsive interface. Reliable model selection, independent configuration settings, and consistent visualization are important for helping users interpret detection results effectively.