CBL - Campus del Baix Llobregat

Projecte llegit

Títol: A Benchmark Dataset and Agent for AI-Based Physical Security Surveillance

Estudiants que han llegit aquest projecte:

Director/a: TRAPOTE BARREIRA, CÉSAR

Departament: FIS

Títol: A Benchmark Dataset and Agent for AI-Based Physical Security Surveillance

Data inici oferta: 12-02-2026     Data finalització oferta: 12-10-2026


Estudis d'assignació del projecte:
    GR ENG TELEMÀTICA
Tipus: Individual
 
Lloc de realització: Fora UPC    
 
        Supervisor/a extern: Majd Eid
        Institució/Empresa: Milestone systems
        Titulació del Director/a: Engineering
 
Paraules clau:
AI, agent, security, system, monitoring
 
Descripció del contingut i pla d'activitats:
The project develops an agentic AI system to verify physical security events using multimodal models (LLM/MLLM).

It combines video, sensor metadata, and access logs to identify real alarms and reduce false positives.

The system includes an agent with memory, capable of perceiving, reasoning, acting, and learning from the data provided by the company.

An event taxonomy is used to structure detection and verification, covering: intrusions, behavioral anomalies, traffic incidents, threats to assets, environmental risks, and system artifacts.
 
Overview (resum en anglès):
Modern security facilities operate large networks of surveillance cameras that generate automatic alarms at high volume, the vast majority of which correspond to harmless events. When operators are overwhelmed by false alarms they lose trust in the system and risk missing genuine incidents, a problem known as alarm fatigue. Traditional automated approaches address this by retrieving visually similar events from a database, but visual similarity is not the same as a real threat. A person kneeling to tie their shoelaces near a perimeter fence can produce a feature vector nearly identical to that of an intruder attempting to climb it. Telling these apart requires reasoning over context, sensor states, access logs, site rules, which retrieval-based systems cannot do.
This thesis presents two main contributions. The first is a benchmark dataset constructed specifically to evaluate AI agents in physical security, organising video anomalies under a taxonomy of five categories and eighteen activity types grounded in ISO/IEC 27001/27002 and the SAFECARE project. The dataset exists in two versions: an internal one using private corporate footage from Milestone Systems, and a public one drawing from MSAD (NeurIPS 2024), NWPU Campus (CVPR 2023), and several smaller sources. Annotation was carried out with IntelliClip, a dedicated tool built as part of this work, which writes clip boundaries and metadata directly to structured JSON. Object tracking was subsequently applied twice, first with YOLOv8, then with the stronger BoxMOT library (YOLOv8x and BotSORT), producing clearly better results.
The second contribution is an autonomous agent that evaluates security events following the ReAct paradigm, alternating between reasoning, tool invocation, and observation until it can issue a final verdict of escalation or dismissal. Tools are treated as precision instruments, called only when the language model has a specific doubt it cannot resolve from prior evidence. Of the agent's thirteen tools, most are fast algorithmic methods; only two invoke a small vision-language model (SmolVLM-256M). Full quantitative evaluation over the entire dataset is left as future work, as running a multimodal agent across hundreds of clips demands substantial and repeated GPU computation best carried out in a dedicated research infrastructure.

© CBLTIC Campus del Baix Llobregat - UPC