Building virtual worlds for safer autonomous driving.

Abstract

Hazardous interactions between vehicles and vulnerable road users often emerge through subtle spatial and motion cues that are difficult to identify from monocular video. VRU-HazardNet turns these cues into an explicit risk score with a two-stream pipeline: MonoTTA and AB3DMOT recover tracked monocular 3D geometry, SEA-RAFT and ResNet-50 encode motion, and a causally masked temporal Transformer fuses both streams. Mean-pooled temporal features pass through a calibrated risk head with temperature scaling and Focal Loss, improving confidence stability and sensitivity to rare hazards. The work also introduces VRUHI, a dedicated benchmark for proactive vehicle–VRU risk assessment rather than post-event recognition.

• ArchitectureUses frozen 3D detection, tracking, and optical-flow front ends with a 4-layer, 8-head, 512-dimensional Transformer for temporal risk reasoning.
• DatasetVRUHI contains 6,000 one-second, 25-frame D2-City clips—2,000 hazardous and 4,000 safe—with severity, interaction, and lane-separation annotations.
• EvidenceAchieves 78.37% AUC and 41.03% F1; combining 3D boxes and flow adds 7.09 AUC points over RGB alone and exceeds R(2+1)D by 2.95 AUC points.

@inproceedings{ni2025vru,
  title={VRU-Centric Hazardous Scenario Detection via Monocular Spatiotemporal Feature Fusion},
  author={Ni, Ying and Li, Siying and Fan, Jialin},
  booktitle={2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC)},
  pages={4685--4690},
  year={2025},
  doi={10.1109/ITSC60802.2025.11423759}
}

Abstract

Safety-critical scenarios are valuable for autonomous-driving validation but rare in natural data, while unconstrained adversarial generation often produces unrealistic or inevitable collisions that leave the tested vehicle no meaningful response. Constrained-Adversarial Policy Optimization (CAPO) introduces adversarial rationality through a two-phase continual-learning framework. Phase I augments independent multi-agent PPO with a learned safety-constraint function that approximates Hamilton–Jacobi reachability, allowing background vehicles to learn realistic task completion and safe interaction. Phase II introduces an autonomous-vehicle expert, freezes the task value function, and optimizes adversarial behavior according to the expert's estimated safety margin. This creates near-conflict interactions that remain challenging and solvable rather than collapsing into unavoidable crashes.

• Constrained policyA dual-value objective combines task advantage with a learned safety value, then uses that constraint to regulate adversarial pressure on the AV expert.
• Training setupTrained in MetaDrive with 20 agents, partial local observations, five random seeds, and 1.5 million steps; compared against Vanilla-IPPO and collision-driven ADV-IPPO.
• ValidationOpen-loop TTC/PET analysis reduces extreme-risk cases, while four closed-loop scenarios preserve corrective space in left-turn, crossing, cut-in, and head-on conflicts.

@inproceedings{fan2025interactive,
  title={Interactive Adversarial Scenario Generation for Autonomous Driving: A Continual Learning Framework with Safety Constraints},
  author={Fan, Jialin and Ni, Ying and Chen, Yuhang and Li, Siying and Sun, Jie and Sun, Jian},
  booktitle={2025 IEEE 28th International Conference on Intelligent Transportation Systems (ITSC)},
  pages={4656--4662},
  year={2025},
  doi={10.1109/ITSC60802.2025.11423840}
}