SafeDrive: Constrained Safe RL for Autonomous Driving

SafeDrive is a bounded simulation project in MetaDrive investigating Constrained Safe Reinforcement Learning for autonomous vehicle navigation (key design principles). Rather than relying on unconstrained reward shaping or manual multi-stage curriculum switching, the project formulates closed-loop autonomous driving as a Constrained Markov Decision Process (CMDP). The objective is to learn a single policy that maximizes progress and speed rewards while strictly bounding cumulative collision and off-road safety costs below a declared threshold ($d \le 1.0$).

System Architecture & Perception

1. Perception & Observation Space (275-D)

The policy receives a 275-dimensional feature vector combining spatial range sensing with ego dynamics and nearby vehicle tracking:

• 240 360° LiDAR Rays: Radial distance measurements ($[0, 50]$ meters) scanning surrounding obstacles and road boundaries.
• 4 Nearby Vehicle Slots (16-D): Relative coordinates $(x, y)$, heading angle, and velocity vectors for the four closest dynamic traffic obstacles.
• Ego Kinematics & Navigation (19-D): Speed, steering angle, angular velocity, lane offset, route checkpoints, and navigation target vectors.

2. Network Architecture & Policy Optimization

We parameterize the policy $\pi_\theta(a|s)$, reward critic $Q_{\phi_R}(s, a)$, and safety cost critic $Q_{\psi_C}(s, a)$ using 512-wide Multi-Layer Perceptrons ([512, 512]) with ReLU activations:

• Policy Network ($\pi_\theta$): Outputs continuous control actions $a = (a_{\text{steer}}, a_{\text{accel}}) \in [-1, 1]^2$ via a squashed Gaussian distribution: \(\pi_\theta(a|s) = \tanh(\mu_\theta(s) + \sigma_\theta(s) \odot \epsilon), \quad \epsilon \sim \mathcal{N}(0, I)\)
• Dual-Critic Architecture ($Q_{\phi_R}, Q_{\psi_C}$): Twin double-critic networks estimate expected cumulative task reward $J_R$ and expected cumulative safety cost $J_C$. Soft Bellman targets are computed via: \(y_R = r + \gamma \left( \min_{j=1,2} Q_{\phi_{R,j}'}(s', a') - \alpha \log \pi_\theta(a'|s') \right), \quad a' \sim \pi_\theta(\cdot|s')\) \(y_C = c + \gamma_c \min_{j=1,2} Q_{\psi_{C,j}'}(s', a'), \quad a' \sim \pi_\theta(\cdot|s')\)
• Policy Loss Formulation: Policy parameters $\theta$ are updated by joint optimization over expected task reward, entropy regularization, and safety constraint penalty: \(J_\pi(\theta) = \mathbb{E}_{s \sim \mathcal{D}, a \sim \pi_\theta} \left[ \alpha \log \pi_\theta(a|s) - \min_{j=1,2} Q_{\phi_{R,j}}(s, a) + \lambda \max_{j=1,2} Q_{\psi_{C,j}}(s, a) \right]\)
• Optimization & Hyperparameters: Actor and critic parameters are optimized using Adam with learning rate $1 \times 10^{-4}$, batch size 256, 12 gradient updates per step (1.0 update-to-data ratio), replay buffer capacity $10^6$, and discount factors $\gamma = 0.99, \gamma_c = 0.99$.

3. SAC-Lagrangian Optimization & PID Cost Adaptation

The Lagrangian objective incorporates a PID feedback mechanism to regulate constraint violations:

• Episode Cost Limit ($d$): $1.0$ cumulative cost limit.
• PID Controller Parameters: $K_p = 0.05$, $K_i = 0.0005$, $K_d = 0.1$, $\alpha_{\text{EMA}} = 0.2$, $\lambda_{\max} = 100.0$.

4. Multi-Domain Simulation Environments

Training is conducted over 1,000 procedural 3-block MetaDrive scenarios (map: 3, training seeds 40000–40999). The environment allocates 12 parallel subprocess workers operating under a frozen multi-domain distribution to cover diverse road geometries and dynamic background traffic conditions:

• 3 Geometry Workers (map: 3, traffic_density: 0.00): Traffic-free procedural road geometry to maintain precise curve handling and lane centering.
• 4 Introductory Traffic Workers (map: 3, traffic_density: 0.05): Light background traffic flow (~2–6 vehicles per map).
• 5 Stress Traffic Workers (map: 3, traffic_density: 0.30): Dense background traffic flow (~13–27 vehicles per map) for high-pressure gap selection and collision avoidance.

5. Evaluation Protocol & Panel Structure

To prevent evaluation leakage and maintain auditability over training runs, we use a strict three-panel evaluation protocol with non-overlapping seed assignments:

• Screening Panel: Evaluates checkpoints every 50,000 steps across 50 episodes per condition (geometry 0.00 and traffic 0.30) using non-overlapping seed range 55000–55049.
• Model Reranking Panel: Evaluates the top 5 screening checkpoints plus the terminal checkpoint across 100 fresh episodes per condition (geometry 0.00 and traffic 0.30) using seed range 60000–60099.
• Sealed Holdout Panel: Single final evaluation of the frozen Reranking Winner across 200 untouched episodes per condition (geometry 0.00 and traffic 0.30) using seed range 70000–70199.

Controlled Ablation Framework

To evaluate the exact contribution of Lagrangian safety cost constraints, we compare SafeDrive SAC-Lagrangian against an unconstrained Vanilla SAC Baseline. Both runs operate under 100% identical environment parameters, 275-D perception specifications, [512, 512] MLP network topology, hyperparameters ($1 \times 10^{-4}$ learning rate, batch size 256, 12 gradient updates per step), 1,000 procedural 3-block maps (map: 3, seeds 40000–40999), 12-worker multi-domain mixtures, and three-panel evaluation protocols.

By keeping all environmental, perceptual, structural, and evaluation parameters fixed, the experimental variable is strictly isolated to the algorithm optimization formulation:

Simulation Rollout & Policy Visualizations

Below are curated video simulations across key autonomous driving scenarios:

1. Interactive Multi-Vehicle Traffic Navigation

2. Precision Curve & Lane-Centering Navigation

3. Roundabout & Complex Junction Merging

4. Emergency Obstacle Evasion & Hazard Avoidance

Empirical Results & Benchmark Performance

Below is the comparative benchmark evaluation of the selected SafeDrive SAC-Lagrangian Winner against the Unconstrained Vanilla SAC Baseline on the final 200-episode Sealed Holdout Panel across both geometry (traffic_density: 0.00) and stress traffic (traffic_density: 0.30) conditions:

Key Design Principles

Key architectural principles driving SafeDrive’s empirical performance include:

1. PID-Controlled Cost Multiplier Adaptation: Implements a proportional-integral-derivative (PID) feedback controller on the dual multiplier $\lambda$, dynamically damping multiplier oscillations and ensuring stable constraint enforcement below $d \le 1.0$.
2. Native Stable-Baselines3 MetaDrive Integration: Wraps MetaDrive’s procedural environment and vectorized multi-worker interfaces directly into Stable-Baselines3 for clean PyTorch training, custom callbacks, and modular CMDP extensions.
3. Vectorized 12-Worker Domain Mixture on Google Colab L4 GPU: Parallelizes 12 environment workers on an L4 GPU across geometry ($0.00$), light traffic ($0.05$), and stress traffic ($0.30$) conditions over 1,000 procedural 3-block maps (map: 3).
4. Controlled Unconstrained Ablation Baseline: Evaluates an unconstrained Vanilla SAC baseline run under identical perception, network architecture, and seeds (40000–40999) to isolate the empirical impact of the PID-Lagrangian safety constraint.

Future Directions & Scalability Roadmap

SafeDrive’s CMDP optimization framework and PID dual control architecture lay the foundation for several high-impact research extensions:

1. CARLA High-Fidelity Simulator Transfer: Transitioning from procedural MetaDrive grid maps to CARLA’s urban environments featuring high-fidelity visual rendering, complex multi-lane weather topologies, real-time CARLA ROS2 integrations, and multi-modal sensory suites (RGB cameras, depth sensors, semantic segmentation, and radar).
2. Model-Based Safe RL & World Models: Incorporating model-based predictive safety (e.g., Safe MBPO or DreamerV3 World Models) to imagine trajectory rollouts and evaluate safety cost bounds $Q_C(s,a)$ inside a latent environment model, dramatically accelerating sample efficiency while maintaining zero-collision guarantees during exploration.
3. Multi-Agent Safe Reinforcement Learning (MARL): Extending the single-agent SAC-Lagrangian formulation to multi-agent game-theoretic settings (e.g., MAPPO-Lagrangian or decentralized dual control) where multiple autonomous vehicles dynamically negotiate right-of-way, multi-lane highway merging, and unsignalized intersection crossings under joint safety bounds.
4. Multimodal Vision-Language-Action (VLA) Guidance: Integrating multimodal foundation models for high-level semantic reasoning (interpreting dynamic road signs, construction signals, and emergency vehicle audio/visual cues) paired with low-level SAC-Lagrangian control bounds for provably safe trajectory execution.

Codebase & Formal Documentation: The complete PyTorch and MetaDrive codebase is maintained in the SafeDrive Repository. For full theoretical derivations, documented negative results and failed experiments, and complete BibTeX citations, visit the GitHub repository. Our implementation directly utilizes the MetaDrive Simulator engine and adopts many of its recommended hyperparameter baseline standards.