Training a Quadrotor to Avoid Obstacles with PPO
A custom Gymnasium environment for 2D quadrotor control, trained end-to-end with Proximal Policy Optimization. Most of the work was in the reward shaping and the simulator, not the RL algorithm.
Goal: a controller that flies a 2D planar quadrotor to a target while avoiding obstacles, learned end-to-end from a reward signal. The challenge was designing a reward that produced the behaviour I wanted without the policy collapsing into any of the obvious local optima (hover forever, ram the wall, ignore the target).
Why PPO, why custom env
Classical controllers work beautifully when you know the trajectory ahead of time, but they're brittle the moment obstacles enter the picture and the path has to be replanned. RL handles that naturally — the policy learns what to do based on the current state, including obstacle positions, without needing a precomputed plan.
I picked PPO because it's stable, sample-efficient enough for a 2D problem, and
well-supported through Stable-Baselines3. The environment is a custom
Gymnasium QuadrotorEnv: writing it from scratch meant I controlled
every part of the reward and dynamics, which turned out to matter a lot.
The environment
State and action
The quadrotor is a 6-DOF planar system:
state ∈ ℝ⁶ : [ pₓ, p_y, vₓ, v_y, θ, ω ] action ∈ ℝ² : [ u₁, u₂ ] rotor forces
Dynamics
Standard planar quadrotor model. Mass m, moment of inertia
I, rotor arm r, gravity g:
ṗₓ = vₓ m·v̇ₓ = −(u₁ + u₂) sin θ ṗ_y = v_y m·v̇_y = (u₁ + u₂) cos θ − m·g θ̇ = ω I·ω̇ = r·(u₁ − u₂)
Integration with simple Euler at a small fixed timestep. The pitch torque is the difference of rotor forces; the total thrust is their sum. This means the action space is naturally collinear with the two things the policy needs to modulate — translation force and rotation torque.
Reward shaping
Most of the time on this project went into reward design. The combined reward has four components, all of which had to be balanced against each other before the policy did anything reasonable:
- Target reaching — exponential decay in distance to target. Sharper than linear, which kept the agent from being satisfied with "kind of close."
- Obstacle avoidance — penalty that scales with how close it gets. Hard-margin penalties (large constant on collision) alone don't shape the policy enough; the gradient near the obstacle has to push it away gently.
- Boundary violation — terminal-ish penalty for leaving the workspace. Important early in training when the policy outputs wild actions.
- Control effort — small quadratic cost on action magnitude. Keeps the trained policy smooth and stops it from saturating the rotors when it doesn't need to.
The biggest mistake I made early was using a sparse reward (only on reaching the target). PPO can't escape the no-signal regime if every episode looks the same to the value function. The exponential distance reward fixed that — every step in the right direction gets a positive signal, so the policy has gradient information from step one.
Gravity compensation
One thing that helped a lot: rather than letting the policy learn from zero, I added an action bias equal to the hover thrust. This way the policy outputs deltas around hover rather than absolute thrusts.
Without this, the policy spends thousands of episodes just learning that without thrust the quadrotor falls. With it, the policy starts from a stable baseline and the only thing it has to figure out is when to push around that baseline. Convergence time dropped noticeably.
Training
PPO via Stable-Baselines3, MLP policy:
| Hyperparameter | Value |
|---|---|
| Learning rate | 9e-3 |
| Batch size | 32 |
| Total training steps | 600,000 |
| Entropy coefficient | 1e-2 |
The learning rate is higher than the usual 3e-4 default. I
ran a small sweep and a more aggressive LR worked because the problem is
low-dimensional and the env is fast — wall-clock convergence beat sample
efficiency considerations.
The entropy coefficient stays at 1e-2 through training (no
decay). This kept enough exploration alive for the policy to discover paths
around obstacles instead of memorising one route to one target.
What worked
After ~600k environment steps the policy reliably reaches targets, navigates around obstacles, and produces smooth control inputs. Most of the convergence happens in the first ~200k steps; the rest is fine-tuning.
The lessons that transferred to later projects: shape the reward densely, bias the action space around a known-good baseline, and don't trust default hyperparameters when the env is small and fast.
Code & paper
github.com / Abhimanyu-0 / PPO_quadrotor↗
Coursework with Prof. Ludovic Righetti and TA Armand Jordana at NYU.