Reinforcement learning

Six control tasks built on the same physics as the rest of the library, plus a trainer that needs nothing but NumPy.

flybots envs                # list the tasks
flybots train hover         # learn to hold position from scratch
flybots play hover --policy policies/hover.npz --gif hover.gif

quadrotor

hover easy

Hold a point in space from a randomised offset, tilt and velocity.

quadrotor

waypoint medium

Fly to a random goal and settle on it, not just pass through it.

quadrotor

landing medium

Descend and touch down gently inside a pad. Reward scales with how soft.

quadrotor

trajectory hard

Track a moving Lissajous reference. The target never stops.

fixed-wing

fw-cruise medium

Hold altitude, airspeed and course on a wing that stalls if you slow down.

fixed-wing

fw-waypoint hard

Fly a route of four waypoints without stalling or losing the aircraft.

Why a gym here

Most drone RL benchmarks either wrap a heavyweight simulator you cannot read, or use dynamics simple enough that what the policy learns does not transfer to anything. These tasks sit on the same 6DOF models the rest of the library uses — the quadrotor with its motor lag, the fixed wing with its stall — so the control problem is the real one, and you can read every line of the thing you are learning against.

The fixed-wing tasks are the ones worth your time. A wing cannot stop, it stalls if it slows down, and altitude, airspeed and heading are all coupled through four surfaces. That is a much less forgiving test of a learned controller than hovering.

The API

Gymnasium's interface, without the Gymnasium dependency:

from uav_sim.gym import make

env = make("hover", seed=0)
observation, info = env.reset()

for _ in range(500):
    action = env.action_space.sample(env.rng)
    observation, reward, terminated, truncated, info = env.step(action)
    if terminated or truncated:
        break

If Gymnasium is installed, the environments register themselves on import and work with any standard RL library:

import gymnasium as gym
import uav_sim.gym          # registers uav_sim/Hover-v0 and friends

env = gym.make("uav_sim/Hover-v0")

pip install "flybots[gym]"

Training

from uav_sim.gym import train, evaluate

result = train("hover", iterations=100, seed=0)
print(result.best_return)

summary = evaluate("hover", result.policy, episodes=25)
print(summary)
# {'mean_return': ..., 'mean_length': ..., 'pct_ground_contact': ...}

The default optimiser is Augmented Random Search over a linear policy — about 80 parameters. On body-frame errors a linear policy is structurally a PD controller, which is the shape these tasks want.

evaluate reports the share of episodes ending in each termination reason alongside the return, because how a policy fails tells you far more than the number.

See Training a policy for how it works and the four choices that decide whether it learns at all.

Design notes

Several of the decisions in these environments are the difference between a task that trains in minutes and one that never leaves the ground:

• Actions are centred on equilibrium, so a zero action hovers or flies level rather than dropping the aircraft.
• Observations are body-frame, matching the frame the action acts in.
• Attitude enters as a rotation matrix, not Euler angles.
• Every candidate is scored on the same episodes.

Designing a task works through why each of those matters, with the measurements that motivated them.

Honest limits

• The trainer is derivative-free. It solves these tasks; it is not a substitute for PPO or SAC on a hard, high-dimensional problem.
• There is no domain randomisation over vehicle parameters, so a policy learned here is tuned to one airframe.
• There is no wind, so a policy never learns to reject it.
• Nothing here is sim-to-real. The models omit ground effect, rotor-wake interaction and aeroelasticity — see what the models do not include.