OpenAI Gymnasium

OpenAI Gymnasium (formerly OpenAI Gym) is the standard toolkit for developing and comparing reinforcement learning (RL) algorithms. Originally created by OpenAI in 2016, it was transferred to the Farama Foundation in 2022 and rebranded as Gymnasium. It provides a universal interface for RL environments — simulated worlds where an AI agent observes a state, takes an action, and receives a reward. The agent learns to maximize cumulative reward through trial and error, eventually discovering optimal strategies for complex tasks.

Gymnasium is completely free and open source under the MIT license. It runs on any platform with Python and includes dozens of built-in environments (CartPole, LunarLander, Atari games, MuJoCo physics). But its real power is the standardized API — any RL algorithm that works with Gymnasium's reset() / step() interface works with any compatible environment. This means you can swap in a custom aerospace environment (satellite station-keeping, drone navigation, aircraft control) and immediately apply any RL algorithm from the literature.

For aerospace, Gymnasium is the foundation that makes RL research practical. You don't build your own RL training loop from scratch — you build a Gymnasium-compatible environment that models your aerospace problem, then plug it into established RL libraries like Stable-Baselines3, RLlib, or CleanRL. This separation of environment from algorithm is what makes aerospace RL research reproducible and portable.

High School

Start with Gymnasium's built-in environments — no aerospace knowledge required. Install Gymnasium via pip, then train an agent on CartPole (balance a pole on a cart) and LunarLander (land a spacecraft on a pad). LunarLander is particularly relevant — it introduces thrust vectoring, fuel management, and landing precision in a simplified 2D environment. These environments teach RL fundamentals: states, actions, rewards, episodes, and the exploration/exploitation tradeoff.

Use a simple algorithm to start — Gymnasium's documentation walks through Q-learning (a tabular RL method) before introducing neural network-based approaches.

Undergraduate

Move from built-in environments to custom aerospace environments. Key projects:

• Build a satellite station-keeping environment: Model two-body orbital dynamics in a custom Gymnasium environment. The agent observes position and velocity, actions are thruster firings, and the reward penalizes altitude deviation and fuel usage
• Aircraft pitch controller: Create a Gymnasium environment using linearized longitudinal dynamics. Compare the RL-learned controller against a traditional PID controller
• Drone obstacle avoidance: Build a 2D or 3D navigation environment with obstacles, train an agent using Stable-Baselines3's PPO or SAC algorithms
• Multi-agent air traffic: Use PettingZoo to create a simplified airspace where multiple aircraft agents learn conflict resolution

David Silver's reinforcement learning course (UCL, free on YouTube) provides the theory. Stable-Baselines3 documentation has Gymnasium-specific tutorials. OpenAI Spinning Up (spinningup.openai.com) is an excellent free resource for understanding RL algorithms.

Advanced / Graduate

Graduate RL research for aerospace typically involves:

• High-fidelity environments: Wrap JSBSim (open-source flight dynamics), Basilisk (spacecraft simulation), or AirSim (drone simulation) with Gymnasium interfaces for realistic training
• Sim-to-real transfer: Train in simulation, deploy on real hardware — bridging the reality gap through domain randomization, physics perturbation, and progressive training
• Safe RL: Constrained RL algorithms that guarantee safety constraints (altitude limits, geofencing, collision avoidance) during both training and deployment
• Hierarchical RL: High-level mission planning combined with low-level control — e.g., a mission planner that selects waypoints while a lower-level controller handles flight dynamics