A JSBSim autopilot project gives you flight traces and training curves. To make those results useful in a portfolio, design a test that another person can repeat.
For a portfolio project, the answer comes from a comparison you can reproduce. Show the task, the starting conditions, the controller you compared against, and the trials where your approach struggled. That gives a reviewer something to examine beyond a successful flight clip.
Our JSBSim reinforcement learning project explains the build: connect a flight simulator to a learning environment and train a controller. This article develops the evaluation plan. It assumes some Python and basic reinforcement learning; if those are new, start with the NASA engine-data project.
The sequence below is AeroEd’s suggested student exercise. Its deliverable is an experiment report with repeatable tests, plots, and a clear account of the limitations.
Know which part of the system you are testing
JSBSim supplies the flight dynamics: how a configured aircraft model moves as forces and control inputs change. Your environment connects those simulated states and controls to the learning algorithm. The resulting policy chooses actions from observations.
If you found the project by searching for gym-jsbsim, note the interface difference. That repository’s examples use the older OpenAI Gym API. AeroEd’s project has you build a wrapper using Gymnasium. Check the environment’s API before combining examples from different tutorials.
Keep a short record of your Python, JSBSim, Gymnasium, and Stable-Baselines3 versions, plus the aircraft model and environment code revision. A repeatable experiment needs all of them.
Define one task before you train
Start with holding a target altitude and heading from a settled flight condition. Choose the aircraft configuration, initial airspeed, atmosphere settings, and duration before comparing controllers. A 60-second simulated flight can be a manageable first debugging exercise; that duration is a suggested starting point, not a performance standard.
Write down which observations the policy receives and which controls it can change. For each value, record its units, sign convention, and range. JSBSim exposes simulation values through its property system; check the properties used by your selected model and wrapper.
Test one easily missed calculation by hand: a heading of 359 degrees and a target of 1 degree are only 2 degrees apart around the circle. An error calculation that reports 358 degrees would give the controller a misleading observation or reward.
Check the environment before trusting the training curve
Gymnasium’s environment API defines what your wrapper must return. A reset supplies an observation and an information dictionary. A step supplies the next observation, reward, separate termination and truncation flags, and an information dictionary. Confirm that observations and actions fit their declared spaces.
Run Stable-Baselines3’s check_env(env) on the environment you intend to train. The custom-environment documentation shows how to import and use it. Use the checker to catch interface errors, then test the physics and task behavior separately.
For the next check, replay a short, fixed sequence of inputs and save the resulting state values. Inspect them for missing values, unexpected jumps, or controls that behave differently from your documented mapping. Repeat the run from the same initial condition before introducing learning.
Record why each episode ends. A failure condition defined by your task belongs in its termination logic. An external cutoff used to stop an otherwise continuing task belongs in truncation. Gymnasium also distinguishes an intrinsic, finite task horizon from an external time limit; the former requires representing remaining time in the observation. Follow its time-limit explanation rather than treating every stopped flight as the same event.
Give the learned controller a fair comparison
Choose a baseline that attempts the same task, such as a conventional feedback controller whose output responds to altitude and heading errors. Document its settings and tune it on your development cases. A controller that holds fixed inputs can help debug the simulation, but label that limited role if you include it in a comparison.
Create a separate evaluation environment. Stable-Baselines3 recommends evaluating separately from training, checking wrappers that change rewards or episode lengths, and running multiple experiments because results can depend on the random seed. See its evaluation guidance.
For this project, save three groups of scenarios:
- Development cases for adjusting rewards, controller settings, and code.
- Final test cases with starting conditions you reserve until those choices are fixed.
- Challenge cases that deliberately change a condition, such as the configured wind, and are reported separately.
Run each controller on the same saved scenarios. Record initial states and any disturbance sequences as well as seeds. Gymnasium’s reset seed does not automatically seed every external component, and its action space has a separate seeding method. Check the API’s seeding notes and verify repeatability in your own wrapper.
Measure the flight as well as the reward
Reward reflects the objective you programmed. To explain what happened in the simulated flight, we suggest recording these additional quantities for every trial:
| Record | What it helps you explain |
|---|---|
| Altitude error over time | How far the model departs from its target, in stated units. |
| Wrapped heading error over time | Whether it maintains direction, including near the 0/360-degree boundary. |
| Control inputs over time | Whether the controller oscillates or repeatedly reaches an input limit. |
| Duration and ending reason | Whether the trial reaches its planned cutoff or ends on a task condition. |
You can summarize altitude and heading errors with root mean square error: square the sampled errors, average them, then take the square root. Use the same sampling interval and report duration alongside the score. A short failed flight with a small error before failure should remain visible in the results.
Show results for every trial, including failures. Distinguish variation across test scenarios from variation across separately trained policies. If you only trained once, say so; your evaluation does not yet establish how reliably training produces that result.
Turn a failure into portfolio evidence
Choose one failure and show its sequence: the starting condition, the first visible departure from the target, the control response, and the ending reason. Then state a specific hypothesis you could test. For example, does a heading-boundary error disappear after correcting the angle calculation while keeping the scenario unchanged?
Package your work with the environment configuration, saved scenarios, baseline settings, policy checkpoint, plots, and commands needed to reproduce the comparison. Explain which parts you wrote and which came from libraries or other assistance. A reader should be able to trace a reported result back to a run.
Keep the conclusion tied to that evidence. JSBSim’s maintainers describe the bundled models of named aircraft as approximations based on public information for educational or entertainment use. A result with one model and set of scenarios establishes a simulation result, not readiness to control a real aircraft. Model scope.
The aerospace engineering pathway connects this work to flight dynamics and controls. When preparing an application, use the internship guide to find roles, then choose the project evidence that addresses their requirements.
Your next step is small: write the task definition and produce one repeatable simulation trace. Save the JSBSim project to My Flight Plan when you are ready to build out the comparison.
Sources checked September 30, 2026, with AI assistance. The task, scenario groups, and reporting checklist are AeroEd’s educational suggestions. This article reports no trained-policy benchmark or flight-test result.