Synthetic Training Data Generation in Omniverse
Generate photorealistic synthetic datasets for aerospace computer vision using NVIDIA Omniverse
Last reviewed: March 2026Overview
Training computer vision models requires thousands to millions of labeled images — and acquiring, annotating, and licensing real-world aerospace data is expensive, slow, and sometimes impossible (on-orbit imagery, classified environments, accident scenarios). Synthetic data generation using physics-based rendering produces photorealistic images with perfect automatic annotation: every pixel's label, every object's 6-DOF pose, every depth value is known exactly — with zero human annotation cost.
NVIDIA Omniverse Replicator is the state-of-the-art platform for synthetic data generation. Built on the Universal Scene Description (USD) format, it combines path-traced photorealistic rendering with programmable randomization (domain randomization): randomize camera position, lighting conditions, object materials, background environments, and object poses to generate a near-infinite variety of training data from a single scene setup.
In this project, you'll build a synthetic data pipeline for a specific aerospace computer vision task: spacecraft pose estimation from monocular images (the SPEED+ benchmark problem from Stanford and ESA). You'll generate 10,000+ synthetic images of a spacecraft model under varied lighting and viewing conditions, train a pose estimation network, and evaluate on real images — demonstrating sim-to-real transfer capability that is the holy grail of synthetic data research.
What You'll Learn
- ✓ Set up NVIDIA Omniverse and script a synthetic data generation pipeline using Omniverse Replicator API
- ✓ Implement domain randomization across lighting (HDR environments, point lights), materials (metallic spacecraft surfaces), and camera parameters
- ✓ Generate annotated training data: bounding boxes, 6-DOF pose labels, depth maps, and instance segmentation masks
- ✓ Train a pose estimation network on synthetic data and evaluate zero-shot sim-to-real transfer on real images
- ✓ Quantify the effect of domain randomization on sim-to-real transfer: too little causes domain gap, too much degrades accuracy
Step-by-Step Guide
Set Up Omniverse and Import Spacecraft Model
Install NVIDIA Omniverse and the Isaac Sim or Replicator extension. You'll need an NVIDIA RTX GPU for path-traced rendering (RTX 3060 or better). Import a spacecraft 3D model — the TANGO satellite model used in the SPEED+ dataset is publicly available in USD format from ESA's open data portal.
Place the model in an Omniverse scene, set up a camera, and render a test frame. Verify the rendering quality: spacecraft metallic surfaces should show realistic reflections and specular highlights. The visual fidelity of your synthetic images directly determines how well models trained on them transfer to real images.
Implement Domain Randomization
Write a Replicator script that randomizes for each frame: camera position (sample uniformly on a sphere at 1–5m radius), spacecraft orientation (uniform random rotation), lighting (HDR environment maps from HDRI Haven, plus 1–3 random point lights with varied intensity and color), and background (random star field, Earth limb images, or pure black).
Randomize material properties slightly: spacecraft surface reflectivity ±20%, solar panel texture variation, thermal blanket wrinkle variation. This prevents the network from memorizing specific material properties instead of learning the underlying geometry.
Generate Annotated Data
Configure Replicator's annotators to automatically produce: RGB images, bounding boxes (2D), 6-DOF pose ground truth (position + quaternion of spacecraft relative to camera), depth maps, and instance segmentation masks. Replicator writes these to disk in KITTI or COCO format with zero human effort.
Generate 10,000 training images and 1,000 validation images. Run generation overnight — path-traced rendering at 640×480 takes 2–10 seconds per frame on an RTX GPU. For faster iteration during development, switch to rasterized rendering (lower quality but 10–100× faster).
Train a Pose Estimation Network
Train a 6-DOF pose estimation network using your synthetic dataset. Use the EfficientPose or FoundPose architecture, which is designed for spacecraft pose estimation and has published results on SPEED+. Alternatively, train a simpler pipeline: a YOLO model for bounding box detection followed by a PnP solver using detected keypoints.
Train for 50–100 epochs on your 10,000 synthetic images. Evaluate on the synthetic validation set — you should see strong performance here (average rotation error < 5°, translation error < 5% of distance). The real challenge is the next step: real image evaluation.
Evaluate Sim-to-Real Transfer
Evaluate your network on the real images from the SPEED+ dataset (spacecraft images taken in a hardware-in-the-loop facility with a real spacecraft mockup). You have trained only on synthetic data — this is a zero-shot sim-to-real evaluation.
Compute the SPEED+ metrics: Average Distance (ADD) for rotation error and Euclidean distance for translation error. Compare against baseline methods trained on real data. Analyze which error modes dominate: is the network failing on position, rotation, or both? Plot failure cases — which viewing angles or lighting conditions cause the largest errors?
Close the Sim-to-Real Gap
Apply strategies to reduce the domain gap: style transfer (apply CycleGAN to make synthetic images look more like real images before training), targeted randomization (increase randomization only in dimensions where synthetic and real images differ most — typically lighting and background), and fine-tuning on a small number of real labeled images.
Measure the improvement from each strategy. Fine-tuning on just 50–200 real images combined with the large synthetic training set typically closes 60–80% of the sim-to-real gap — demonstrating that synthetic pre-training dramatically reduces the real data requirement.
Career Connection
See how this project connects to real aerospace careers.
Space Operations →
Spacecraft pose estimation is critical for autonomous docking, debris capture, and on-orbit servicing — ESA, NASA, and JAXA are all investing in vision-based rendezvous for future missions
Aerospace Engineer →
On-board computer vision systems for autonomous spacecraft require training data that cannot be collected in orbit — synthetic data generation is the only viable approach, and engineers who can build these pipelines are in high demand
Drone & UAV Ops →
Synthetic data for detect-and-avoid, landing zone detection, and object recognition is increasingly used by UAV companies to supplement costly real flight data collection
Astronaut →
Future crewed missions to the Moon and Mars will involve manual or semi-autonomous proximity operations where computer vision systems trained on synthetic data will be critical safety tools
Go Further
Push toward state-of-the-art synthetic-to-real transfer:
- Photorealistic Earth backgrounds — integrate real Sentinel-2 or Planet Labs satellite imagery as background environments to make the synthetic on-orbit scenes more realistic
- Multi-object scenes — extend from single spacecraft to multi-object scenes (debris cloud, formation flying) and train a detection + tracking pipeline
- Active learning with synthetic data — identify which synthetic viewing conditions the network is least confident on, generate more synthetic data for those conditions specifically, and retrain iteratively
- Publish on SPEED+ — submit your approach to the SPEED+ pose estimation challenge leaderboard maintained by ESA and Stanford; top methods are published at the IAC