A Unity 3D underwater simulation to generate synthetic, labeled data for computer vision.
The simulation was built using Unity 2022 LTS. It uses the HDRP render pipeline and Unity Perception. See this tutorial to get started with Unity Perception.
The simulation is setup to generate synthetic images of the gate and buoy with automatically generated bounding boxes for RoboSub glyphs. The simulation occurs underwater, in a simulated swimming pool.
The camera captures the gate and buoy from a variety of different distances and angles. In addition to the camera movements, the sun's position, buoy and gate glyphs, and gate boxes are also randomized. This ensures that the camera captures the gate and buoy under a variety of different conditions leading to a more robust CV model.
The code is written in C# in the following files:
- CameraController.cs: Controls the camera movements.
- RandomizeGateHue.cs: Randomizes the hue of the gate.
- RotateGlyphs.cs: Randomizes the order and orientation of the buoy glyphs.
- RandomizeGateGlyphs.cs: Randomizes the order and orientation of the gate glyphs.
- RandomizeBoxes.cs: Randomizes the order of the gate boxes.
- Sun.cs: Randomizes the sun's position.
Once Miniconda is installed, create the cv-sim environment with:
conda env create -f environment.ymlThen, activate the cv-sim environment with:
conda activate cv-simLastly, due to a backwards compatiability issue with opencv, downgrade opencv with:
pip install opencv-python==4.8.0.74An error about dependency resolution may appear. This can be safely ignored.
In the future, only the conda activate cv-sim command needs to be executed to set up the Python environment for this repository.
Important
Before running the simulation for the first time, the dataset generation path needs to be specified. Open the Project Settings window by selecting the menu Edit > Project Settings... and switch to the Perception pane. Change the Base Path to the Datasets directory of this repository.
Use the play button at the top of the Unity editor to run the simulation. The following option controls whether the simulation saves generated images and their corresponding bounding boxes to disk:
- Open the inspector for "Main Camera"
- Scroll down in the inspector to "Perception Camera (Script)"
- Check/uncheck "Save Camera RGB Output to Disk"
Warning
Generated datasets can occupy a lot of disk space.
Each time the simulation is run with the above option checked, a new folder is created inside the Datasets directory.
The new folder contains the generated images and bounding boxes in SOLO format, created by Unity.
To visualize a SOLO dataset, use Voxel51:
pysolotools-fiftyone "Datasets/<SOLO Dataset>"Note
The viewer will initially open with a subset of frames visible. The data is still being imported in the background. Once all frames are imported, click the "FiftyOne" button in the top left to update the viewer.
To convert from SOLO to COCO format, run:
solo2coco "Datasets/<SOLO Dataset>" <Output Path>Important
solo2coco creates the real coco dataset inside an extraneous coco directory. In the following steps, either specify coco/coco instead of just coco or remove the unnecessary parent folder.
Many converters exist to convert COCO to other formats.
Navigate to <COCO dataset>/images and manually delete any poor quality images. Unity Perception usually outputs some bad images at the beginning of generation.
Use bbox_filter.py to filter out poor quality bounding boxes.
python Datasets/bbox_filter.py <COCO dataset path>Upload a COCO dataset to Roboflow to easily train a model or share the dataset with others.
python Datasets/roboflow_upload.py <COCO dataset path>Images that are successfully uploaded are moved to the success directory. You may need to rerun roboflow_upload.py if any images fail to upload.
For a complete guide on our CV workflow, visit Computer Vision Workflow on our wiki.