RealDriveSim: A Realistic Multi-Modal Multi-Task Synthetic Dataset for Autonomous Driving
Arpit Jadon*, Haoran Wang, Phillip Thomas, Michael Stanley, S. Nathaniel Cibik, Rachel Laurat, Omar Maher, Lukas Hoyer, Ozan Unal*, Dengxin Dai
Arpit Jadon*, Haoran Wang, Phillip Thomas, Michael Stanley, S. Nathaniel Cibik, Rachel Laurat, Omar Maher, Lukas Hoyer, Ozan Unal*, Dengxin Dai
As perception models continue to develop, the need for large-scale datasets increases. However, data annotation remains far too expensive to effectively scale and meet the demand. Synthetic datasets provide a solution to boost model performance with substantially reduced costs. However, current synthetic datasets remain limited in their scope, realism, and are designed for specific tasks and applications. In this work, we present RealDriveSim, a realistic multi-modal synthetic dataset for autonomous driving that not only supports popular 2D computer vision applications but also their LiDAR counterparts, providing fine-grained annotations for up to 64 classes. We extensively evaluate our dataset for a wide range of applications and domains, demonstrating state-of-the-art results compared to existing synthetic benchmarks. The dataset is publicly available at https://realdrivesim.github.io/
| Dataset | Camera | LiDAR | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Adverse-W | 2D Seg. | Sem-Cls. | 2D Det. | 3D Det. | Depth | Optical-Flow | MOT | 3D Det. | 3D Seg. | Sem-Cls. | Scene-Flow | MOT | SLAM | |
| SYNTHIA | ✓ | ✓ | 22 | ✓ | ✓ | ✓ | ✗ | ✗ | - | - | - | - | - | - |
| GTA-V | ✓ | ✓ | 19 | ✗ | ✗ | ✗ | ✗ | ✗ | - | - | - | - | - | - |
| VIPER | ✓ | ✓ | 32 | ✓ | ✓ | ✗ | ✓ | ✓ | - | - | - | - | - | - |
| Synscapes | ✗ | ✓ | 19 | ✓ | ✓ | ✓ | ✗ | ✗ | - | - | - | - | - | - |
| SHIFT | ✓ | ✓ | 23 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | - | ✗ | ✓ | ✗ |
| PreSIL | ✗ | ✓ | 12 | ✓ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | 12 | ✗ | ✗ | ✗ |
| SynLIDAR | - | - | - | - | - | - | - | - | ✗ | ✓ | 32 | ✗ | ✗ | ✗ |
| RealDriveSim | ✓ | ✓ | 61 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 64 | ✓ | ✓ | ✓ |
| Modality | Download (File Size) |
|---|---|
| 2D Modalities | |
| Images (RGB) | 367.42 GB |
| 2D Object Detection | 1.78 GB |
| 2D Semantic Segmentation | 7.75 GB |
| 2D Instance Segmentation | 2.54 GB |
| Depth Maps | 605.66 GB |
| 2D Motion Vectors / Optical Flow | 91.54 GB |
| 3D Modalities | |
| 3D Point Clouds | 430.33 GB |
| 3D Object Detection | 62 GB |
| 3D Semantic Segmentation | 3.53 GB |
| 3D Instance Segmentation | 1.29 GB |
| 3D Motion Vectors / Scene Flow | 96.95 GB |
| Other | |
| Calibration Data | 6 MB |
| Modality | Download (File Size) |
|---|---|
| 2D Modalities | |
| Images (RGB) | 73.52 GB |
| 2D Object Detection | 367.51 MB |
| 2D Semantic Segmentation | 1.55 GB |
| 2D Instance Segmentation | 522.26 MB |
| Depth Maps | 121.13 GB |
| 2D Motion Vectors / Optical Flow | 19.18 GB |
| 3D Modalities | |
| 3D Point Clouds | 86.09 GB |
| 3D Object Detection | 12.40 GB |
| 3D Semantic Segmentation | 725.52 MB |
| 3D Instance Segmentation | 268.09 MB |
| 3D Motion Vectors / Scene Flow | 19.60 GB |
| Other | |
| Calibration Data | 6 MB |
| Modality | Download (File Size) |
|---|---|
| 2D Modalities | |
| Images (RGB) | 11.54 GB |
| 2D Object Detection | 74.15 MB |
| 2D Semantic Segmentation | 323.89 MB |
| 2D Instance Segmentation | 107.68 MB |
| Depth Maps | 24.38 GB |
| 2D Motion Vectors / Optical Flow | 3.75 GB |
| 3D Modalities | |
| 3D Point Clouds | 6.16 GB |
| 3D Object Detection | 2.35 GB |
| 3D Semantic Segmentation | 75.96 MB |
| 3D Instance Segmentation | 28.14 MB |
| 3D Motion Vectors / Scene Flow | 1.46 GB |
| Other | |
| Calibration Data | 248 KB |
| Modality | Download (File Size) |
|---|---|
| 2D Modalities | |
| Images (RGB) | 4.07 GB |
| 2D Object Detection | 25.62 MB |
| 2D Semantic Segmentation | 119.61 MB |
| 2D Instance Segmentation | 37.79 MB |
| Depth Maps | 8.57 GB |
| 2D Motion Vectors / Optical Flow | 1.42 GB |
| 3D Modalities | |
| 3D Point Clouds | 2.24 GB |
| 3D Object Detection | 820.03 MB |
| 3D Semantic Segmentation | 27.68 MB |
| 3D Instance Segmentation | 9.66 MB |
| 3D Motion Vectors / Scene Flow | 554.15 MB |
| Other | |
| Calibration Data | 88 KB |
| Dataset Name | No. of Sequences | No. of Frames | Remarks |
|---|---|---|---|
| Normal Weather – Full | 6,343 | 126,680 | Contains clear weather [including cloudy and overcase conditions] images taken at different times of the day. |
| Normal Weather – Sampled | 6,343 | 25,372 | Smaller version of the full normal weather dataset with every 5th frame uniformly sampled from each full sequence. |
| Adverse Weather – Batch 1 | 258 | 5,160 | Contains foggy, night, and rainy scenes |
| Adverse Weather – Batch 2 | 90 | 1,800 | Contains foggy, night, and rainy scenes |
Note 1: The experiments in our paper were conducted using a combined dataset consisting of Normal Weather – Sampled Dataset, Adverse Weather [Batch 1], and Adverse Weather [Batch 2]. If the full Normal Weather dataset is too large for your needs, we recommend using the same sampled version as used in our experiments.
Note 2: Compared to the dataset used in the paper, we have added one additional sequence each to Adverse Weather Batch 1 and Batch 2.
Note 3: PD-SDK is no longer available or needed; the mappings above include all classes and IDs, so please use them directly.
| Label Name | Label ID |
|---|---|
| Animal | 0 |
| Bicycle | 1 |
| Bicyclist | 2 |
| Building | 3 |
| Bus | 4 |
| Car | 5 |
| Caravan/RV | 6 |
| ConstructionVehicle | 7 |
| CrossWalk | 8 |
| Fence | 9 |
| HorizontalPole | 10 |
| LaneMarking | 11 |
| LimitLine | 12 |
| Motorcycle | 13 |
| Motorcyclist | 14 |
| OtherDriveableSurface | 15 |
| OtherFixedStructure | 16 |
| OtherMovable | 17 |
| OtherRider | 18 |
| Overpass/Bridge/Tunnel | 19 |
| OwnCar(EgoCar) | 20 |
| ParkingMeter | 21 |
| Pedestrian | 22 |
| Railway | 23 |
| Road | 24 |
| RoadBarriers | 25 |
| RoadBoundary(Curb) | 26 |
| RoadMarking | 27 |
| SideWalk | 28 |
| Sky | 29 |
| TemporaryConstructionObject | 30 |
| Terrain | 31 |
| TowedObject | 32 |
| TrafficLight | 33 |
| TrafficSign | 34 |
| Train | 35 |
| Truck | 36 |
| Vegetation | 37 |
| VerticalPole | 38 |
| WheeledSlow | 39 |
| LaneMarkingOther | 40 |
| LaneMarkingGap | 41 |
| Fence(Transparent) | 42 |
| StaticObject(Trashcan) | 43 |
| Vegetation(Bush) | 44 |
| OtherPole | 45 |
| Powerline | 46 |
| SchoolBus | 47 |
| ParkingLot | 48 |
| RoadMarkingSpeed | 49 |
| Vegetation(GroundCover) | 50 |
| Vegetation(Grass) | 51 |
| Vegetation(Tree) | 52 |
| Debris | 53 |
| RoadBoundary(CurbFlat) | 54 |
| LaneMarking(Parking) | 55 |
| LaneMarking(ParkingIndicator) | 56 |
| RoadMarkingArrows | 57 |
| RoadMarkingBottsDots | 58 |
| StopLine | 59 |
| ChannelizingDevice | 60 |
| LaneMarkingSpan | 61 |
| StaticObject(BikeRack) | 62 |
| ParkingSpot | 63 |
| RoadBoundary(CurbTop) | 64 |
| RoadBoundary(CurbSide) | 65 |
| RoadBoundary(CurbRoadLevel) | 66 |
| Water | 67 |
| GuardRail | 68 |
| Wall | 69 |
| WheelStopper | 70 |
| ParkingMarker | 71 |
| Van | 103 |
| ConstructionVehicle(Truck) | 104 |
| DiffusionPrimitive | 200 |
| Custom | 201 |
| Parking Gap | 202 |
| Multipath(Noise) | 225 |
| ThermalNoise(Noise) | 226 |
| Fog(Noise) | 227 |
| Rain(Noise) | 228 |
| TireSplash(Noise) | 229 |
| Void | 255 |
Note 1: The experiments in our paper were conducted using a combined dataset consisting of Normal Weather – Sampled Dataset, Adverse Weather [Batch 1], and Adverse Weather [Batch 2]. If the full Normal Weather dataset is too large for your needs, we recommend using the same sampled version as used in our experiments.
Note 2: Compared to the dataset used in the paper, we have added one additional sequence each to Adverse Weather Batch 1 and Batch 2.
Note 3: PD-SDK is no longer available or needed; the mappings above include all classes and IDs, so please use them directly.
Paper
Dataset
