274 lines
11 KiB
Markdown
274 lines
11 KiB
Markdown
|
# Getting Started
|
||
|
|
The dataset configs are located within [tools/cfgs/dataset_configs](../tools/cfgs/dataset_configs),
|
||
|
|
and the model configs are located within [tools/cfgs](../tools/cfgs) for different datasets.
|
||
|
|
|
||
|
|
|
||
|
|
## Dataset Preparation
|
||
|
|
|
||
|
|
Currently we provide the dataloader of KITTI, NuScenes, Waymo, Lyft and Pandaset. If you want to use a custom dataset, Please refer to our [custom dataset template](CUSTOM_DATASET_TUTORIAL.md).
|
||
|
|
|
||
|
|
### KITTI Dataset
|
||
|
|
* Please download the official [KITTI 3D object detection](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) dataset and organize the downloaded files as follows (the road planes could be downloaded from [[road plane]](https://drive.google.com/file/d/1d5mq0RXRnvHPVeKx6Q612z0YRO1t2wAp/view?usp=sharing), which are optional for data augmentation in the training):
|
||
|
|
* If you would like to train [CaDDN](../tools/cfgs/kitti_models/CaDDN.yaml), download the precomputed [depth maps](https://drive.google.com/file/d/1qFZux7KC_gJ0UHEg-qGJKqteE9Ivojin/view?usp=sharing) for the KITTI training set
|
||
|
|
* NOTE: if you already have the data infos from `pcdet v0.1`, you can choose to use the old infos and set the DATABASE_WITH_FAKELIDAR option in tools/cfgs/dataset_configs/kitti_dataset.yaml as True. The second choice is that you can create the infos and gt database again and leave the config unchanged.
|
||
|
|
|
||
|
|
```
|
||
|
|
OpenPCDet
|
||
|
|
├── data
|
||
|
|
│ ├── kitti
|
||
|
|
│ │ │── ImageSets
|
||
|
|
│ │ │── training
|
||
|
|
│ │ │ ├──calib & velodyne & label_2 & image_2 & (optional: planes) & (optional: depth_2)
|
||
|
|
│ │ │── testing
|
||
|
|
│ │ │ ├──calib & velodyne & image_2
|
||
|
|
├── pcdet
|
||
|
|
├── tools
|
||
|
|
```
|
||
|
|
|
||
|
|
* Generate the data infos by running the following command:
|
||
|
|
```python
|
||
|
|
python -m pcdet.datasets.kitti.kitti_dataset create_kitti_infos tools/cfgs/dataset_configs/kitti_dataset.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
### NuScenes Dataset
|
||
|
|
* Please download the official [NuScenes 3D object detection dataset](https://www.nuscenes.org/download) and
|
||
|
|
organize the downloaded files as follows:
|
||
|
|
```
|
||
|
|
OpenPCDet
|
||
|
|
├── data
|
||
|
|
│ ├── nuscenes
|
||
|
|
│ │ │── v1.0-trainval (or v1.0-mini if you use mini)
|
||
|
|
│ │ │ │── samples
|
||
|
|
│ │ │ │── sweeps
|
||
|
|
│ │ │ │── maps
|
||
|
|
│ │ │ │── v1.0-trainval
|
||
|
|
├── pcdet
|
||
|
|
├── tools
|
||
|
|
```
|
||
|
|
|
||
|
|
* Install the `nuscenes-devkit` with version `1.0.5` by running the following command:
|
||
|
|
```shell script
|
||
|
|
pip install nuscenes-devkit==1.0.5
|
||
|
|
```
|
||
|
|
|
||
|
|
* Generate the data infos by running the following command (it may take several hours):
|
||
|
|
```python
|
||
|
|
# for lidar-only setting
|
||
|
|
python -m pcdet.datasets.nuscenes.nuscenes_dataset --func create_nuscenes_infos \
|
||
|
|
--cfg_file tools/cfgs/dataset_configs/nuscenes_dataset.yaml \
|
||
|
|
--version v1.0-trainval
|
||
|
|
|
||
|
|
# for multi-modal setting
|
||
|
|
python -m pcdet.datasets.nuscenes.nuscenes_dataset --func create_nuscenes_infos \
|
||
|
|
--cfg_file tools/cfgs/dataset_configs/nuscenes_dataset.yaml \
|
||
|
|
--version v1.0-trainval \
|
||
|
|
--with_cam
|
||
|
|
```
|
||
|
|
|
||
|
|
### Waymo Open Dataset
|
||
|
|
* Please download the official [Waymo Open Dataset](https://waymo.com/open/download/),
|
||
|
|
including the training data `training_0000.tar~training_0031.tar` and the validation
|
||
|
|
data `validation_0000.tar~validation_0007.tar`.
|
||
|
|
* Unzip all the above `xxxx.tar` files to the directory of `data/waymo/raw_data` as follows (You could get 798 *train* tfrecord and 202 *val* tfrecord ):
|
||
|
|
```
|
||
|
|
OpenPCDet
|
||
|
|
├── data
|
||
|
|
│ ├── waymo
|
||
|
|
│ │ │── ImageSets
|
||
|
|
│ │ │── raw_data
|
||
|
|
│ │ │ │── segment-xxxxxxxx.tfrecord
|
||
|
|
| | | |── ...
|
||
|
|
| | |── waymo_processed_data_v0_5_0
|
||
|
|
│ │ │ │── segment-xxxxxxxx/
|
||
|
|
| | | |── ...
|
||
|
|
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1/ (old, for single-frame)
|
||
|
|
│ │ │── waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1.pkl (old, for single-frame)
|
||
|
|
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_global.npy (optional, old, for single-frame)
|
||
|
|
│ │ │── waymo_processed_data_v0_5_0_infos_train.pkl (optional)
|
||
|
|
│ │ │── waymo_processed_data_v0_5_0_infos_val.pkl (optional)
|
||
|
|
| | |── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_multiframe_-4_to_0 (new, for single/multi-frame)
|
||
|
|
│ │ │── waymo_processed_data_v0_5_0_waymo_dbinfos_train_sampled_1_multiframe_-4_to_0.pkl (new, for single/multi-frame)
|
||
|
|
│ │ │── waymo_processed_data_v0_5_0_gt_database_train_sampled_1_multiframe_-4_to_0_global.np (new, for single/multi-frame)
|
||
|
|
|
||
|
|
├── pcdet
|
||
|
|
├── tools
|
||
|
|
```
|
||
|
|
* Install the official `waymo-open-dataset` by running the following command:
|
||
|
|
```shell script
|
||
|
|
pip3 install --upgrade pip
|
||
|
|
# tf 2.0.0
|
||
|
|
pip3 install waymo-open-dataset-tf-2-5-0 --user
|
||
|
|
```
|
||
|
|
|
||
|
|
* Extract point cloud data from tfrecord and generate data infos by running the following command (it takes several hours,
|
||
|
|
and you could refer to `data/waymo/waymo_processed_data_v0_5_0` to see how many records that have been processed):
|
||
|
|
```python
|
||
|
|
# only for single-frame setting
|
||
|
|
python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos \
|
||
|
|
--cfg_file tools/cfgs/dataset_configs/waymo_dataset.yaml
|
||
|
|
|
||
|
|
# for single-frame or multi-frame setting
|
||
|
|
python -m pcdet.datasets.waymo.waymo_dataset --func create_waymo_infos \
|
||
|
|
--cfg_file tools/cfgs/dataset_configs/waymo_dataset_multiframe.yaml
|
||
|
|
# Ignore 'CUDA_ERROR_NO_DEVICE' error as this process does not require GPU.
|
||
|
|
```
|
||
|
|
|
||
|
|
Note that you do not need to install `waymo-open-dataset` if you have already processed the data before and do not need to evaluate with official Waymo Metrics.
|
||
|
|
|
||
|
|
### Argoverse2 Dataset
|
||
|
|
* Download the **Argoverse 2 Sensor Dataset** from the [official website](https://www.argoverse.org/av2.html#download-link), and then extract them.
|
||
|
|
* Install the official API of Argoverse 2
|
||
|
|
```shell
|
||
|
|
pip install av2==0.2.0
|
||
|
|
```
|
||
|
|
* Generate info files for `train` and `val`.
|
||
|
|
```python
|
||
|
|
python -m pcdet.datasets.argo2.argo2_dataset --root_path data/argo2/sensor --output_dir data/argo2
|
||
|
|
```
|
||
|
|
- Note that this [issue](https://github.com/argoverse/av2-api/issues/102) from the argo2 api might be noticed.
|
||
|
|
- If the CPU memory of your machine is limited, you can set `--workers=0` in the training script.
|
||
|
|
- The organized files are as follows:
|
||
|
|
```
|
||
|
|
OpenPCDet
|
||
|
|
├── data
|
||
|
|
│ ├── argo2
|
||
|
|
│ │ │── ImageSets
|
||
|
|
│ │ │ ├──train.txt & val.txt
|
||
|
|
│ │ │── training
|
||
|
|
│ │ │ ├──velodyne
|
||
|
|
│ │ │── sensor
|
||
|
|
│ │ │ ├──val
|
||
|
|
│ │ │── argo2_infos_train.pkl
|
||
|
|
│ │ │── argo2_infos_val.pkl
|
||
|
|
│ │ │── val_anno.feather
|
||
|
|
|
||
|
|
├── pcdet
|
||
|
|
├── tools
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
### ONCE Dataset
|
||
|
|
* Please download train/val/test of the official [ONCE Dataset](https://once-for-auto-driving.github.io/download.html#downloads) and
|
||
|
|
organize the downloaded files as follows:
|
||
|
|
* Note that the whole dataset is large (2TB) and most scenes are unlabeled, so if you only need ONCE for supervised 3D object detection and model development, you can just download the training/validation/testing split. If you use ONCE for semi-supervised/self-supervised 3D object detection, you can choose to download the respective unlabeled splits (unlabeled small split: 100k unlabeled scenes; unlabeled medium split: 500k unlabeled scenes; unlabeled large split: 1M unlabeled scenes).
|
||
|
|
```
|
||
|
|
ONCE_Benchmark
|
||
|
|
├── data
|
||
|
|
│ ├── once
|
||
|
|
│ │ │── ImageSets
|
||
|
|
| | | ├──train.txt
|
||
|
|
| | | ├──val.txt
|
||
|
|
| | | ├──test.txt
|
||
|
|
| | | ├──raw_small.txt (100k unlabeled)
|
||
|
|
| | | ├──raw_medium.txt (500k unlabeled)
|
||
|
|
| | | ├──raw_large.txt (1M unlabeled)
|
||
|
|
│ │ │── data
|
||
|
|
│ │ │ ├──000000
|
||
|
|
| | | | |──000000.json (infos)
|
||
|
|
| | | | |──lidar_roof (point clouds)
|
||
|
|
| | | | | |──frame_timestamp_1.bin
|
||
|
|
| | | | | ...
|
||
|
|
| | | | |──cam0[1-9] (images)
|
||
|
|
| | | | | |──frame_timestamp_1.jpg
|
||
|
|
| | | | | ...
|
||
|
|
| | | | ...
|
||
|
|
├── pcdet
|
||
|
|
├── tools
|
||
|
|
```
|
||
|
|
|
||
|
|
* Generate the data infos by running the following command:
|
||
|
|
```python
|
||
|
|
python -m pcdet.datasets.once.once_dataset --func create_once_infos --cfg_file tools/cfgs/dataset_configs/once_dataset.yaml
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
### Lyft Dataset
|
||
|
|
* Please download the official [Lyft Level5 perception dataset](https://level-5.global/data/perception) and
|
||
|
|
organize the downloaded files as follows:
|
||
|
|
```
|
||
|
|
OpenPCDet
|
||
|
|
├── data
|
||
|
|
│ ├── lyft
|
||
|
|
│ │ │── ImageSets
|
||
|
|
│ │ │── trainval
|
||
|
|
│ │ │ │── data & maps(train_maps) & images(train_images) & lidar(train_lidar) & train_lidar
|
||
|
|
│ │ │── test
|
||
|
|
│ │ │ │── data & maps(test_maps) & test_images & test_lidar
|
||
|
|
├── pcdet
|
||
|
|
├── tools
|
||
|
|
```
|
||
|
|
|
||
|
|
* Install the `lyft-dataset-sdk` with version `0.0.8` by running the following command:
|
||
|
|
```shell script
|
||
|
|
pip install -U lyft_dataset_sdk==0.0.8
|
||
|
|
```
|
||
|
|
|
||
|
|
* Generate the training & validation data infos by running the following command (it may take several hours):
|
||
|
|
```python
|
||
|
|
python -m pcdet.datasets.lyft.lyft_dataset --func create_lyft_infos \
|
||
|
|
--cfg_file tools/cfgs/dataset_configs/lyft_dataset.yaml
|
||
|
|
```
|
||
|
|
* Generate the test data infos by running the following command:
|
||
|
|
```python
|
||
|
|
python -m pcdet.datasets.lyft.lyft_dataset --func create_lyft_infos \
|
||
|
|
--cfg_file tools/cfgs/dataset_configs/lyft_dataset.yaml --version test
|
||
|
|
```
|
||
|
|
|
||
|
|
* You need to check carefully since we don't provide a benchmark for it.
|
||
|
|
|
||
|
|
|
||
|
|
## Pretrained Models
|
||
|
|
If you would like to train [CaDDN](../tools/cfgs/kitti_models/CaDDN.yaml), download the pretrained [DeepLabV3 model](https://download.pytorch.org/models/deeplabv3_resnet101_coco-586e9e4e.pth) and place within the `checkpoints` directory. Please make sure the [kornia](https://github.com/kornia/kornia) is installed since it is needed for `CaDDN`.
|
||
|
|
```
|
||
|
|
OpenPCDet
|
||
|
|
├── checkpoints
|
||
|
|
│ ├── deeplabv3_resnet101_coco-586e9e4e.pth
|
||
|
|
├── data
|
||
|
|
├── pcdet
|
||
|
|
├── tools
|
||
|
|
```
|
||
|
|
|
||
|
|
## Training & Testing
|
||
|
|
|
||
|
|
|
||
|
|
### Test and evaluate the pretrained models
|
||
|
|
* Test with a pretrained model:
|
||
|
|
```shell script
|
||
|
|
python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --ckpt ${CKPT}
|
||
|
|
```
|
||
|
|
|
||
|
|
* To test all the saved checkpoints of a specific training setting and draw the performance curve on the Tensorboard, add the `--eval_all` argument:
|
||
|
|
```shell script
|
||
|
|
python test.py --cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE} --eval_all
|
||
|
|
```
|
||
|
|
|
||
|
|
* To test with multiple GPUs:
|
||
|
|
```shell script
|
||
|
|
sh scripts/dist_test.sh ${NUM_GPUS} \
|
||
|
|
--cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}
|
||
|
|
|
||
|
|
# or
|
||
|
|
|
||
|
|
sh scripts/slurm_test_mgpu.sh ${PARTITION} ${NUM_GPUS} \
|
||
|
|
--cfg_file ${CONFIG_FILE} --batch_size ${BATCH_SIZE}
|
||
|
|
```
|
||
|
|
|
||
|
|
|
||
|
|
### Train a model
|
||
|
|
You could optionally add extra command line parameters `--batch_size ${BATCH_SIZE}` and `--epochs ${EPOCHS}` to specify your preferred parameters.
|
||
|
|
|
||
|
|
|
||
|
|
* Train with multiple GPUs or multiple machines
|
||
|
|
```shell script
|
||
|
|
sh scripts/dist_train.sh ${NUM_GPUS} --cfg_file ${CONFIG_FILE}
|
||
|
|
|
||
|
|
# or
|
||
|
|
|
||
|
|
sh scripts/slurm_train.sh ${PARTITION} ${JOB_NAME} ${NUM_GPUS} --cfg_file ${CONFIG_FILE}
|
||
|
|
```
|
||
|
|
|
||
|
|
* Train with a single GPU:
|
||
|
|
```shell script
|
||
|
|
python train.py --cfg_file ${CONFIG_FILE}
|
||
|
|
```
|