[enhance]: Improve documentation of modules and dataset customization (#3821)

* update tutorials for changing training settings

* update

* Update details of hook

* resolve comments

* reformat

* Update module documentations

* add notes for classes specification

README.md

@@ -109,12 +109,11 @@ Some other methods are also supported in [projects using MMDetection](./docs/pro
 
 Please refer to [install.md](docs/install.md) for installation and dataset preparation.
 
-
 ## Getting Started
 
 Please see [getting_started.md](docs/getting_started.md) for the basic usage of MMDetection.
 We provide [colab tutorial](demo/MMDet_Tutorial.ipynb) for beginners.
-There are also tutorials for [finetuning models](docs/tutorials/finetune.md), [adding new dataset](docs/tutorials/new_dataset.md), [designing data pipeline](docs/tutorials/data_pipeline.md), and [adding new modules](docs/tutorials/new_modules.md).
+There are also tutorials for [finetuning models](docs/tutorials/finetune.md), [adding new dataset](docs/tutorials/new_dataset.md), [designing data pipeline](docs/tutorials/data_pipeline.md), [customizing models](docs/tutorials/customize_models.md), and [customizing runtime settings](docs/tutorials/customize_runtime.md).
 
 For trouble shooting, please refer to [trouble_shooting.md](docs/trouble_shooting.md)
 
@@ -127,7 +126,6 @@ We appreciate all contributions to improve MMDetection. Please refer to [CONTRIB
 MMDetection is an open source project that is contributed by researchers and engineers from various colleges and companies. We appreciate all the contributors who implement their methods or add new features, as well as users who give valuable feedbacks.
 We wish that the toolbox and benchmark could serve the growing research community by providing a flexible toolkit to reimplement existing methods and develop their own new detectors.
 
-
 ## Citation
 
 If you use this toolbox or benchmark in your research, please cite this project.
@@ -146,7 +144,6 @@ If you use this toolbox or benchmark in your research, please cite this project.
 }
 ```
 
-
 ## Contact
 
 This repo is currently maintained by Kai Chen ([@hellock](http://github.com/hellock)), Yuhang Cao ([@yhcao6](https://github.com/yhcao6)), Wenwei Zhang ([@ZwwWayne](https://github.com/ZwwWayne)),

docs/tutorials/new_dataset.md → docs/tutorials/customize_dataset.md

-# Tutorial 2: Adding New Dataset
+# Tutorial 2: Customize Datasets
 
-## Customize datasets by reorganizing data
+## Support new data format
 
-### Reorganize dataset to existing format
+To support a new data format, you can either convert them to existing formats (COCO format or PASCAL format) or directly convert them to the middle format. You could also choose to convert them offline (before training by a script) or online (implement a new dataset and do the conversion at training). In MMDetection, we recommand to convert the data into COCO formats and do the conversion offline, thus you only need to modify the config's data annotation pathes and classes after the conversion to your data.
+
+### Reorganize new data formats to existing format
 
 The simplest way is to convert your dataset to existing dataset formats (COCO or PASCAL VOC).
 
@@ -42,6 +44,7 @@ The annotation json files in COCO format has the following necessary keys:
 ```
 
 There are three necessary keys in the json file:
+
 - `images`: contains a list of images with their informations like `file_name`, `height`, `width`, and `id`.
 - `annotations`: contains the list of instance annotations.
 - `categories`: contains the list of categories names and their ID.
@@ -80,7 +83,12 @@ data = dict(
 
 We use this way to support CityScapes dataset. The script is in [cityscapes.py](https://github.com/open-mmlab/mmdetection/blob/master/tools/convert_datasets/cityscapes.py) and we also provide the finetuning [configs](https://github.com/open-mmlab/mmdetection/blob/master/configs/cityscapes).
 
-### Reorganize dataset to middle format
+**Note**
+
+1. For instance segmentation datasets, **MMDetection only supports evaluating mask AP of dataset in COCO format for now**.
+2. It is recommanded to convert the data offline before training, thus you can still use `CocoDataset` and only need to modify the path of annotations and the training classes.
+
+### Reorganize new data format to middle format
 
 It is also fine if you do not want to convert the annotation format to COCO or PASCAL format.
 Actually, we define a simple annotation format and all existing datasets are
@@ -94,7 +102,9 @@ annotations like crowd/difficult/ignored bboxes, we use `bboxes_ignore` and `lab
 to cover them.
 
 Here is an example.
-```
+
+```python
+
 [
     {
         'filename': 'a.jpg',
@@ -208,14 +218,19 @@ dataset_A_train = dict(
 )
 ```
 
-## Customize datasets by mixing dataset
+## Customize datasets by dataset wrappers
+
+MMDetection also supports many dataset wrappers to mix the dataset or modify the dataset distribution for training.
+Currently it supports to three dataset wrappers as below:
 
-MMDetection also supports to mix dataset for training.
-Currently it supports to concat and repeat datasets.
+- `RepeatDataset`: simply repeat the whole dataset.
+- `ClassBalancedDataset`: repeat dataset in a class balanced manner.
+- `ConcatDataset`: concat datasets.
 
 ### Repeat dataset
 
 We use `RepeatDataset` as wrapper to repeat the dataset. For example, suppose the original dataset is `Dataset_A`, to repeat it, the config looks like the following
+
 ```python
 dataset_A_train = dict(
         type='RepeatDataset',
@@ -234,6 +249,7 @@ We use `ClassBalancedDataset` as wrapper to repeat the dataset based on category
 frequency. The dataset to repeat needs to instantiate function `self.get_cat_ids(idx)`
 to support `ClassBalancedDataset`.
 For example, to repeat `Dataset_A` with `oversample_thr=1e-3`, the config looks like the following
+
 ```python
 dataset_A_train = dict(
         type='ClassBalancedDataset',
@@ -245,6 +261,7 @@ dataset_A_train = dict(
         )
     )
 ```
+
 You may refer to [source code](../../mmdet/datasets/dataset_wrappers.py) for details.
 
 ### Concatenate dataset
@@ -260,7 +277,9 @@ There are three ways to concatenate the dataset.
         pipeline=train_pipeline
     )
     ```
+
     If the concatenated dataset is used for test or evaluation, this manner supports to evaluate each dataset separately. To test the concatenated datasets as a whole, you can set `separate_eval=False` as below.
+
     ```python
     dataset_A_train = dict(
         type='Dataset_A',
@@ -287,6 +306,7 @@ There are three ways to concatenate the dataset.
         test = dataset_A_test
         )
     ```
+
     If the concatenated dataset is used for test or evaluation, this manner also supports to evaluate each dataset separately.
 
 3. We also support to define `ConcatDataset` explicitly as the following.
@@ -304,13 +324,14 @@ There are three ways to concatenate the dataset.
             datasets=[dataset_A_val, dataset_B_val],
             separate_eval=False))
     ```
+
     This manner allows users to evaluate all the datasets as a single one by setting `separate_eval=False`.
 
 **Note:**
+
 1. The option `separate_eval=False` assumes the datasets use `self.data_infos` during evaluation. Therefore, COCO datasets do not support this behavior since COCO datasets do not fully rely on `self.data_infos` for evaluation. Combining different types of ofdatasets and evaluating them as a whole is not tested thus is not suggested.
 2. Evaluating `ClassBalancedDataset` and `RepeatDataset` is not supported thus evaluating concatenated datasets of these types is also not supported.
 
-
 A more complex example that repeats `Dataset_A` and `Dataset_B` by N and M times, respectively, and then concatenates the repeated datasets is as the following.
 
 ```python
@@ -353,12 +374,12 @@ data = dict(
 
 ```
 
-### Modify classes of existing dataset
+## Modify Dataset Classes
 
-With existing dataset types, we can modify the class names of them to train subset of the dataset.
+With existing dataset types, we can modify the class names of them to train subset of the annotations.
 For example, if you want to train only three classes of the current dataset,
 you can modify the classes of dataset.
-The dataset will subtract subset of the data which contains at least one class in the `classes`.
+The dataset will filter out the ground truth boxes of other classes automatically.
 
 ```python
 classes = ('person', 'bicycle', 'car')
@@ -378,6 +399,7 @@ car
 ```
 
 Users can set the classes as a file path, the dataset will load it and convert it to a list automatically.
+
 ```python
 classes = 'path/to/classes.txt'
 data = dict(
@@ -385,3 +407,8 @@ data = dict(
     val=dict(classes=classes),
     test=dict(classes=classes))
 ```
+
+**Note**:
+
+- Before MMDetection v2.5.0, the dataset will filter out the empty GT images automatically if the classes are set and there is no way to disable that through config. This is an undesirable behavior and introduces confusion because if the classes are not set, the dataset only filter the empty GT images when `filter_empty_gt=True` and `test_mode=False`. After MMDetection v2.5.0, we decouple the image filtering process and the classes modification, i.e., the dataset will only filter empty GT images when `filter_empty_gt=True` and `test_mode=False`, no matter whether the classes are set. Thus, setting the classes only influences the annotations of classes used for training and users could decide whether to filter empty GT images by themselves.
+-

docs/tutorials/customize_losses.md

-# tutorial 5:Customize Losses
+# Tutorial 6: Customize Losses
 
 MMDetection provides users with different loss functions. But the default configuration may be not applicable for different datasets or models, so users may want to modify a specific loss to adapt the new situation.
 

docs/tutorials/new_modules.md → docs/tutorials/customize_models.md

-# Tutorial 4: Adding New Modules
+# Tutorial 4: Customize Models
 
-## Develop new components
-
-We basically categorize model components into 4 types.
+We basically categorize model components into 5 types.
 
 - backbone: usually an FCN network to extract feature maps, e.g., ResNet, MobileNet.
 - neck: the component between backbones and heads, e.g., FPN, PAFPN.
 - head: the component for specific tasks, e.g., bbox prediction and mask prediction.
 - roi extractor: the part for extracting RoI features from feature maps, e.g., RoI Align.
+- loss: the component in head for calculating losses, e.g., FocalLoss, L1Loss, and GHMLoss.
+
+## Develop new components
 
-### Add new backbones
+### Add a new backbone
 
 Here we show how to develop new components with an example of MobileNet.
 
@@ -36,22 +37,25 @@ class MobileNet(nn.Module):
         pass
 ```
 
-#### 2. Import the module.
+#### 2. Import the module
 
 You can either add the following line to `mmdet/models/backbones/__init__.py`
+
 ```python
 from .mobilenet import MobileNet
 ```
 
 or alternatively add
+
 ```python
 custom_imports = dict(
     imports=['mmdet.models.backbones.mobilenet'],
     allow_failed_imports=False)
 ```
+
 to the config file to avoid modifying the original code.
 
-#### 3. Use the backbone in your config file.
+#### 3. Use the backbone in your config file
 
 ```python
 model = dict(
@@ -89,7 +93,7 @@ class PAFPN(nn.Module):
         pass
 ```
 
-#### 2. Import the module.
+#### 2. Import the module
 
 You can either add the following line to `mmdet/models/necks/__init__.py`,
 
@@ -104,9 +108,10 @@ custom_imports = dict(
     imports=['mmdet.models.necks.mobilenet'],
     allow_failed_imports=False)
 ```
+
 to the config file and avoid modifying the original code.
 
-#### 3. Modify the config file.
+#### 3. Modify the config file
 
 ```python
 neck=dict(
@@ -255,10 +260,12 @@ Last, the users need to add the module in
 `mmdet/models/bbox_heads/__init__.py` and `mmdet/models/roi_heads/__init__.py` thus the corresponding registry could find and load them.
 
 Alternatively, the users can add
+
 ```python
 custom_imports=dict(
     imports=['mmdet.models.roi_heads.double_roi_head', 'mmdet.models.bbox_heads.double_bbox_head'])
 ```
+
 to the config file and achieve the same goal.
 
 The config file of Double Head R-CNN is as the following
@@ -294,7 +301,6 @@ Since MMDetection 2.0, the config system supports to inherit configs such that t
 The Double Head R-CNN mainly uses a new DoubleHeadRoIHead and a new
 `DoubleConvFCBBoxHead`, the arguments are set according to the `__init__` function of each module.
 
-
 ### Add new loss
 
 Assume you want to add a new loss as `MyLoss`, for bounding box regression.
@@ -337,19 +343,24 @@ class MyLoss(nn.Module):
 ```
 
 Then the users need to add it in the `mmdet/models/losses/__init__.py`.
+
 ```python
 from .my_loss import MyLoss, my_loss
 
 ```
+
 Alternatively, you can add
+
 ```python
 custom_imports=dict(
     imports=['mmdet.models.losses.my_loss'])
 ```
+
 to the config file and achieve the same goal.
 
 To use it, modify the `loss_xxx` field.
 Since MyLoss is for regression, you need to modify the `loss_bbox` field in the head.
+
 ```python
 loss_bbox=dict(type='MyLoss', loss_weight=1.0))
 ```

docs/tutorials/index.rst

@@ -2,8 +2,8 @@
    :maxdepth: 2
 
    finetune.md
-   new_dataset.md
+   customize_dataset.md
    data_pipeline.md
-   new_modules.md
+   customize_models.md
    customize_runtime.md
    customize_losses.md