Frequently Asked Questions¶
We list some common troubles faced by many users and their corresponding solutions here. Feel free to enrich the list if you find any frequent issues and have ways to help others to solve them. If the contents here do not cover your issue, please create an issue using the provided templates and make sure you fill in all required information in the template.
Compatibility issue between MMCV and MMDetection; “ConvWS is already registered in conv layer”; “AssertionError: MMCV==xxx is used but incompatible. Please install mmcv>=xxx, <=xxx.”
Please install the correct version of MMCV for the version of your MMDetection following the installation instruction.
“No module named ‘mmcv.ops’”; “No module named ‘mmcv._ext’”.
Uninstall existing mmcv in the environment using
pip uninstall mmcv.
Install mmcv-full following the installation instruction.
“RTX 30 series card fails when building MMCV or MMDet”
Temporary work-around: do
MMCV_WITH_OPS=1 MMCV_CUDA_ARGS='-gencode=arch=compute_80,code=sm_80' pip install -e .. The common issue is
nvcc fatal : Unsupported gpu architecture 'compute_86'. This means that the compiler should optimize for sm_86, i.e., nvidia 30 series card, but such optimizations have not been supported by CUDA toolkit 11.0. This work-around modifies the compile flag by adding
MMCV_CUDA_ARGS='-gencode=arch=compute_80,code=sm_80', which tells
nvccto optimize for sm_80, i.e., Nvidia A100. Although A100 is different from the 30 series card, they use similar ampere architecture. This may hurt the performance but it works.
PyTorch developers have updated that the default compiler flags should be fixed by pytorch/pytorch#47585. So using PyTorch-nightly may also be able to solve the problem, though we have not tested it yet.
“invalid device function” or “no kernel image is available for execution”.
Check if your cuda runtime version (under
conda list cudatoolkitversion match.
python mmdet/utils/collect_env.pyto check whether PyTorch, torchvision, and MMCV are built for the correct GPU architecture. You may need to set
TORCH_CUDA_ARCH_LISTto reinstall MMCV. The GPU arch table could be found here, i.e. run
TORCH_CUDA_ARCH_LIST=7.0 pip install mmcv-fullto build MMCV for Volta GPUs. The compatibility issue could happen when using old GPUS, e.g., Tesla K80 (3.7) on colab.
Check whether the running environment is the same as that when mmcv/mmdet has compiled. For example, you may compile mmcv using CUDA 10.0 but run it on CUDA 9.0 environments.
“undefined symbol” or “cannot open xxx.so”.
If those symbols are CUDA/C++ symbols (e.g., libcudart.so or GLIBCXX), check whether the CUDA/GCC runtimes are the same as those used for compiling mmcv, i.e. run
python mmdet/utils/collect_env.pyto see if
"MMCV CUDA Compiler"is the same as
If those symbols are PyTorch symbols (e.g., symbols containing caffe, aten, and TH), check whether the PyTorch version is the same as that used for compiling mmcv.
python mmdet/utils/collect_env.pyto check whether PyTorch, torchvision, and MMCV are built by and running on the same environment.
setuptools.sandbox.UnpickleableException: DistutilsSetupError(“each element of ‘ext_modules’ option must be an Extension instance or 2-tuple”)
If you are using miniconda rather than anaconda, check whether Cython is installed as indicated in #3379. You need to manually install Cython first and then run command
pip install -r requirements.txt.
You may also need to check the compatibility between the
PyTorchin your environment.
Check you GCC version and use GCC 5.4. This usually caused by the incompatibility between PyTorch and the environment (e.g., GCC < 4.9 for PyTorch). We also recommend the users to avoid using GCC 5.5 because many feedbacks report that GCC 5.5 will cause “segmentation fault” and simply changing it to GCC 5.4 could solve the problem.
Check whether PyTorch is correctly installed and could use CUDA op, e.g. type the following command in your terminal.
python -c 'import torch; print(torch.cuda.is_available())'
And see whether they could correctly output results.
If Pytorch is correctly installed, check whether MMCV is correctly installed.
python -c 'import mmcv; import mmcv.ops'
If MMCV is correctly installed, then there will be no issue of the above two commands.
If MMCV and Pytorch is correctly installed, you man use
pdbto set breakpoints or directly add ‘print’ in mmdetection code and see which part leads the segmentation fault.
“Loss goes Nan”
Check if the dataset annotations are valid: zero-size bounding boxes will cause the regression loss to be Nan due to the commonly used transformation for box regression. Some small size (width or height are smaller than 1) boxes will also cause this problem after data augmentation (e.g., instaboost). So check the data and try to filter out those zero-size boxes and skip some risky augmentations on the small-size boxes when you face the problem.
Reduce the learning rate: the learning rate might be too large due to some reasons, e.g., change of batch size. You can rescale them to the value that could stably train the model.
Extend the warmup iterations: some models are sensitive to the learning rate at the start of the training. You can extend the warmup iterations, e.g., change the
warmup_itersfrom 500 to 1000 or 2000.
Add gradient clipping: some models requires gradient clipping to stabilize the training process. The default of
None, you can add gradient clippint to avoid gradients that are too large, i.e., set
optimizer_config=dict(_delete_=True, grad_clip=dict(max_norm=35, norm_type=2))in your config file. If your config does not inherits from any basic config that contains
optimizer_config=dict(grad_clip=None), you can simply add
“GPU out of memory”
There are some scenarios when there are large amount of ground truth boxes, which may cause OOM during target assignment. You can set
gpu_assign_thr=Nin the config of assigner thus the assigner will calculate box overlaps through CPU when there are more than N GT boxes.
with_cp=Truein the backbone. This uses the sublinear strategy in PyTorch to reduce GPU memory cost in the backbone.
Try mixed precision training using following the examples in
loss_scalemight need further tuning for different models.
“RuntimeError: Expected to have finished reduction in the prior iteration before starting a new one”
This error indicates that your module has parameters that were not used in producing loss. This phenomenon may be caused by running different branches in your code in DDP mode.
You can set
find_unused_parameters = Truein the config to solve the above problems(but this will slow down the training speed.
If the version of your MMCV >= 1.4.1, you can get the name of those unused parameters with
Save the best model
It can be turned on by configuring
evaluation = dict(save_best=‘auto’). In the case of the
autoparameter, the first key in the returned evaluation result will be used as the basis for selecting the best model. You can also directly set the key in the evaluation result to manually set it, for example,
evaluation = dict(save_best='mAP' ).
Resume training with
If you use
ExpMomentumEMAHookin training, you can’t just use command line parameters
--cfg-options resume_fromto restore model parameters during resume, i.e., the command
python tools/train.py configs/yolox/yolox_s_8x8_300e_coco.py --resume-from ./work_dir/yolox_s_8x8_300e_coco/epoch_x.pthwill not work. Since
ExpMomentumEMAHookneeds to reload the weights, taking the
yolox_salgorithm as an example, you should modify the values of
resume_fromin two places of the config as below:
# Open configs/yolox/yolox_s_8x8_300e_coco.py directly and modify all resume_from fields resume_from=./work_dir/yolox_s_8x8_300e_coco/epoch_x.pth custom_hooks=[... dict( type='ExpMomentumEMAHook', resume_from=./work_dir/yolox_s_8x8_300e_coco/epoch_x.pth, momentum=0.0001, priority=49) ]
COCO Dataset, AP or AR = -1
According to the definition of COCO dataset, the small and medium areas in an image are less than 1024 (32*32), 9216 (96*96), respectively.
If the corresponding area has no object, the result of AP and AR will set to -1.
styleparameter in ResNet allows either
caffestyle. It indicates the difference in the Bottleneck module. Bottleneck is a stacking structure of
1x1-3x3-1x1convolutional layers. In the case of
caffemode, the convolution layer with
stride=2is the first
1x1convolution, while in
pyorchmode, it is the second
stride=2. A sample code is as below:
if self.style == 'pytorch': self.conv1_stride = 1 self.conv2_stride = stride else: self.conv1_stride = stride self.conv2_stride = 1
ResNeXt parameter description
ResNeXt comes from the paper
Aggregated Residual Transformations for Deep Neural Networks. It introduces group and uses “cardinality” to control the number of groups to achieve a balance between accuracy and complexity. It controls the basic width and grouping parameters of the internal Bottleneck module through two hyperparameters
cardinality. An example configuration name in MMDetection is
mask_rcnnrepresents the algorithm using Mask R-CNN,
x101represents the backbone network using ResNeXt-101, and
64x4drepresents that the bottleneck block has 64 group and each group has basic width of 4.
Since the detection model is usually large and the input image resolution is high, this will result in a small batch of the detection model, which will make the variance of the statistics calculated by BatchNorm during the training process very large and not as stable as the statistics obtained during the pre-training of the backbone network . Therefore, the
norm_eval=Truemode is generally used in training, and the BatchNorm statistics in the pre-trained backbone network are directly used. The few algorithms that use large batches are the
norm_eval=Falsemode, such as NASFPN. For the backbone network without ImageNet pre-training and the batch is relatively small, you can consider using