语义分割训练VOC格式数据集loss总是为0.000


#1

大家好,

请教下:

我使用的labelme标记的分割数据集,然后通过labelme里面提供的py脚本进行了转换,转为VOC的格式。
但不管我是用pspnet还是deeplab模型,训练总是显示loss为0.000, 然后pixacc 为一个固定值,具体log如下:

(gluonenv) ai2@ai2 ~/semSegTest $ python wmtrain.py 
Number of GPUs: 1
preparing dataset train...
preparing dataset val...
init dataloader with train dataset...
init dataloader with val dataset...
loading model....
DeepLabV3(
  (head): _DeepLabHead(
    (block): HybridSequential(
      (0): Conv2D(256 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30__deeplabhead0_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
      (2): Activation(relu)
      (3): Dropout(p = 0.1, axes=())
      (4): Conv2D(256 -> 1, kernel_size=(1, 1), stride=(1, 1))
    )
    (aspp): _ASPP(
      (project): HybridSequential(
        (0): Conv2D(1280 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30__deeplabhead0_hybridsequential5_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
        (2): Activation(relu)
        (3): Dropout(p = 0.5, axes=())
      )
      (concurent): HybridConcurrent(
        (0): HybridSequential(
          (0): Conv2D(2048 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
          (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30__deeplabhead0_hybridsequential0_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
          (2): Activation(relu)
        )
        (1): HybridSequential(
          (0): Conv2D(2048 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(12, 12), dilation=(12, 12), bias=False)
          (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30__deeplabhead0_hybridsequential1_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
          (2): Activation(relu)
        )
        (2): HybridSequential(
          (0): Conv2D(2048 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(24, 24), dilation=(24, 24), bias=False)
          (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30__deeplabhead0_hybridsequential2_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
          (2): Activation(relu)
        )
        (3): HybridSequential(
          (0): Conv2D(2048 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(36, 36), dilation=(36, 36), bias=False)
          (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30__deeplabhead0_hybridsequential3_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
          (2): Activation(relu)
        )
        (4): _AsppPooling(
          (gap): HybridSequential(
            (0): GlobalAvgPool2D(size=(1, 1), stride=(1, 1), padding=(0, 0), ceil_mode=True, global_pool=True, pool_type=avg, layout=NCHW)
            (1): Conv2D(2048 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
            (2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30__deeplabhead0_hybridsequential4_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
            (3): Activation(relu)
          )
        )
      )
    )
  )
  (conv1): HybridSequential(
    (0): Conv2D(3 -> 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
    (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=64)
    (2): Activation(relu)
    (3): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    (4): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_syncbatchnorm1_', momentum=0.9, use_global_stats=False, in_channels=64)
    (5): Activation(relu)
    (6): Conv2D(64 -> 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  )
  (relu): Activation(relu)
  (layer2): HybridSequential(
    (0): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(256 -> 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm1_', momentum=0.9, use_global_stats=False, in_channels=128)
      (conv3): Conv2D(128 -> 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm2_', momentum=0.9, use_global_stats=False, in_channels=512)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=128)
      (conv2): Conv2D(128 -> 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (relu3): Activation(relu)
      (downsample): HybridSequential(
        (0): Conv2D(256 -> 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_down2_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=512)
      )
    )
    (1): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(512 -> 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm4_', momentum=0.9, use_global_stats=False, in_channels=128)
      (conv3): Conv2D(128 -> 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm5_', momentum=0.9, use_global_stats=False, in_channels=512)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm3_', momentum=0.9, use_global_stats=False, in_channels=128)
      (conv2): Conv2D(128 -> 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (relu3): Activation(relu)
    )
    (2): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(512 -> 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm7_', momentum=0.9, use_global_stats=False, in_channels=128)
      (conv3): Conv2D(128 -> 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm8_', momentum=0.9, use_global_stats=False, in_channels=512)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm6_', momentum=0.9, use_global_stats=False, in_channels=128)
      (conv2): Conv2D(128 -> 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (relu3): Activation(relu)
    )
    (3): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(512 -> 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm10_', momentum=0.9, use_global_stats=False, in_channels=128)
      (conv3): Conv2D(128 -> 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm11_', momentum=0.9, use_global_stats=False, in_channels=512)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers2_syncbatchnorm9_', momentum=0.9, use_global_stats=False, in_channels=128)
      (conv2): Conv2D(128 -> 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (relu3): Activation(relu)
    )
  )
  (layer3): HybridSequential(
    (0): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(512 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm1_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv3): Conv2D(256 -> 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm2_', momentum=0.9, use_global_stats=False, in_channels=1024)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv2): Conv2D(256 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (relu3): Activation(relu)
      (downsample): HybridSequential(
        (0): Conv2D(512 -> 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_down3_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=1024)
      )
    )
    (1): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(1024 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm4_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv3): Conv2D(256 -> 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm5_', momentum=0.9, use_global_stats=False, in_channels=1024)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm3_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv2): Conv2D(256 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
      (relu3): Activation(relu)
    )
    (2): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(1024 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm7_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv3): Conv2D(256 -> 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm8_', momentum=0.9, use_global_stats=False, in_channels=1024)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm6_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv2): Conv2D(256 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
      (relu3): Activation(relu)
    )
    (3): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(1024 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm10_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv3): Conv2D(256 -> 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm11_', momentum=0.9, use_global_stats=False, in_channels=1024)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm9_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv2): Conv2D(256 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
      (relu3): Activation(relu)
    )
    (4): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(1024 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm13_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv3): Conv2D(256 -> 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm14_', momentum=0.9, use_global_stats=False, in_channels=1024)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm12_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv2): Conv2D(256 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
      (relu3): Activation(relu)
    )
    (5): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(1024 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm16_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv3): Conv2D(256 -> 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm17_', momentum=0.9, use_global_stats=False, in_channels=1024)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers3_syncbatchnorm15_', momentum=0.9, use_global_stats=False, in_channels=256)
      (conv2): Conv2D(256 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
      (relu3): Activation(relu)
    )
  )
  (layer1): HybridSequential(
    (0): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(128 -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm1_', momentum=0.9, use_global_stats=False, in_channels=64)
      (conv3): Conv2D(64 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm2_', momentum=0.9, use_global_stats=False, in_channels=256)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=64)
      (conv2): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (relu3): Activation(relu)
      (downsample): HybridSequential(
        (0): Conv2D(128 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_down1_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
      )
    )
    (1): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(256 -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm4_', momentum=0.9, use_global_stats=False, in_channels=64)
      (conv3): Conv2D(64 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm5_', momentum=0.9, use_global_stats=False, in_channels=256)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm3_', momentum=0.9, use_global_stats=False, in_channels=64)
      (conv2): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (relu3): Activation(relu)
    )
    (2): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(256 -> 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm7_', momentum=0.9, use_global_stats=False, in_channels=64)
      (conv3): Conv2D(64 -> 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm8_', momentum=0.9, use_global_stats=False, in_channels=256)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers1_syncbatchnorm6_', momentum=0.9, use_global_stats=False, in_channels=64)
      (conv2): Conv2D(64 -> 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (relu3): Activation(relu)
    )
  )
  (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_syncbatchnorm2_', momentum=0.9, use_global_stats=False, in_channels=128)
  (maxpool): MaxPool2D(size=(3, 3), stride=(2, 2), padding=(1, 1), ceil_mode=False, global_pool=False, pool_type=max, layout=NCHW)
  (layer4): HybridSequential(
    (0): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(1024 -> 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm1_', momentum=0.9, use_global_stats=False, in_channels=512)
      (conv3): Conv2D(512 -> 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm2_', momentum=0.9, use_global_stats=False, in_channels=2048)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=512)
      (conv2): Conv2D(512 -> 512, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2), bias=False)
      (relu3): Activation(relu)
      (downsample): HybridSequential(
        (0): Conv2D(1024 -> 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_down4_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=2048)
      )
    )
    (1): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(2048 -> 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm4_', momentum=0.9, use_global_stats=False, in_channels=512)
      (conv3): Conv2D(512 -> 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm5_', momentum=0.9, use_global_stats=False, in_channels=2048)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm3_', momentum=0.9, use_global_stats=False, in_channels=512)
      (conv2): Conv2D(512 -> 512, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
      (relu3): Activation(relu)
    )
    (2): BottleneckV1b(
      (relu1): Activation(relu)
      (conv1): Conv2D(2048 -> 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn2): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm7_', momentum=0.9, use_global_stats=False, in_channels=512)
      (conv3): Conv2D(512 -> 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm8_', momentum=0.9, use_global_stats=False, in_channels=2048)
      (relu2): Activation(relu)
      (bn1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30_resnetv1s_layers4_syncbatchnorm6_', momentum=0.9, use_global_stats=False, in_channels=512)
      (conv2): Conv2D(512 -> 512, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4), bias=False)
      (relu3): Activation(relu)
    )
  )
  (auxlayer): _FCNHead(
    (block): HybridSequential(
      (0): Conv2D(1024 -> 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (1): SyncBatchNorm(ndev=1, eps=1e-05, fix_gamma=False, key='deeplabv30__fcnhead0_hybridsequential0_syncbatchnorm0_', momentum=0.9, use_global_stats=False, in_channels=256)
      (2): Activation(relu)
      (3): Dropout(p = 0.1, axes=())
      (4): Conv2D(256 -> 1, kernel_size=(1, 1), stride=(1, 1))
    )
  )
)
create criterion
init trainer...
Starting Epoch: 0
Total Epochs: 30
  0%|                                                                                                                            | 0/80 [00:00<?, ?it/s][10:58:31] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:97: Running performance tests to find the best convolution algorithm, this can take a while... (set the environment variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)
Epoch 0, training loss 0.000: 100%|█████████████████████████████████████████████████████████████████████████████████████| 80/80 [00:45<00:00,  2.21it/s]
Epoch 0, validation pixAcc: 0.762, mIoU: 0.762: 100%|█████████████████████████████████████████████████████████████████████| 6/6 [00:00<00:00,  6.14it/s]
Epoch 1, training loss 0.000:  42%|████████████████████████████████████▏                                                | 34/80 [00:15<00:20,  2.21it/s]

我的训练配置参数是:

    dataset = 'wm_pascal'
    model = 'deeplab'
    backbone = 'resnet50'
    workers = 2
    base_size = 520
    crop_size = 480
    train_split = 'train'
    aux = True
    aux_weight = 0.5
    epochs = 30 #50
    start_epoch = 0
    batch_size = 2
    test_batch_size = 2
    lr = 0.01 # learning rate
    momentum = 0.9
    weight_decay = 0.0001
    no_wd = False # whether to remove weight decay on bias and beta/gamma for batchnorm layers.
    no_cuda = False
    ngpus = 1 # 4
    kvstore = 'device'
    dtype = 'float32'
    resume = None # path to resume file : None or a path
    #resume = 'runs/wm_pascal/psp/res101/checkpoint.params'
    checkname = 'res50'
    model_zoo = None
    eval = False
    no_val = False
    syncbn = True

请教下,出现这种情况一般什么原因,目前还是小白阶段,训练脚本由mxnet官网的train.py 进行了简单修改完成,主要是让其用我自己的dataset。

另外: 为什么gluoncv目前对于dataset的api居然不支持自定义的呢? 非得是从标准dataset官方下载的? 这样我不得不将标准的进行一个修改然后还要放入到gluoncv的源码里面去。


#2

没训练过语义分割任务, 说下dataset的问题, gluoncv中的dataset类大都是为常见数据集COCO VOC定制实现的, 他们本身就属于自定义的一种.

对于你的分割数据集, 可以继承gluon.data.Dataset来实现自定义加载, 就和gluoncv中做的事情一样.


#3

您说的可以继承 gluon.data.Dataset是可以,问题是,我自己继承之后实现的类, 并不能在gluoncv的源码之外加载, 因为,实际运行发现 训练里面的 get_segmentation_model 这个函数,还是要从gluoncv的官方源码路径里面去根据名称找dataset,然而我自己的dataset并没有什么api可以add进入那个列表里面。

gluoncv.data.__init__.py

from .cityscapes import CitySegmentation
from .pascal_voc.segmentation import VOCSegmentation
from .pascal_aug.segmentation import VOCAugSegmentation
from .ade20k.segmentation import ADE20KSegmentation
from .segbase import ms_batchify_fn
from .recordio.detection import RecordFileDetection
from .lst.detection import LstDetection
from .mixup.detection import MixupDetection
from .mypascal_voc.segmentation import WMVOCSegmentation

datasets = {
    'ade20k': ADE20KSegmentation,
    'pascal_voc': VOCSegmentation,
    'pascal_aug': VOCAugSegmentation,
    'coco' : COCOSegmentation,
    'citys' : CitySegmentation,
    'wm_pascal': WMVOCSegmentation
}

def get_segmentation_dataset(name, **kwargs):
    """Segmentation Datasets"""
    return datasets[name.lower()](**kwargs)

官方的这个默认实现,列表就是这样的。


#4

loss为0的问题,发现原因了,就是自己定义的 WMVOCSegmentation 里面,因为只有一类目标,所以错误的把NUM置为1了,然后classname列表里面忘记了加background, 导致实际训练时候模型把黑色的background认为是目标了,那么实际上计算损失永远都是没有损失了。 修正之后就ok了。


#5

请问你知道Validation pixAcc一直是1.000,mIoU一直0.500是什么原因吗?


#6

您好,请问您是怎么解决这个问题的?