Efficientnetb1-b7 hyper parameters

[pichuang1984]

First of all thanks for the fantastic code!

I am wondering if anyone has successfully reproduce (or close to it) the results for Efficientnetb1-b7? I am able to reproduce b0 with jiefengpeng's setting:
./distributed_train.sh 8 ../ImageNet/ --model efficientnet_b0 -b 256 --sched step --epochs 500 --decay-epochs 3 --decay-rate 0.963 --opt rmsproptf --opt-eps .001 -j 8 --warmup-epochs 5 --weight-decay 1e-5 --drop 0.2 --color-jitter .06 --model-ema --lr .128

The same setting (with adjusted drop rate) for b1 came with only 78.11 (with EMA enabled), compared to 78.8% reported in the paper.

[rwightman]

@pichuang1984 Unfortunately, nobody has shared successful h-params with the larger models and I haven't had a chance to try. If anyone does, I'll start a collection of hparams on the README (with attribution) for any shared successes.

A few things to try:

• Google trains with a different batch norm epsilon and momentum than PyTorch defaults. You can use the flag --bn-tf to enable both. I have trained with the PyTorch defaults for other models, but perhaps it has more impact for larger models? Downside to this is that you always need to set the epsilon to non-default for inference as well (something to remember for deployment).
• I assume you enabled drop_connect in the model entrypoint as has been mentioned in other issues? kwargs['drop_connect_rate'] = 0.2
• I added experimental implementation of AutoAugmentation a while back. I tested it with a ResNet model that is faster to train and it worked well, slight improvement. Since Google had success with AutoAugment for their EfficientNet, it might have an impact? Use the flag --aa v0 for a setup that's similar to Google's efficientnet AA or --aa original for one closer to the origina AA paper. The color jitter flag is ignored when aa is set.

I added a memory efficient Swish impl last week so you should see some memory usage improvements while training...

[rwightman]

Also, with 8 * 256 effective batch size, you might want to try bumping up the the EMA decay a notch .. maybe --model-ema-decay 0.9999 (default is 0.9998)

[pichuang1984]

Thanks for the detailed feedback. I am in the process of trying couple settings and will get back to you if any of these runs are (or not) able to reproduce the reported accuracy.

[pichuang1984]

i am getting a pickle error when trying to use the autoAugment. It seems that since it is a lambda function it is not pickable during multiprocessing (distributed training). I am curious if you have encountered this issue before?

[rwightman]

@pichuang1984 I'm aware of pickle concerns with lambdas but haven't run into the problem. I can run distributed training w/ AA enabled (on one Linux machine with multiple GPUs) and don't have any problems.

I'm not sure it's the distributed processes though, since there shouldn't be any sharing of dataset / transforms for that. Seems more likely to be the main process sharing the dataset with the worker processes... I know people have had issues with lambdas + transforms on Windows with PyTorch... I'll make a note to replace them with normal functions sometime next week but you can probably do it yourself quicker.

[pichuang1984]

@rwightman I am actually using a customized training script so that might be why. Not sure exactly what the problem is yet but your point makes sense. I will continue debugging as well as working on replacing them. Also looking forward to your implementation.

[pichuang1984]

I am able to closely reproduce B1 results (top1/top5: 78.74/94.4) based on the parms from https://arxiv.org/pdf/1908.06022.pdf. Here are some settings I use/derive:
decay_epoch: 2.4 decay_rate: 0.99 color_jitter: 0.1 base_lr = 0.016 ema_decay = 0.9998
The actual learning rate is: (total # of images) / 256 * 0.016 per iteration

I also tried my implementation of the auto augment but didn't see an accuracy increase.

[rwightman]

@pichuang1984 Thanks for the update. Did you switch the LR scheduler setup so that it's adjust on updates and not epochs?

[pichuang1984]

@rwightman No i didn't, still using your implementation based on epochs. I am also using pytorch BN default instead of TF. Using TF BN parms actually drop the accuracy by 0.8% for B1

The same receipt doesn't seen to work for B2 though. I am still ~ 1% short from the paper. Any recommendation will be appreciated for B2/B3 and so on.

[rwightman]

@pichuang1984 I just finished training B2 to 80.4%. Weights added and hparms posted in the 'what's new', I used my new RandAugment implemenation.

Something to watch out for when using distributed training is that the distributed validation while it's training (and thus checkpoint selection) doesn't necessarily match up exactly the actual validation from the saved weights. This can be sometimes improved or mitigated by averaging a few checkpoints after the fact or keeping a larger number of checkpoints and validating against all to find the best afterwards.

[michaelklachko]

@rwightman how long did it take to train B2 from scratch? How many GPUs did you use, and which ones?

[pichuang1984]

@rwightman This is very cool! I have not been able to reproduce results for B2/B3. So far my best results are B2: 79.7% and B3: 81.1%. Will try your hparams.

Is this caused by the difference between running validation on single GPU vs distributed? Do you think if we enforce the validation to run on only single GPU will that mitigate this problem?

[rwightman]

@michaelklachko not super quick, 13 days on two Titan RTX running in FP16 with AMP at near full tilt (95% utilization).. I got impatient and killed training at epoch 420 or so and went through the results. Things had levelled off for a while by that point.

@pichuang1984 I think the biggest reason is the batchnorm running stats, they remain on on each distributed node separately and aren't synched. When validation is performed each GPU is using slightly different BN stats for the validation, the stats from rank=0 end up being saved in the checkpoint. One can use sync-bn but it really slows things down as the BN calcs are synced each batch, and can actually hurt training with larger batch sizes. I have tried looking for an implemenation that maybe syncs the stats once per epoch, after the training iteration, before validation and model saving, but I didn't find anything like that. I may try it myself someday and see if it helps...

[rwightman]

But I must say, with the addition of the JIT + memory optimized Swish implementation I added, it has made this process quicker, allowed slightly larger batch sizes, probably would have been a number of days more before I made that change.

[pichuang1984]

@rwightman Ah I see. I think there is an all_reduce call in Pytorch (https://pytorch.org/docs/stable/distributed.html) that we can utilize? We can just sync up the BN parameters once before validation instead of every mini-batch.