Parameters used to get 94.714 top5 with efficientnet_b2?

[DannyHernandez]

Hey! I'm a researcher at OpenAI looking into trends in compute used by models. I'm excited to find this repo, since it's the only one with EfficientNet that claims to approximately reproduce the original performance.

I've got two runs going on machines with 8 P100's

./distributed_train.sh 8 /tmp/imagenet-extracted/ --model efficientnet_b0 --lr 0.035 -b 64 --drop 0.2 --img-size 224 --sched step --epochs 550 --decay-epochs 2 --decay-rate 0.975 --opt rmsproptf -j 8 --warmup-epochs 5 --warmup-lr 1e-6 --weight-decay 1e-5 --opt-eps .001 --model-ema

./distributed_train.sh 8 /tmp/imagenet-extracted/ --model efficientnet_b2 --lr 0.0175 -b 32 --drop 0.2 --img-size 224 --sched step --epochs 550 --decay-epochs 2 --decay-rate 0.975 --opt rmsproptf -j 8 --warmup-epochs 5 --warmup-lr 1e-6 --weight-decay 1e-5 --opt-eps .001 --model-ema

Where the only change I made from the parameters recommended here was scaling the learning rate you used, .27 based on the difference in batch size

I'd be very interested in what specific learning rate plus other hyperparameters you used in your efficientnet-b2 run referenced in the ReadMe, and what has worked out best in b0 runs, since the learning rate above was given for a model family rather than b0 specifically.

[rwightman]

My native B2 weights were actually fine-tuned from the original TF B2 weights. Saves a lot of time with limited resources. On my 3 gpu system, using AMP for mixed-prec (mostly for the bigger batches as they are pascal, no speed gains for FP16):

./distributed_train.sh 3 /data/x/imagenet/ --model efficientnet_b2 --lr 0.0008 --sched step --epochs 50 --decay-epochs 3 --decay-rate 0.963 --opt rmsproptf --opt-eps .001 -j 4 --warmup-epochs 0 --weight-decay 1e-5 --drop 0.25 --color-jitter 0.06 --model-ema -b 64 --amp --resume tf_efficientnet_b2.pth

I ran that 4 times, with lr .0008, .0005, .0003, .0001 as an h-param search of sorts, killed them after 30-40 epochs when convergence seemed to stop... averaged the best resulting checkpoints and ended up with the posted weights... a small improvement on the originals, less drop on ImageNet-V2 test set, but the new TF auto augment weights are much better. In the code, I uncommented the drop_connect arg, (still need to add a cmd line arg for that). I set drop_connect a little below the default for two runs at around 0.15 and .2 for the other two.

For training from scratch I'm still experimenting and trying to narrow in on better defaults. Closer to what you're looking for, I'm currently running an experiment training my custom definition of a 'MixNet XL' (see https://github.com/rwightman/pytorch-image-models/blob/mixnet_xl/timm/models/gen_efficientnet.py#L1717). Somewhere between an EffNet-B2 and B3 in size at 11.9M params. It's at just over 80 top-1 now, 94.8ish top-5 after 250 and some epochs, but it looks like it's not going to get much further. I was hoping/expecting > 81. I think I was a bit aggressive on the hparams for this run (see below).

Running this on two Titan RTX w/ AMP:
./distributed_train.sh 2 /data/x/imagenet/ --model mixnet_xl -b 128 --amp --sched step --epochs 500 --decay-epochs 3 --decay-rate 0.963 --opt rmsproptf --opt-eps .001 -j 4 --warmup-epochs 5 --weight-decay 1e-5 --drop 0.225 --color-jitter .1 --lr .02 --model-ema

drop_connect is set to 0.175 here even though it's not mentioned in the MixNet paper it seems useful.

The TF impl uses global_batch_size * .016 / 256 for learning rate. So I'm a bit high for this run but I wanted to see how it went. And also decay is a bit more aggressive. I think I might do another experiment with initial lr .016 and .97 decay, drop the color_jitter back in the .05-.07 range.

I checked the TF impl a few times and they don't seem to be doing color augmentation for these models in their default training setup. But their AutoAugment setup does have color aug in some of the choices. So I'm not quite sure what to think here, but generally I've found it hard to reproduce the stated results in any of these models with the default '0.4' brightness/contrast/saturation jitter factors. I usually leave it enabled, but much lower.

[rwightman]

If you do find some good h-params and good results, I'd love to know. I'm happy add any good weights you manage to train for B3-B7 for the PyTorch models. Given how long the process takes, they aren't high on my priority list to tackle with limited resources.

[DrJimFan]

@DannyHernandez could you please post your learning curves, trained from scratch using the commands you listed? It would be very helpful to verify my runs. Thanks!

[cxxgtxy]

@DannyHernandez
I found good models on par with efficientnet and they report their reproduction results for EfficientNet.
https://arxiv.org/abs/1908.06022