Setting differential learning rates

[jcpayne]

I'm just wondering if the ability to set different learning rates for parameter groups within the model is implemented in Detectron2; for example could you train three parameter groups where the bottom layers of the model are training slower than the top, with lrs = np.array([lr/9, lr/3, lr]?

If I'm reading the code right, it looks like build_optimizer() and build_lr_scheduler() both are both set up to use a single learning rate for all of the layers in a model, but then WarmupMultiStepLR and WarmupCosineLR both have a variable that holds a list of learning rates (self.base_lrs).

I may have missed it, but I don't see where you would set the parameter groups or the corresponding learning rates.

Thanks.

[ppwwyyxx]

You can use any custom optimizer (https://detectron2.readthedocs.io/en/latest/tutorials/training.html), so you can set param groups to whatever needed.

detectron2/projects/Panoptic-DeepLab/train_net.py

Lines 111 to 112 in c152862

	@classmethod
	def build_optimizer(cls, cfg, model):

is one example

[jcpayne]

Sorry to be dumb but where would you define the parameter groups and where would you set the corresponding learning rates?

[ppwwyyxx]

That's about pytorch optimizer and not detectron2, see https://pytorch.org/docs/stable/optim.html for some docs

[jcpayne]

OK thanks for the pointer. I'll check it out.