Dropout implementation for ensemble

[johanenforcer]

How would I go about implementing dropout in an efficient manner, I am aware of the post where overfitting has been discussed ,where dropout has been discussed but wanted to discuss these matters more. The point is not to reduce overfitting, but rather just use the dropout itself for the sake of creating an ensemble of model networks for an active learning query strategy by sampling different model variants. That being said, I am interested in knowing whether it is fine to implement the dropout technique straight after the backbone has finished, since I am retrieving activation volumes from three different conv layers in the head section of the network? Is the regular dropout such as self.drop = nn.Dropout2d(0.1)` fine to use or should there be other considerations like for example using dropblock instead on the activation volume from the backbone?

Thanks in advance!

[glenn-jocher]

@johanenforcer yes you can add a dropout layer by modifying modules directly (i.e. in models/common.py) or by simply adding a line to your model.yaml file. If you modify the model.yaml file though you need to ensure that all layers after the modified layer have their input indexes updated accordingly, i.e. probably adding +1 to their values.

[johanenforcer]

Would you rather recommend using it in the last layer, specifically the detection module like you have done previously, or after each activation function following the various conv layers in the various modules that exist?

Like for example would using dropout on the inputs to the final detect layer be fine, and not hurt performance significantly much?

self.drop = nn.Dropout2d(0.1) or just regular dropout for that matter

def forward(self, x):
    # x = x.copy()  # for profiling
    z = []  # inference output
    self.training |= self.export
    for i in range(self.nl):
        x[i] = self.m[i](self.drop(x[i]))  # conv

[glenn-jocher]

@johanenforcer I don't know, you'd have to experiment with various configurations and then compare them to a baseline trained side by side.

[johanenforcer]

Yes! Thanks for all the answers!

[nanfei666]

@johanenforcer

Would you rather recommend using it in the last layer, specifically the detection module like you have done previously, or after each activation function following the various conv layers in the various modules that exist?

Like for example would using dropout on the inputs to the final detect layer be fine, and not hurt performance significantly much?

self.drop = nn.Dropout2d(0.1) or just regular dropout for that matter

def forward(self, x):
    # x = x.copy()  # for profiling
    z = []  # inference output
    self.training |= self.export
    for i in range(self.nl):
        x[i] = self.m[i](self.drop(x[i]))  # conv

im trying to do the same thing, do you find which way is better?

[abhinavdhere]

@glenn-jocher Similar to OP I'm looking to add dropout in Yolov8 detect, but it seems the mechanism of loading models has changed since your last response here. There exists no models/common.py file now, and the model configuration gets loaded from yaml files but there is no visible yaml file in the repository that I could modify. Would be very helpful if you could take suggest how to add dropout in detection with the current code structure.