Active Learning Module #1037
Replies: 8 comments 15 replies
-
|
Hello! Thank you for starting this topic. Active learning and online learning should play well together. As I understand it, active learning helps in speeding-up the labeling process. That assumes there is a labeling process in place, like with a user interface. Therefore, I don't understand how an active learning strategy can be implemented without a labeling process. My question to you is this: what kind of user experience do you envision? Do you have a good example of active learning? Would you simply store samples to label in a queue and delegate the labeling to a dedicated labeling tool? I'm all for implementing active learning, but I think the first step is to describe a user experience, and think things through before getting into an implementation :) |
Beta Was this translation helpful? Give feedback.
-
|
Sure! I forked the repo, and created a branch named active-learning. I've implemented two simple AL methods: one based on random selection and the other based on uncertainty. The training loop would look with something like this: It would be a classifier wrapper which inside decides if the label is going to be required or not. Of course this is a naive implementation considering that we have the labels on the fly. The way it is implemented right now, if y is None, we simply dont use this instance, but in the future a method |
Beta Was this translation helpful? Give feedback.
-
|
Great! You're fast :) So my issue here is that see the value that the active learning is providing, other than the fact it trains on less data. That makes looks it look like the I think we can make an active learning more useful than that. I think you should still use a wrapper. What I suggest is that the base_classifier = HoeffdingAdaptiveTreeClassifier()
al_classifier = ALUncertainty(base_classifier, budget = 0.2, uncertaintyStrategy = "SelSampling")
for x, y in dataset.take(10000):
y_pred = al_classifier.predict_one(x)And then for training you would do: for x in al_classifier.samples_to_label:
y = get_label(x)
al_classifier.learn_one(x, y)The I had a chat with @smastelini and he suggested something similar too. What do you think? |
Beta Was this translation helpful? Give feedback.
-
|
+1 to that! @MaxHalford and I envisioned similar strategies to expose active learning methods. I like this design a lot! |
Beta Was this translation helpful? Give feedback.
-
|
Active Learning is not for learning from less data, is a tool to select the most informative instances to our model to learn because we assume that our stream is not fully labeled. We are not going to have beforehand which samples are more informative or not. Therefore, I don't think makes sense to have a "samples to label" since we are deciding if we label or not an instance on the fly. We receive an instance, use the model to predict and after we decide if we need (and we can get) the associated label. Then, I think that this loop only makes sense if they are async, as soon that we predict for x al_classifier would populate samples to_label and it will trigger learn_one as soon as the label is available. |
Beta Was this translation helpful? Give feedback.
-
Right, I believe we're aligned. I suggested what I suggested because I didn't see how the loop you suggested catered to this.
To be clear, the model has to be always available to make predictions. The labeling would happen asynchronously in a different process.
Yes exactly. And we can have a labeling UI in the background. At a previous job I implemented a (batch) active learning tool to classify blobs of pixels as signatures or not. I think I could build a dataset and make an example out of it. |
Beta Was this translation helpful? Give feedback.
-
|
I have a setup to generate a streaming datasets of blobs to classify as signatures or not. I believe I can write a nice live demo of addressing this usecase with active learning :) |
Beta Was this translation helpful? Give feedback.
-
|
Check out this example from Rubrix, which uses small-text for active learning under the hood. |
Beta Was this translation helpful? Give feedback.
-
|
Cool, that is what I was thinking about how it would look. The main difference is that it is not online learning, right? It is selecting the instances to label based on the whole dataset not instance by instance. |
Beta Was this translation helpful? Give feedback.
-
|
Yes indeed, most active learning tools/interfaces work with batch models in mind. |
Beta Was this translation helpful? Give feedback.
-
|
Hello. A Drift Detection Method Based on Active Learning. I used the strategy of Zloibaite. I. Zliobaite, A. Bifet, B.Pfahringer, G. Holmes. “Active Learning with |
Beta Was this translation helpful? Give feedback.
-
|
Is it interesting for the library to provide the implementation of the detection method based on active learning? |
Beta Was this translation helpful? Give feedback.
-
|
Do you mean the paper you developed in your paper? Sure, but there's a lot of groundwork to do before. We need to thing what active learning would look like in River, how would people use it, how would it connect to the rest of the Python ecosystem, how it would work with the rest of River. |
Beta Was this translation helpful? Give feedback.
-
|
Hey! I implemented a basic active learning setup in #1162. I've added a card to remind us to look for better active learning strategies than the basic one I implemented. However, I'm quite confident we now have a strong foundation on which to build. |
Beta Was this translation helpful? Give feedback.
-
|
Very nice. |
Beta Was this translation helpful? Give feedback.
-
|
Cheers! Feel welcome to try out and contribute better strategies. Also let me know if this foundation is not flexible enough, we can still make changes. |
Beta Was this translation helpful? Give feedback.
-
|
Hello. There are an error in from river import datasets. ModuleNotFoundError: No module named 'river.utils.vectordict' In the folder, the file vectordict it has the .pyx extension
|
Beta Was this translation helpful? Give feedback.
-
|
Can you provide more context? What version of River have you installed? Or this in development? |
Beta Was this translation helpful? Give feedback.
-
|
The problem was resolved. |
Beta Was this translation helpful? Give feedback.
-
|
I trying make git push with pre-commit, but this erro is showing. It looks like an SSH connection error to install yaml. Does anyone know how to help? Thank you `### version information error information` |
Beta Was this translation helpful? Give feedback.
-
|
Are you sure you've followed the instructions in CONTRIBUTING.md? Also what version of Python are you using? It looks like you're missing a dependency in your setup. Next time can you please open a separate discussion? This is off topic and is notifying people for no good reason. |
Beta Was this translation helpful? Give feedback.
-
Do you think it is necessary to implement an active learning module for river? Similar to MOA active, with some classic sampling methods such as Random Selection, FixedUncertainty, VarUncertainty, RandVarUncertainty, SelSampling?
I'm available to implement it since I have already implemented prototype for personal purpose.
Beta Was this translation helpful? Give feedback.
All reactions