Plotting the steps of progressive_val_score

[anneborcherding]

Hi everyone,

I really like the feature to see the accuracy at different steps of the progressive_val_score evaluation and was wondering if it would be possible to return the data in a way for it to be easily plotted.

Currently, I am using the following workaround:

from river import metrics
from river import ensemble
from river import evaluate
from river import datasets

from io import StringIO
import matplotlib.pyplot as plt

dataset = datasets.Phishing()
model = ensemble.AdaptiveRandomForestClassifier(n_models=3, seed=42)
metric = metrics.Accuracy()

# write results of the evaluation into a dummy file

dummy_file = StringIO()
acc = evaluate.progressive_val_score(dataset=dataset, model=model, metric=metric, print_every=100, file=dummy_file)
history = dummy_file.getvalue()

# read the data from the dummy file

steps = []
accuracies = []

for line in history.split("\n"):
   vals = line.split(" ")
   if len(vals) == 3:
       steps.append(int(vals[0][1:-1].replace(",","")))
       accuracies.append(float(vals[2][0:-1]))

# plot the data

plt.plot(steps, accuracies, label=f'Adaptive Random Forest (final acc: {acc})')
plt.xlabel("Evaluation Steps")
plt.ylabel("Accuracy")
plt.title("Accuracy during progressive evaluation")
plt.legend()
plt.show()

Which results in the following graph:

Would it be possible to include that data processing into the evaluation? If you have any hints on how one could/should integrate it, I could of course create a starting point for discussion and put that into a pull request.

All the best
Anne

[MaxHalford]

Hey there! It's a good question. You can do this, which is a bit more terse:

from itertools import count
from river import metrics
from river import ensemble
from river import datasets
from river.evaluate.progressive_validation import _progressive_validation
import matplotlib.pyplot as plt

dataset = datasets.Phishing()
model = ensemble.AdaptiveRandomForestClassifier(n_models=3, seed=42)
metric = metrics.Accuracy()

checkpoints = count(100, 100)
accuracies = []
for step in _progressive_validation(dataset=dataset, model=model, metric=metric, checkpoints=checkpoints):
    accuracies.append(step['Accuracy'])

plt.plot(accuracies, label=f'Adaptive Random Forest (final acc: {accuracies[-1]:.2%})')
plt.xlabel("Evaluation Steps")
plt.ylabel("Accuracy")
plt.title("Accuracy during progressive evaluation")
plt.legend()
plt.show()

It's a bit ugly because you have to import a private method, as well as format the metric yourself. I'll make this a bit more accessible in the next release.

Hope this helps :)

[MaxHalford]

Done in #901, you'll see it in the next release

[anneborcherding]

Great, thank you so much!