Removing transformers for 'completed' groups

[StijnKas]

Hi! I'm working on my thesis and using River for streaming analysis over process data. Thus far I'm liking river, but there are some performance issues since I essentially use a lot of transformer functions to represent an entire "case" at once. This is necessary, because when training a model I cannot just use the last seen value in the stream since it's missing values that have appeared earlier in the process' lifecycle. I can circumvent this by using transformer functions such as mean and min/max, and I wrote some of my own transformer functions to get the first/last (non-NA) value within a case/group, or the order of events within a case as separate column-features.

This works, but since I'm using quite large process logs (over a million records, with tens of thousands of cases), these transformers are starting to slow down my system. However, there is a point at which I don't need them: before I have seen the final label in the input data I can just predict for each incoming data instance thus I need the transformers, but once the label has appeared in the data (for instance, a 'customer accepted' event), the model can be trained on that datapoint, and afterwards there is no point to using the model because the final label is actually known for the case. Therefore, it is also fine to remove all of the transformer functions for that group, because that data is no longer needed to train the model.

Since my laptop actually can't handle not removing the transformers, I have found a way to remove them manually, for both individual transformers and transformerUnions (inside a for loop, called after learn_one is called on x):

#To remove groups from a transformer:
del transformername.groups[transformername._get_key(x)]

#To remove groups from multiple transformers in a TransformerUnion:
for name in transformergroup.transformers:
    transformername = transformergroup[name]
    del transformername.groups[transformername._get_key(x)]

As far as I know, the transformers have no way to remove groups from them apart from this (admittedly crude) way. I realise this might be quite a niche application for transformers and most applications wouldn't need to remove groups, but it would make for much cleaner code, and perhaps someone else might find it useful.

Would it make sense to add a 'remove_groups' function to the base transformer and TransformerUnion class, or is there a better way to remove unnecessary transformer groups?

[MaxHalford]

I'm sorry to be blunt but I have no idea what you're talking about. What's a case? Do you have reference to something we can read?

[StijnKas]

My apologies: let me try to give an example.

Say we have a dataset from a bank which handles loan requests (BPIC12&BPIC17 are common datasets in the process mining world and they describe just that). All loan requests are recorded into one dataset, but within that dataset are multiple ‘cases’: individual applications for a loan, where each case is assigned a case id. Every row in the set is a recorded activity, typically recorded through an automated system. The dataset may look like this, although this example is very simplified:

caseid	activity	resource	timestamp	amount	credit_score
1	request	customer	1/1/2021	100
2	request	customer	2/1/2021	4000
2	check credit score	company	3/1/2021		750
1	change	customer	3/1/2021	200
1	check credit score	company	3/1/2021		400
2	accepted	customer	4/1/2021
1	denied	company	4/1/2021

My prediction task is to predict whether a loan gets accepted, rejected or denied as the case is still ongoing. This label is found within the dataset itself: the ‘activity’ column will eventually consist of an activity called ‘accepted’, ‘rejected’ or ‘denied’. In my approach, I simply iterate through the dataset and predict for each entry until I find one of those three values in the ‘activity’ column, at which point I train the model on the data it’s seen for that case thus far. After that point it is of course no longer necessary to predict the outcome/label for that case, since it’s already appeared in the data. If I’ve already trained the model on a ‘case id’, I just don't do anything with that data.

Now, because my aim is to predict something for an entire case, and not just for the last value I’ve seen within that case, I need to somehow represent the entire case when trying to train the model (within process literature, this is referred to as 'encoding'). For this, I accumulate features over time by combining multiple transformers. For instance, I can take the average and/or the last value I’ve seen of that feature (within the case). For my example, maybe I would like to know the very last value for the 'amount' column and the 'credit_score' column in order to make a prediction, so I define multiple transformers and join them in a TransformerUnion. When making a prediction or when training the model, I call learn one and transform one on the latest row and use that as input for the model.

This works, as I essentially tabulate the stream set in ‘regular’ features, which the model can handle. However, like I mentioned, once the model is trained on that case the data is no longer necessary. This is when I would like to remove the groups from the transformers. Here is a comparison of storage when removing and when not removing the transformers, where the prediction performance is exactly the same (with model training and prediction omitted):

#encodings is a TransformerUnion, with in this case around 15 transformers. In a real setup I would probably have more.

#dataset in this case is the BPIC 2017 dataset, see https://www.win.tue.nl/bpi/doku.php?id=2017:challenge

outcomes = {‘accepted’, ‘rejected’, ‘denied’}
seen=set()
for x in tqdm(dataset, total = dataset_length):

    if x[‘caseid'] not in seen:
        encoding = encodings.learn_one(x).transform_one(x)

        if x[activity] in outcomes:
	    #here I would define ‘y’ based on the activity column,
	    #and then train the model using encoding as the X

            seen.add(x['caseid'])

	else:
	    #here I would use encoding as the X for a predict_one

If I run this on the dataset of around 1.2 million rows (200MB in uncompressed csv), and then call:

encodings._memory_usage

Encodings now has a full memory usage of 337 MB. If I instead remove the groups from the encodings object after I trained the model on it as such:

#encodings is a TransformerUnion, with in this case around 15 transformers. In a real setup I would probably have more.

#dataset in this case is the BPIC 2017 dataset.

outcomes = {‘accepted’, ‘rejected’, ‘denied’}
seen=set()
for x in tqdm(dataset, total = dataset_length):

    if x[‘caseid'] not in seen:
        encoding = encodings.learn_one(x).transform_one(x)

        if x[activity] in outcomes:
	    #here I would define ‘y’ based on the activity column,
	    #and then train the model using encoding as the X

            seen.add(x['caseid])

            for name in encodings.transformers:
                encoder = encodings[name]
                del encoder.groups[encoder._get_key(x)]
		
	else:
	    #here I would use encoding as the X for a predict_one

If I now call:

encodings._memory_usage

The resulting size is only 1.16 MB. This has no impact on my prediction performance whatsoever, but clears up a lot of memory!