streamkmeans visualization

[drakeleeb]

How do I pull information to plot as streamkeans is running? Not sure where is the best place to grab the evolving cluster centers and clustered data. Are there any examples for this use-case? Thanks.

[MaxHalford]

@smastelini @hoanganhngo610 have fun with this one :) (seems there's a lot of interest in clustering!). Also @vsoch might want to chime in. I'm sure it's possible to do some matplotlib judo to get this to work. We just don't have any good examples right now in the documentation, as far as I'm aware of.

In general, I don't think it should be River's goal to provide visualization utilities. There are just too many use cases to cater to. We could however imagine providing another library for online viz purposes. It's just a lot of work right now.

[vsoch]

@drakeleeb I've done something similar! It could be an "on demand" view (given you have a web interface that someone will load and then run it on demand) or you could have it run at some interval / by some other trigger. For:

1. Determining if a model has centers and finding them
2. Sending the data to a web interface to render
3. (optionally) finding the currently assigned data

The general steps I used are the following:

Get the model and make sure it has centroids (not all do, and maybe you can skip this step if you are only using clustering models: https://github.com/spack/spack-monitor/blob/ad70f9fce241cf866ee10ebddd32564dc18b5c00/spackmon/apps/main/views/analysis.py#L26-L43

@ratelimit(key="ip", rate=rl_rate, block=rl_block)
@never_cache
def view_clusters(request):
    """
    View machine learning clusters for our model!
    """
    # Get a django client
    client = DjangoClient()

    model = client.get_model(cfg.MODEL_NAME)
    have_centroids = False
    if online_ml.get_centers(model) is not None:
        have_centroids = True
    return render(
        request,
        "online_ml/clusters.html",
        {"have_centroids": have_centroids, "model_name": cfg.MODEL_NAME},
    )

From a web UI, I want the centroids to be fetched by the view on demand (and not grabbed in advance and then passed to it) so I have a function that checks for centroids:

if online_ml.get_centers(model) is not None:
    have_centroids = True

and that is basically looking at model.steps or directly at model.centers (you'll have model.steps for a pipeline, more common, and just the model with centers directly on it without any other steps): https://github.com/spack/spack-monitor/blob/ad70f9fce241cf866ee10ebddd32564dc18b5c00/spackmon/apps/main/online_ml.py#L22-L31

def get_centers(model):
    """
    Helper function to derive centroids from a model
    """
    if hasattr(model, "steps"):
        for step_name, step in model.steps.items():
            if hasattr(step, "centers") and step.centers:
                return step.centers
    elif hasattr(model, "centers") and model.centers:
        return model.centers

Once you know that you have centroids, what actually I found easy to do is just pass the boolean to the interface, and have a request to a different endpoint explicitly to return the centroids (and then render into a visualization, for example): https://github.com/spack/spack-monitor/blob/ad70f9fce241cf866ee10ebddd32564dc18b5c00/spackmon/apps/main/templates/online_ml/clusters.html#L160-L177. That hits this endpoint function: https://github.com/spack/spack-monitor/blob/ad70f9fce241cf866ee10ebddd32564dc18b5c00/spackmon/apps/main/views/analysis.py#L46-L53

def get_centroids(request, name):
    """
    JsonResponse to just retrun centroids
    """
    client = DjangoClient()
    model = client.get_model(name)
    centers = online_ml.get_centers(model)
    return JsonResponse({"centers": online_ml.generate_embeddings(centers)})

Which is going to use that same function to get the centers, but instead give them to another function to return a reduced version (e.g., high dimension reduced to 2 so we can plot x and y). I basically generate a distance matrix then run TSNE and return that. That happens here https://github.com/spack/spack-monitor/blob/ad70f9fce241cf866ee10ebddd32564dc18b5c00/spackmon/apps/main/online_ml.py#L43-L63

def generate_embeddings(centers):
    df = pandas.DataFrame(centers)

    # 200 rows (centers) and N columns (words)
    df = df.transpose()
    df = df.fillna(0)

    # Create a distance matrix
    distance = pandas.DataFrame(
        squareform(pdist(df)), index=list(df.index), columns=list(df.index)
    )

    # Make the tsne (output embeddings go into docs for visual)
    fit = manifold.TSNE(n_components=2)
    embedding = fit.fit_transform(distance)
    emb = pandas.DataFrame(embedding, index=distance.index, columns=["x", "y"])
    emb.index.name = "name"

    # cluster ids
    emb["name"] = emb.index
    return emb.to_dict(orient="records")

But the above is a more complicated server setup - you can literally just grab the model centers, either at model.steps["<model_name>"].centers or just model.centers and do what you want with them! My particular approach generates this:

And actually I just realized I added this example to Django River ML (A Django Plugin for river) in the sample all (how to run here).

So that's a "do it yourself" means to get centroids and then view them, and if you wanted to actually assign your data to the current centroids you'd have to loop through some set of data and get a prediction (a number for a centroid) and then save that (and possibly allow a user to mouse over a centroid and then see assigned). A static version of that is here https://github.com/buildsi/spack-monitor-nlp-dev/blob/a99fa2fb780ad708ab1ee5b08cdeaa1222a2a83b/2.online-ml.py#L223-L247

But there are other cool ideas in your question:

• Can we have a library that provides visualization for river? I think so - the question would be what format do we want. For streaming what comes to mind (that makes sense) is some kind of simple flask app that uses sockets to update data at some frequency. Otherwise it would be "on demand" (e.g., refresh to see new updates). But choosing this approach would then force us to choose a particular server backend, and I don't think we'd want that - we'd want something that can be used in Flask or Django or whatever. Or we'd want to use some of the cool interactive libraries like bokeh / plotly / holoviz / pycharts. The choice really depends on your goal - a server approach is for server applications, and the others are more suited for data science. For a server that needs to load quickly, a main issue would be the size of the model / time to render a new update. And probably for the plotting /JS libraries we'd want to decide on that first, and make some simple examples.
• Can we keep track of data assignments (and update them) so we can (on demand) see some kind of visualization without needing to run the full set of predictions? This is actually the case that I ran into the other day Question about centroid changes #891 and I concluded with changing clusters/microclusters (basically everything except for kmeans) it wouldn't be possible to store an assignment. But - if there is some way to represent state, e.g., after running a learn and having some change of cluster state:
  • "cluster 10 was split into 11 and 12 == reclassify just everything assigned with 10"
  • "cluster 10 was deleted" == reclassify everything that was in 10
  • "cluster 10 and 11 were merged in 12" == re-assign everything in 10 and 11 to 12.
    Then you could always just use the current assignments of your datum (stored elsewhere, like your database) to plot quickly (with the centroids). The strength of river is that we don't save all the data points - but that's also (sort of) can be a weakness that the model loses memory over time. The assignments change as do the centroids. My original efforts on my current dataset used Doc2Vec and embeddings, and it was great because I could plot every single point but then the data got larger and I could barely run anything. That's when I googled "streaming ML" or something similar and found river, haha.

In general, I don't think it should be River's goal to provide visualization utilities. There are just too many use cases to cater to. We could however imagine providing another library for online viz purposes. It's just a lot of work right now.

Yep 100% agree - plotting support should be separate. I'd enjoy thinking / working on this if there is a larger interest and people want to discuss how to go about it! I suspect most folks at the moment are "rolling their own" and arguably it doesn't have to be that way.

[hoanganhngo610]

This, is just, basically, beyond any expectation that I can ever think of. To be honest, I myself cannot even think of anything more complete than this. Kudos to @vsoch, once again.

[hoanganhngo610]

@drakeleeb Thank you so much for raising such an interesting question. I think @vsoch have just covered most of what you need to know upon querying the cluster centers and the cluster results; however, I think that upon my way of doing clustering benchmarking, I found quite an interesting way to visualize the results live using Holoviews. In fact, this practice has been done by @jacobmontiel during his tutorial on River in IJCAI 2020. I am intending to write a short notebook for the clustering module itself, and it should be released within no time!

[MaxHalford]

Very cool! I think we have some decision tree plots that would nicely here. Also some progressive validation charts would be nice to nice. I can contribute them once this is set up. Let us know how we can help :)

[vsoch]

Will do! I should be able to work on it most of today (already started) and get something up as soon as I get the design / branding / layout / automation to my liking. Since it's sphinx we will want to decide to deploy to readthedocs (supports automated builds and easy versioning) versus manual to GitHub pages. I'm good with either - the first is just connecting the repo and setting up in readthedocs, and the second I can write a workflow that deploys to GitHub pages.

[MaxHalford]

Gut feeling is that GitHub Pages would be nice because the URL would be riverml.xyz/viz. That being said, if readthedocs allows using a custom domain, then it's all good

[vsoch]

@MaxHalford I am setting it up - I need more permissions than you've given me to enable head branch cleanup, add the CircleCI integration, and enable pages. I'm a pretty seasoned developer, you can trust me with these things.

[vsoch]

okay here is the start! https://online-ml.github.io/viz/ It looks like I do have permissions to enable pages (and @MaxHalford could you please enable branch cleanup?) When @MaxHalford approves I have a circle recipe ready to go to allow us to preview the site when someone contributes a PR. Most of my time this morning was tweaking the style/branding of the template, and then writing contributing docs. We're on our way! 🎉