Human descriptions for Clay embeddings

[brunosan]

I was thinking that once we start having embeddings for locations, these will have some mathematical representation for the features within. The equivalent of semantics for LLMs. These will cluster when we have a group of embeddings according with common contents. E.g. a cluster for forests, or cities. That's the whole idea of similarity search. What this won't give us is the human words for that concept. We need someone to tag the common thing on that cluster. We could crowsource there, specially as human descriptions of locations remain the same when we improve the embeddings, might even help us add more trainning data when the embeddings merge different semantics. This should not be hard, especially with a good UI.

Moreover, I want to believe we already have those descriptions in some other way. E.g. OSM could give us some sense of what's inside (POIs, class attributes like road or highway, ...). Also Wikipedia entries that are geolocated. Let's call these "human labels", either direct or via OSM, Wiki or other sources that describe same locations.

My hunch here is that we could train a CLIP style cross-domain link so that we learn to bridge between the Clay geo semantics and human semantic labels. That is, Clay will learn to tag and describe in human terms the content it sees. This would be huge for LLM bridges where Clay can significantly augment their capabilities, or create synthetic text describing any location in detail.

The stretch goal here is that the semantics nature of LLM embeddings might be able to extrapolate to new Clay semantics. E.g. the LLM embeddings for "crop" + "water" will probably point close to "floods". Meanhile, "crop" and "water" are OSM tags but not "floods". I do wonder if this "Clay CLIP" approach will learn to tag flooded areas as such based on the LLM semantic inference.