search_after param (used to paginate hits e.g. in Discover), 1/2 strategies

[trzysiek]

#1099 1/2 PR

After this PR instead of discarding search_after param, we simply add tuple(sort_field_1, sort_field_2, ...)<(search_after_1, search_after_2, ...) to the WHERE clause, so we'll (almost) properly return paginated hits (never wrong ones, but we can skip some valid ones)
If sort_field_i is asc, we swap sort_field_i with search_after_i in the expression above, I think it gives exactly what we want.

Timestamp fields use fromUnixTimestampMilli64 conversion - it'll work because the search_afterparam is what we returned in the previous request ourselves, for previous hits. And for dates/timestamps we always return unix_timestamp in ms.

This strategy is not fully correct, because we'll skip all hits with exactly the same sort values (timestamps in ms, usually), but it's some progress anyway. Before this PR, queries with this param basically didn't work at all, now they have a small bug.

TODO (I'll finish in next PR):

• Finish Bulletproof strategy, which fixes the limitation above. Should be quick as I already got a lot of tests here, and there's not really a lot of code in the logic itself.
• Fix conversion function, we always use from...Milli64. It'll always work for Clickhouse, but for Hydrolix and DateTime field (without 64), it won't. It's a known issue, quick to fix. I'd do it in another PR, as before this one, search_after didn't work anyway, so there's no regression here.

Screens of this working after all the commits:

[cloudflare-workers-and-pages]

Deploying quesma with    Cloudflare Pages

Latest commit:	7afebeb
Status:	✅  Deploy successful!
Preview URL:	https://80d7440c.quesma.pages.dev
Branch Preview URL:	https://search-after-2.quesma.pages.dev

View logs

[jakozaur]

Great work!

1. Having more generic timestamp method accomodating for different types would be awesome.
2. Not sure what if we don't have timestamp.
3. Tests.

[jakozaur]

This is not always correct, as we might have timestamps in seconds, not milliseconds.

[trzysiek]

It's always correct, because we read up the value from this:

func FormatSortValue(i any) any {
	switch v := i.(type) {
	case time.Time:
		// When returned as part of `sort`, timestamps are always returned as millis
		return v.UnixMilli()
	default:
		return i
	}
}

As you see, if it's present, it'll always be milliseconds

[trzysiek]

But I definitely agree that the way it works and why it does, is way too cryptic.

[jakozaur]

Same here.

[trzysiek]

2. Not sure what if we don't have timestamp.

Tested that, and
a) we will return empty sort field for every hit
b) even if we have timestamp, we can sort by some other field, and my solution won't work anyway
edit: now it will

So unfortunately there's some TODO to do.

[trzysiek]

1. Having more generic timestamp method accomodating for different types would be awesome.

Agreed, but let me do that in a quick next PR, this one already is quite big, wouldn't like to mess in a couple more places.