-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Respect size
parameter in aggregations (1/3 - refactor)
#35
Conversation
ae2d25c
to
d4f9eb0
Compare
080517c
to
5b73ca3
Compare
quesma/model/simple_query_test.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
simply copied from queryparser
package
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok it got moved, not copied.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
code is the same as before, added only a few checks which were missing.
Moved from big single file to a separate one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rest of those functions will also have finish...
name, I think more appropriate, if you don't agree, maybe let's argue in next PR when I change all
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
moved to model
quesma/util/utils.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's only used in tests, these prints help when they fail
9532bb1
to
4b4efc3
Compare
size
parameter in aggregationssize
parameter in aggregations
size
parameter in aggregationssize
parameter in aggregations (1/3) refactor
size
parameter in aggregations (1/3) refactorsize
parameter in aggregations (1/3 - refactor)
1ceacc1
to
5973ca4
Compare
The general comment is that "LIMIT N BY " is an unusual SQL method. Most databases do not support this. Quesma would still need to generate a query for that database without this clause. |
Ok, I would need walkthrough. I see a lot of refactoring changes which are ok. Some logic there I don't understand. |
quesma/model/simple_query_test.go
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok it got moved, not copied.
aggrBuilder.Type = aggr | ||
aggrBuilder.whereBuilder = model.CombineWheres(cw.Ctx, aggrBuilder.whereBuilder, filter.Sql) | ||
aggrBuilder.Aggregators = append(aggrBuilder.Aggregators, model.NewAggregatorEmpty(filter.Name)) | ||
*resultAccumulator = append(*resultAccumulator, aggrBuilder.buildBucketAggregation(nil)) // nil for now, will be changed |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don;t get it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Used to be like that before.
.Type
keeps what type of aggregation it is, here some filters
.
.whereBuilder
keeps where
clause as string, I combine, so do old_where AND this_filter's_where
.Aggregators
= list of aggregation names to be put in response's JSON. We add filter.name
where it's kept.
resultAccumulator
= result list of all queries we need to send to DB, we add 1 here for each filter
5973ca4
to
da2fe90
Compare
bd3e400
to
2e55d63
Compare
Basically moves
SimpleQuery
fromqueryparser
package tomodel
. Also starts a small refactor toaggregation_parser
, which is the motivation for this change. Before we had 1 functionparse_some_aggregation_type
. Now I split it into 2:parse
andprocess
. I think it simplifies a lot of things + enables making plugins for new aggregations quite easy.edit2: comments below are obsolete. Changed this PR into refactor, next will follow.
Edit: I think it works 100% fine now, although this issue is generally pretty tricky, and hard to get right, and code is so far pretty ugly, because of a lot of trial and error approaches.
1 test was failing after some small fixes, and I finally managed to make it work at the end. Will check rest of tests tomorrow, but I'm 99% they'll pass.
Generally, plan is to transform (multiple)
terms
(simple split into buckets, orGROUP BY
with limit) (e.g. withLIMITs 2
and5
) into something like this:First
terms
uses normalLIMIT N
,2+ terms
useLIMIT N BY [previous terms]
You can paste this into our Clickhouse to see that such a query is correct
Old comments:
Before: we didn't add
LIMIT x
anywhere, so we were displaying all buckets.After: well, I tried visualizing a lot of charts in OpenSearch and it seems to work everywhere, even this
Other
bar.BUT
a) not important, fixed on Tuesday I'm pretty sure the code isn't perfect, as I said I can't break it easily in UI, but with more complex aggregations it probably will not work 100% correctly. The PR code isn't prettiest, maybe some small refactor which is needed anyway will help with that correctness issue.
b) still valid, but not very important, and not very hard
Order (by)
parameter in query still isn't respected.I'd leave b) for next PR.