Skip to content

Commit

Permalink
Ingest - types testing (#1129)
Browse files Browse the repository at this point in the history
This PR adds an ingest test to check what types are supported. The test
is quite strict right now. It compares the ingested field value
literally with the query result. A less strict human/Kibana-acceptable
comparison will be added in the future.

It prints the following table:

<img width="1326" alt="Screenshot 2024-12-20 at 11 52 48"
src="https://github.com/user-attachments/assets/8a14ee47-d194-4801-9220-744020d03fa7"
/>
  • Loading branch information
nablaone authored Dec 30, 2024
1 parent d4745d3 commit e556ec5
Show file tree
Hide file tree
Showing 4 changed files with 501 additions and 2 deletions.
48 changes: 48 additions & 0 deletions ci/it/configs/quesma-ingest-types.yml.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
frontendConnectors:
- name: elastic-ingest
type: elasticsearch-fe-ingest
config:
listenPort: 8080
- name: elastic-query
type: elasticsearch-fe-query
config:
listenPort: 8080
backendConnectors:
- name: my-minimal-elasticsearch
type: elasticsearch
config:
url: "http://{{ .elasticsearch_host }}:{{ .elasticsearch_port }}"
user: elastic
password: quesmaquesma
- name: my-clickhouse-instance
type: clickhouse-os
config:
url: clickhouse://{{ .clickhouse_host }}:{{ .clickhouse_port }}
ingestStatistics: true
processors:
- name: my-query-processor
type: quesma-v1-processor-query
config:
indexes:
"*":
target:
- my-clickhouse-instance

- name: my-ingest-processor
type: quesma-v1-processor-ingest
config:
indexes:
"*":
target:
- my-clickhouse-instance

pipelines:
- name: my-elasticsearch-proxy-read
frontendConnectors: [ elastic-query ]
processors: [ my-query-processor ]
backendConnectors: [ my-minimal-elasticsearch, my-clickhouse-instance ]
- name: my-elasticsearch-proxy-write
frontendConnectors: [ elastic-ingest ]
processors: [ my-ingest-processor ]
backendConnectors: [ my-minimal-elasticsearch, my-clickhouse-instance ]

5 changes: 3 additions & 2 deletions ci/it/configs/quesma-ingest.yml.template
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ processors:
- my-clickhouse-instance
"*":
target:
- my-minimal-elasticsearch
- my-clickhouse-instance
- name: my-ingest-processor
type: quesma-v1-processor-ingest
config:
Expand Down Expand Up @@ -179,7 +179,8 @@ processors:
- my-clickhouse-instance
"*":
target:
- my-minimal-elasticsearch
- my-clickhouse-instance

pipelines:
- name: my-elasticsearch-proxy-read
frontendConnectors: [ elastic-query ]
Expand Down
5 changes: 5 additions & 0 deletions ci/it/integration_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,3 +61,8 @@ func TestABTestcase(t *testing.T) {
testCase := testcases.NewABTestcase()
runIntegrationTest(t, testCase)
}

func TestIngestTypesTestcase(t *testing.T) {
testCase := testcases.NewIngestTypesTestcase()
runIntegrationTest(t, testCase)
}
Loading

0 comments on commit e556ec5

Please sign in to comment.