Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

schemaOverride configuration is ignored #1083

Open
nablaone opened this issue Dec 9, 2024 · 0 comments
Open

schemaOverride configuration is ignored #1083

nablaone opened this issue Dec 9, 2024 · 0 comments

Comments

@nablaone
Copy link
Member

nablaone commented Dec 9, 2024

Adding schemaOverride for a field doesn't work.

Here is a scenario:

  1. Run Quesma with the config:
disableTelemetry: true
logging:
  enableSQLTracing: true
ingestStatistics: false

frontendConnectors:
  - name: elastic-ingest
    type: elasticsearch-fe-ingest
    config:
      listenPort: 8080
      disableAuth: true
  - name: elastic-query
    type: elasticsearch-fe-query
    config:
      listenPort: 8080
backendConnectors:
  - name: E
    type: elasticsearch
    config:
      url: "http://elasticsearch:9200"
  - name: C
    type: clickhouse-os
    config:
      url: "clickhouse://clickhouse:9000"

processors:
  - name: my-query-processor
    type: quesma-v1-processor-query
    config:
      indexes:
        test:
          schemaOverrides:
            foo:
              targetColumnName: "foofoo"
              type: "text"
          target: [ C ]

        '*':
          target: [ E ]

  - name: my-ingest-processor
    type: quesma-v1-processor-ingest
    config:
      indexes:
        test:
          schemaOverrides:
            foo:
              type: "text"
              targetColumnName: "foofoo"
          target: [ C ]

        '*':
            target: [ E ]

pipelines:
  - name: my-pipeline-elasticsearch-query-clickhouse
    frontendConnectors: [ elastic-query ]
    processors: [ my-query-processor ]
    backendConnectors: [ E, C ]
  - name: my-pipeline-elasticsearch-ingest-to-clickhouse
    frontendConnectors: [ elastic-ingest ]
    processors: [ my-ingest-processor ]
    backendConnectors: [ E, C ]

2 . Add document

curl http://localhost:8080/test/_doc -d '{"foo": 1, "bar": 2}'
  1. Check logs. Quesma creates following table:
quesma-1  | Dec  9 13:14:08.786 INF quesma/ingest/processor.go:802 > DDL query execution: CREATE TABLE IF NOT EXISTS "test"
quesma-1  | (
quesma-1  |     "@timestamp" DateTime64(3) DEFAULT now64(),
quesma-1  |     "attributes_values" Map(String,String),
quesma-1  |     "attributes_metadata" Map(String,String),
quesma-1  |
quesma-1  |
quesma-1  |     "bar" Nullable(Int64) COMMENT  'quesmaMetadataV1:fieldName=bar',
quesma-1  |     "foo" Nullable(Int64) COMMENT  'quesmaMetadataV1:fieldName=foo',
quesma-1  |
quesma-1  | )
quesma-1  | ENGINE = MergeTree
quesma-1  | ORDER BY ("@timestamp")
quesma-1  |

I was expecting the "foofoo" column instead of "foo".

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant