Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

config changes to get kaldb working locally #1

Closed
wants to merge 3 commits into from
Closed

config changes to get kaldb working locally #1

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ef3c8f0
config changes
Feb 8, 2024
672ae54
turn on bulk ingest by default
autata Feb 12, 2024
58ac612
add notes/todos to make search endpoint more straightforward
autata Feb 14, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .java-version
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
21.0
2 changes: 2 additions & 0 deletions .run/KalDb_cache.run.xml → .run/KalDb Cache Node.run.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="KalDb Cache Node" type="Application" factoryName="Application">
<option name="ALTERNATIVE_JRE_PATH" value="21" />
<option name="ALTERNATIVE_JRE_PATH_ENABLED" value="true" />
<envs>
<env name="NODE_ROLES" value="CACHE" />
</envs>
Expand Down
2 changes: 2 additions & 0 deletions .run/KalDb_index.run.xml → .run/KalDb Index Node.run.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="KalDb Index Node" type="Application" factoryName="Application">
<option name="ALTERNATIVE_JRE_PATH" value="21" />
<option name="ALTERNATIVE_JRE_PATH_ENABLED" value="true" />
<envs>
<env name="NODE_ROLES" value="INDEX" />
</envs>
Expand Down
2 changes: 2 additions & 0 deletions .run/KalDb_manager.run.xml → .run/KalDb Manager Node.run.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="KalDb Manager Node" type="Application" factoryName="Application">
<option name="ALTERNATIVE_JRE_PATH" value="21" />
<option name="ALTERNATIVE_JRE_PATH_ENABLED" value="true" />
<envs>
<env name="NODE_ROLES" value="MANAGER" />
</envs>
Expand Down
2 changes: 2 additions & 0 deletions .run/KalDb_preprocessor.run.xml → .run/KalDb Preprocessor Node.run.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="KalDb Preprocessor Node" type="Application" factoryName="Application">
<option name="ALTERNATIVE_JRE_PATH" value="21" />
<option name="ALTERNATIVE_JRE_PATH_ENABLED" value="true" />
<envs>
<env name="NODE_ROLES" value="PREPROCESSOR" />
</envs>
Expand Down
2 changes: 2 additions & 0 deletions .run/KalDb_query.run.xml → .run/KalDb Query Node.run.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="KalDb Query Node" type="Application" factoryName="Application">
<option name="ALTERNATIVE_JRE_PATH" value="21" />
Copy link

@mansu mansu Feb 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The run config works for me without this. So, it may be an issue with your intelliJ setup.

<option name="ALTERNATIVE_JRE_PATH_ENABLED" value="true" />
<envs>
<env name="NODE_ROLES" value="QUERY" />
</envs>
Expand Down
2 changes: 2 additions & 0 deletions .run/KalDb_recovery.run.xml → .run/KalDb Recovery Node.run.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
<component name="ProjectRunConfigurationManager">
<configuration default="false" name="KalDb Recovery Node" type="Application" factoryName="Application">
<option name="ALTERNATIVE_JRE_PATH" value="21" />
<option name="ALTERNATIVE_JRE_PATH_ENABLED" value="true" />
<envs>
<env name="NODE_ROLES" value="RECOVERY" />
</envs>
Expand Down
3 changes: 2 additions & 1 deletion Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,5 @@ RUN cd /work; mvn package -DskipTests
FROM amazoncorretto:21
COPY --from=build /work/kaldb/target/kaldb.jar /
COPY --from=build /work/config/config.yaml /
ENTRYPOINT [ "java", "-jar", "./kaldb.jar", "config.yaml" ]

ENTRYPOINT [ "java", "--enable-preview" , "-jar", "./kaldb.jar", "config.yaml"]
2 changes: 1 addition & 1 deletion config/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,4 +140,4 @@ preprocessorConfig:
dataTransformer: ${PREPROCESSOR_TRANSFORMER:-json}
rateLimiterMaxBurstSeconds: ${PREPROCESSOR_RATE_LIMITER_MAX_BURST_SECONDS:-1}
kafkaPartitionStickyTimeoutMs: ${KAFKA_PARTITION_STICKY_TIMEOUT_MS:-0}
useBulkApi: ${KALDB_PREPROCESSOR_USE_BULK_API:-false}
useBulkApi: ${KALDB_PREPROCESSOR_USE_BULK_API:-true}
2 changes: 1 addition & 1 deletion docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ services:
- ALLOW_ANONYMOUS_LOGIN=yes

kafka:
image: 'bitnami/kafka:3.2.3'
image: 'bitnami/kafka:latest'
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo: pin working version

container_name: kaldb_kafka
ports:
- '9092:9092'
Expand Down
6 changes: 6 additions & 0 deletions kaldb/src/main/java/com/slack/kaldb/elasticsearchApi/OpenSearchRequest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,14 +85,20 @@ private static String getQueryString(JsonNode body) {
return queryString;
}

// This is required. consider returning a more useful error to the user than 500
// or setting a reasonable default (default seems a good option here)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably should use whatever default size ES uses so it works like their docs

private static int getHowMany(JsonNode body) {
return body.get("size").asInt();
}

// This is required. consider returning a more useful error to the user than 500
// or setting a reasonable default (not sure what that would be)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

probably should return something like this instead of 500
{
"error": {
"code": 400,
"message": "Bad Request",
"details": "The 'query.gte' field is required but missing in the request."
}
}

private static long getStartTimeEpochMs(JsonNode body) {
return body.get("query").findValue("gte").asLong();
}

// This is required. consider returning a more useful error to the user than 500
// or setting a reasonable default (not sure what that would be)
private static long getEndTimeEpochMs(JsonNode body) {
return body.get("query").findValue("lte").asLong();
}
Expand Down