Skip to content

Commit

Permalink
docs: Vector fixes (#5094)
Browse files Browse the repository at this point in the history
* docs: using-vector.md: Adjust Vector remap configuration to silence errors/warnings

* docs: using-vector.md: Provide a link to the index configuration code so it doesn't go out of sync
  • Loading branch information
jpds authored Jul 5, 2024
1 parent 1fba1d1 commit 622a12f
Showing 1 changed file with 5 additions and 78 deletions.
83 changes: 5 additions & 78 deletions docs/log-management/send-logs/using-vector.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,83 +43,9 @@ docker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 7280:7280 quickwit/quickwit

## Taking advantage of Quickwit's native support for logs

Let's embrace the OpenTelemetry standard and take advantage of Quickwit features. With the native support for OpenTelemetry standards, Quickwit already comes with an index called `otel-logs_v0_6` that is compatible with the OpenTelemetry [logs data model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md). This means we can start pushing log data without any prior usual index setup.

Here is the OpenTelemetry index configuration for reference.

```yaml title="otel-logs_v0_6"
version: 0.7

index_id: otel-logs-v0_6

doc_mapping:
mode: strict
field_mappings:
- name: timestamp_nanos
type: datetime
input_formats: [unix_timestamp]
output_format: unix_timestamp_nanos
indexed: false
fast: true
fast_precision: milliseconds
- name: observed_timestamp_nanos
type: datetime
input_formats: [unix_timestamp]
output_format: unix_timestamp_nanos
- name: service_name
type: text
tokenizer: raw
- name: severity_text
type: text
tokenizer: raw
fast: true
- name: severity_number
type: u64
fast: true
- name: body
type: json
- name: attributes
type: json
tokenizer: raw
fast: true
- name: dropped_attributes_count
type: u64
indexed: false
- name: trace_id
type: bytes
- name: span_id
type: bytes
- name: trace_flags
type: u64
indexed: false
- name: resource_attributes
type: json
tokenizer: raw
fast: true
- name: resource_dropped_attributes_count
type: u64
indexed: false
- name: scope_name
type: text
indexed: false
- name: scope_version
type: text
indexed: false
- name: scope_attributes
type: json
indexed: false
- name: scope_dropped_attributes_count
type: u64
indexed: false

timestamp_field: timestamp_nanos

indexing_settings:
commit_timeout_secs: 10

search_settings:
default_search_fields: [body.message]
```
Let's embrace the OpenTelemetry standard and take advantage of Quickwit features. With the native support for OpenTelemetry standards, Quickwit already comes with an index called `otel-logs_v0_7` that is compatible with the OpenTelemetry [logs data model](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/logs/data-model.md). This means we can start pushing log data without any prior usual index setup.

The OpenTelemetry index configuration can be found in the [quickwit-opentelemetry/src/otlp/logs.rs](https://github.com/quickwit-oss/quickwit/blob/main/quickwit/quickwit-opentelemetry/src/otlp/logs.rs) source file.

## Setup Vector

Expand All @@ -140,7 +66,7 @@ inputs = [ "generate_syslog"]
type = "remap"
source = '''
structured = parse_syslog!(.message)
.timestamp_nanos, err = to_unix_timestamp(structured.timestamp, unit: "nanoseconds")
.timestamp_nanos = to_unix_timestamp!(structured.timestamp, unit: "nanoseconds")
.body = structured
.service_name = structured.appname
.resource_attributes.source_type = .source_type
Expand All @@ -163,6 +89,7 @@ source = '''
.scope_name = structured.msgid
del(.message)
del(.timestamp)
del(.service)
del(.source_type)
'''

Expand Down

0 comments on commit 622a12f

Please sign in to comment.