Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Getting error with tempo deployment #4339

Open
Aniketsingh1 opened this issue Nov 18, 2024 · 2 comments
Open

Getting error with tempo deployment #4339

Aniketsingh1 opened this issue Nov 18, 2024 · 2 comments

Comments

@Aniketsingh1
Copy link

The following successfully deployed tempo as daemonset on one evening:
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: tempo
labels:
app: tempo
spec:
selector:
matchLabels:
app: tempo
template:
metadata:
labels:
app: tempo
spec:
nodeSelector:
nodepool: aksuser
containers:
- name: tempo
image: grafana/tempo:latest
args:
- -config.file=/etc/tempo/tempo.yaml
ports:
- containerPort: 3200
- containerPort: 4317 # OTLP gRPC port
- containerPort: 4318 # OTLP HTTP port
- containerPort: 9095 # Frontend port
volumeMounts:
- name: tempo-config
mountPath: /etc/tempo
- name: tempo-storage
mountPath: /var/tempo
securityContext:
runAsUser: 10001
runAsGroup: 10001
readinessProbe:
httpGet:
path: /metrics
port: 3200
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /metrics
port: 3200
initialDelaySeconds: 5
periodSeconds: 10
volumes:
- name: tempo-config
configMap:
name: tempo-config
- name: tempo-storage
persistentVolumeClaim:
claimName: tempo-pvc

apiVersion: v1
kind: ConfigMap
metadata:
name: tempo-config
labels:
app: tempo
data:
tempo.yaml: |
server:
http_listen_port: 3200

distributor:
  receivers:
    jaeger:
      protocols:
        grpc: {}
        thrift_binary: {}
        thrift_compact: {}
        thrift_http: {}
    otlp:
      protocols:
        grpc: 
        http: {}   # OTLP HTTP receiver

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces  # Ensure storage path is consistent
    block:
      bloom_filter_false_positive: .05
    wal:
      path: /var/tempo/wal
    pool:
      max_workers: 100
      queue_depth: 10000 

compactor:
  compaction:
    compaction_window: 1h
    max_block_bytes: 100_000_000
    block_retention: 1h
    compacted_block_retention: 10m

querier:
  frontend_worker:
    frontend_address: tempo.monitoring.svc.cluster.local:9095
    
frontend:
  queue:
    size: 10000
    workers: 10
    
metrics_generator:
  processor:
    span_metrics:
      dimensions:
        - http.method
        - http.status_code
        - service.version
    service_graphs:
      dimensions:
        - http.method
        - http.status_code
        - service.version
  storage:
    path: /tmp/tempo/generator/metrics
    remote_write:
      - url: http://mimir:9009/api/v1/push
        send_exemplars: true

overrides:
  metrics_generator_processors: ['service-graphs', 'span-metrics', 'local-blocks']

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: tempo-pvc
labels:
app: tempo
spec:
accessModes:

  • ReadWriteMany
    storageClassName: azurefile
    resources:
    requests:
    storage: 20Gi

apiVersion: v1
kind: Service
metadata:
name: tempo
labels:
app: tempo
spec:
ports:

  • port: 3200
    targetPort: 3200
    protocol: TCP
    name: http
  • port: 4317 # OTLP gRPC port
    targetPort: 4317
    protocol: TCP
    name: otlp-grpc
  • port: 4318 # OTLP HTTP port
    targetPort: 4318
    protocol: TCP
    name: otlp-http
  • port: 9095 # Frontend port
    targetPort: 9095
    protocol: TCP
    name: frontend
    selector:
    app: tempo

but suddenly i have started getting error
Found 8 pods, using pod/tempo-d6mm5
failed parsing config: failed to parse configFile /etc/tempo/tempo.yaml: yaml: unmarshal errors:
line 48: field frontend not found in type app.Config
which i am not able to get.
If i remove the section
frontend:
queue:
size: 10000
workers: 10
then nothing appears on grafana.
Please help in solving this
@mapno
Copy link
Member

mapno commented Nov 18, 2024

Hi @Aniketsingh1. The key for the frontend's config block is query_frontend, not frontend. You can refer to the config docs for more info: https://grafana.com/docs/tempo/latest/configuration/.

@Aniketsingh1
Copy link
Author

I have made the changes as per your suggestion . Please see below -

apiVersion: apps/v1
kind: DaemonSet
metadata:
name: tempo
labels:
app: tempo
spec:
selector:
matchLabels:
app: tempo
template:
metadata:
labels:
app: tempo
spec:
nodeSelector:
nodepool: aksuser
containers:
- name: tempo
image: grafana/tempo:latest
args:
- -config.file=/etc/tempo/tempo.yaml
ports:
- containerPort: 3200
- containerPort: 4317 # OTLP gRPC port
- containerPort: 4318 # OTLP HTTP port
- containerPort: 9095 # Frontend port
volumeMounts:
- name: tempo-config
mountPath: /etc/tempo
- name: tempo-storage
mountPath: /var/tempo
securityContext:
runAsUser: 10001
runAsGroup: 10001
readinessProbe:
httpGet:
path: /metrics
port: 3200
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /metrics
port: 3200
initialDelaySeconds: 5
periodSeconds: 10
volumes:
- name: tempo-config
configMap:
name: tempo-config
- name: tempo-storage
persistentVolumeClaim:
claimName: tempo-pvc

apiVersion: v1
kind: ConfigMap
metadata:
name: tempo-config
labels:
app: tempo
data:
tempo.yaml: |
server:
http_listen_port: 3200

distributor:
  receivers:
    jaeger:
      protocols:
        grpc: {}
        thrift_binary: {}
        thrift_compact: {}
        thrift_http: {}
    otlp:
      protocols:
        grpc: 
        http: {}   # OTLP HTTP receiver

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1

storage:
  trace:
    backend: local
    local:
      path: /var/tempo/traces  # Ensure storage path is consistent

compactor:
  compaction:
    compaction_window: 1h
    max_block_bytes: 100_000_000
    block_retention: 1h
    compacted_block_retention: 10m

querier:
  frontend_worker:
    frontend_address: tempo.monitoring.svc.cluster.local:9095
    
query_frontend:
    max_retries: 3
    max_batch_size: 10
    multi_tenant_queries_enabled: true
    search:
        concurrent_jobs: 500
        target_bytes_per_job: 104857600
        default_result_limit: 50
        max_result_limit: 1000
    metrics:
        concurrent_jobs: 500
    
metrics_generator:
  processor:
    span_metrics:
      dimensions:
        - http.method
        - http.status_code
        - service.version
    service_graphs:
      dimensions:
        - http.method
        - http.status_code
        - service.version
  storage:
    path: /tmp/tempo/generator/metrics
    remote_write:
      - url: http://mimir:9009/api/v1/push
        send_exemplars: true

overrides:
  metrics_generator_processors: ['service-graphs', 'span-metrics', 'local-blocks']

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: tempo-pvc
labels:
app: tempo
spec:
accessModes:

  • ReadWriteMany
    storageClassName: azurefile
    resources:
    requests:
    storage: 20Gi

apiVersion: v1
kind: Service
metadata:
name: tempo
labels:
app: tempo
spec:
ports:

  • port: 3200
    targetPort: 3200
    protocol: TCP
    name: http
  • port: 4317 # OTLP gRPC port
    targetPort: 4317
    protocol: TCP
    name: otlp-grpc
  • port: 4318 # OTLP HTTP port
    targetPort: 4318
    protocol: TCP
    name: otlp-http
  • port: 9095 # Frontend port
    targetPort: 9095
    protocol: TCP
    name: frontend
    selector:
    app: tempo

But for this my tempo pods keep on restarting and give following logs:

Image

level=error ts=2024-11-18T11:19:35.927530275Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:35.962757781Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:35.984148428Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.036197128Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.067920294Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.135006167Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.142163249Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.241419494Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.242481506Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.277725112Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.282868671Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.307168951Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.321308214Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.346485605Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.365782627Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.391990929Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.406182093Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.638882275Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.737145508Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.760388976Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.805709398Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.818890547Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.88404808Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.964289283Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.974415997Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:36.986648234Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.127188615Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.128290227Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.144462909Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.159807982Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.161869705Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.199088624Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.276237891Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.279418227Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.284651586Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.323869227Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.358041011Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.547357641Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.583641349Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.680879242Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.728063973Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.744240555Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.859507451Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.866734833Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.896936372Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.960163883Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.967344064Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.969509889Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:37.974671847Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.03808636Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.057272176Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.109454362Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.140725814Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.179933555Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.181083368Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.187267838Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.220053006Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.412246868Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.419379148Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.555755882Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.646964708Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.699164695Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.768408674Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.778561788Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.833817509Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.845072036Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.849270383Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.858508087Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.881759449Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.905957521Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.946136973Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.948290897Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:38.994501617Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.072793797Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.075958833Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.081144791Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.092390418Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.225764418Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.296967619Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.54340049Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.604614479Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.61987035Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.626016919Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.666223372Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.67144853Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.699631847Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.709843162Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.756049382Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.772182563Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.809418182Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.862632181Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.86880585Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.876955942Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.903263538Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:39.97549115Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.004721879Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.052137812Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.057438372Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.132790519Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.38623777Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.474408261Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.486642499Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.545844165Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.609018775Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.6111902Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.611296401Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.616475759Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.639753621Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.648968125Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.706229269Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.784445748Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.79173993Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=error ts=2024-11-18T11:19:40.793943855Z caller=frontend_processor.go:76 msg="error contacting frontend" address=10.100.57.252:9095 err="rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.100.57.252:9095: connect: connection refused""
level=warn ts=2024-11-18T11:19:44.623167023Z caller=server.go:2220 msg="GET /metrics 1.150312ms, error: write tcp 10.0.3.245:3200->10.0.2.68:36958: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:19:44.623400925Z caller=server.go:2220 msg="GET /metrics 1.392316ms, error: write tcp 10.0.3.245:3200->10.0.2.68:36960: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:19:54.623574327Z caller=server.go:2220 msg="GET /metrics 1.269814ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57610: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:19:54.623644428Z caller=server.go:2220 msg="GET /metrics 1.397615ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57608: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:04.623780857Z caller=server.go:2220 msg="GET /metrics 1.246113ms, error: write tcp 10.0.3.245:3200->10.0.2.68:56498: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:04.623920559Z caller=server.go:2220 msg="GET /metrics 1.385915ms, error: write tcp 10.0.3.245:3200->10.0.2.68:56500: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=info ts=2024-11-18T11:20:05.582257386Z caller=flush.go:249 msg="flushing block" tenant=single-tenant block=01898622-f55e-41f4-854a-7dc48e0ddd19
level=info ts=2024-11-18T11:20:05.582300786Z caller=flush.go:259 msg="completing block" tenant=single-tenant blockID=01898622-f55e-41f4-854a-7dc48e0ddd19
level=info ts=2024-11-18T11:20:05.590374574Z caller=flush.go:266 msg="block completed" tenant=single-tenant blockID=01898622-f55e-41f4-854a-7dc48e0ddd19 duration=8.071088ms
level=error ts=2024-11-18T11:20:05.590422175Z caller=flush.go:232 org_id=single-tenant msg="error performing op in flushQueue" op=0 block=01898622-f55e-41f4-854a-7dc48e0ddd19 attempts=2 err="error completing wal block with local backend: error creating iterator for /var/tempo/wal/01898622-f55e-41f4-854a-7dc48e0ddd19+single-tenant+vParquet4/0000000001: error opening file: open /var/tempo/wal/01898622-f55e-41f4-854a-7dc48e0ddd19+single-tenant+vParquet4/0000000001: no such file or directory"
level=info ts=2024-11-18T11:20:05.590442375Z caller=flush.go:394 org_id=single-tenant msg="retrying op in flushQueue" op=0 block=01898622-f55e-41f4-854a-7dc48e0ddd19 backoff=1m0s
level=warn ts=2024-11-18T11:20:14.622769387Z caller=server.go:2220 msg="GET /metrics 1.070512ms, error: write tcp 10.0.3.245:3200->10.0.2.68:43506: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:14.625128212Z caller=server.go:2220 msg="GET /metrics 3.418737ms, error: write tcp 10.0.3.245:3200->10.0.2.68:43504: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:24.623634231Z caller=server.go:2220 msg="GET /metrics 1.853118ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57114: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:24.623655431Z caller=server.go:2220 msg="GET /metrics 1.913118ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57106: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=info ts=2024-11-18T11:20:29.254574454Z caller=frontend.go:333 msg="received shutdown notification from querier" querier=tempo-2qjzf
level=warn ts=2024-11-18T11:20:34.623334276Z caller=server.go:2220 msg="GET /metrics 1.150412ms, error: write tcp 10.0.3.245:3200->10.0.2.68:48226: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:34.623672879Z caller=server.go:2220 msg="GET /metrics 1.490315ms, error: write tcp 10.0.3.245:3200->10.0.2.68:48228: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:44.62358927Z caller=server.go:2220 msg="GET /metrics 1.335614ms, error: write tcp 10.0.3.245:3200->10.0.2.68:49110: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:44.623890673Z caller=server.go:2220 msg="GET /metrics 1.638117ms, error: write tcp 10.0.3.245:3200->10.0.2.68:49112: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:54.623031183Z caller=server.go:2220 msg="GET /metrics 1.105712ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46412: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:20:54.623058383Z caller=server.go:2220 msg="GET /metrics 1.144012ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46408: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:04.623630028Z caller=server.go:2220 msg="GET /metrics 1.419915ms, error: write tcp 10.0.3.245:3200->10.0.2.68:40514: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:04.623631128Z caller=server.go:2220 msg="GET /metrics 1.419515ms, error: write tcp 10.0.3.245:3200->10.0.2.68:40512: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=info ts=2024-11-18T11:21:05.591470803Z caller=flush.go:249 msg="flushing block" tenant=single-tenant block=01898622-f55e-41f4-854a-7dc48e0ddd19
level=info ts=2024-11-18T11:21:05.591511803Z caller=flush.go:259 msg="completing block" tenant=single-tenant blockID=01898622-f55e-41f4-854a-7dc48e0ddd19
level=info ts=2024-11-18T11:21:05.597519863Z caller=flush.go:266 msg="block completed" tenant=single-tenant blockID=01898622-f55e-41f4-854a-7dc48e0ddd19 duration=6.00636ms
level=error ts=2024-11-18T11:21:05.597558063Z caller=flush.go:232 org_id=single-tenant msg="error performing op in flushQueue" op=0 block=01898622-f55e-41f4-854a-7dc48e0ddd19 attempts=3 err="error completing wal block with local backend: error creating iterator for /var/tempo/wal/01898622-f55e-41f4-854a-7dc48e0ddd19+single-tenant+vParquet4/0000000001: error opening file: open /var/tempo/wal/01898622-f55e-41f4-854a-7dc48e0ddd19+single-tenant+vParquet4/0000000001: no such file or directory"
level=error ts=2024-11-18T11:21:05.597577163Z caller=flush.go:271 org_id=single-tenant msg="Block exceeded max completion errors. Deleting. POSSIBLE DATA LOSS" tenant=single-tenant attempts=3 block=01898622-f55e-41f4-854a-7dc48e0ddd19
level=warn ts=2024-11-18T11:21:14.622929646Z caller=server.go:2220 msg="GET /metrics 1.202813ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46666: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:14.623037447Z caller=server.go:2220 msg="GET /metrics 1.310414ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46660: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:24.623203111Z caller=server.go:2220 msg="GET /metrics 1.272313ms, error: write tcp 10.0.3.245:3200->10.0.2.68:34660: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:24.623257111Z caller=server.go:2220 msg="GET /metrics 1.325514ms, error: write tcp 10.0.3.245:3200->10.0.2.68:34670: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:34.62358787Z caller=server.go:2220 msg="GET /metrics 1.104211ms, error: write tcp 10.0.3.245:3200->10.0.2.68:50592: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:34.623730871Z caller=server.go:2220 msg="GET /metrics 1.255412ms, error: write tcp 10.0.3.245:3200->10.0.2.68:50584: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=info ts=2024-11-18T11:21:44.281346334Z caller=frontend.go:333 msg="received shutdown notification from querier" querier=tempo-2qjzf
level=warn ts=2024-11-18T11:21:44.62361411Z caller=server.go:2220 msg="GET /metrics 1.287814ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46508: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:44.623793612Z caller=server.go:2220 msg="GET /metrics 1.445616ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46510: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:54.623922186Z caller=server.go:2220 msg="GET /metrics 1.404715ms, error: write tcp 10.0.3.245:3200->10.0.2.68:58170: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:21:54.624261789Z caller=server.go:2220 msg="GET /metrics 1.686417ms, error: write tcp 10.0.3.245:3200->10.0.2.68:58168: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:04.623180231Z caller=server.go:2220 msg="GET /metrics 1.292513ms, error: write tcp 10.0.3.245:3200->10.0.2.68:39456: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:04.623201131Z caller=server.go:2220 msg="GET /metrics 1.328614ms, error: write tcp 10.0.3.245:3200->10.0.2.68:39454: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:14.623128284Z caller=server.go:2220 msg="GET /metrics 1.221013ms, error: write tcp 10.0.3.245:3200->10.0.2.68:36272: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:14.623128284Z caller=server.go:2220 msg="GET /metrics 1.224213ms, error: write tcp 10.0.3.245:3200->10.0.2.68:36274: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:24.623292779Z caller=server.go:2220 msg="GET /metrics 1.205912ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57274: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:24.623646183Z caller=server.go:2220 msg="GET /metrics 1.547716ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57262: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:34.623536838Z caller=server.go:2220 msg="GET /metrics 1.107511ms, error: write tcp 10.0.3.245:3200->10.0.2.68:39218: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:34.623778641Z caller=server.go:2220 msg="GET /metrics 1.413914ms, error: write tcp 10.0.3.245:3200->10.0.2.68:39228: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:44.62401825Z caller=server.go:2220 msg="GET /metrics 1.275614ms, error: write tcp 10.0.3.245:3200->10.0.2.68:60368: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:44.62403795Z caller=server.go:2220 msg="GET /metrics 1.243013ms, error: write tcp 10.0.3.245:3200->10.0.2.68:60362: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:54.623255613Z caller=server.go:2220 msg="GET /metrics 1.140011ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57232: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:22:54.623346214Z caller=server.go:2220 msg="GET /metrics 1.231112ms, error: write tcp 10.0.3.245:3200->10.0.2.68:57234: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:23:04.623114383Z caller=server.go:2220 msg="GET /metrics 1.341114ms, error: write tcp 10.0.3.245:3200->10.0.2.68:54498: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:23:04.623107383Z caller=server.go:2220 msg="GET /metrics 1.350614ms, error: write tcp 10.0.3.245:3200->10.0.2.68:54500: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:23:14.623407884Z caller=server.go:2220 msg="GET /metrics 1.036011ms, error: write tcp 10.0.3.245:3200->10.0.2.68:49800: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:23:14.623735987Z caller=server.go:2220 msg="GET /metrics 1.363314ms, error: write tcp 10.0.3.245:3200->10.0.2.68:49790: write: broken pipe ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=info ts=2024-11-18T11:23:18.230597174Z caller=frontend.go:333 msg="received shutdown notification from querier" querier=tempo-2qjzf
level=warn ts=2024-11-18T11:23:24.623086562Z caller=server.go:2220 msg="GET /metrics 1.128811ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46958: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "
level=warn ts=2024-11-18T11:23:24.623173563Z caller=server.go:2220 msg="GET /metrics 1.219712ms, error: write tcp 10.0.3.245:3200->10.0.2.68:46956: write: connection reset by peer ws: false; Accept: /; Connection: close; User-Agent: kube-probe/1.29; "

Am i missing something?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Aniketsingh1 @mapno