You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Thanos, Prometheus and Golang version used:
quay.io/thanos/thanos:v0.31.0
Object Storage Provider: own Object Storage with S3 endpoint
What happened:
At some point after a restart of a Thanos-store pod several blocks are no able to be loaded from our object storage , it is not a huge number but it still results in data loss
What you expected to happen:
The Thanos-store pod should be able to load all of the blocks.
Full logs to relevant components:
level=warn ts=2024-03-27T13:56:45.548274508Z caller=bucket.go:652 msg="loading block failed" elapsed=3.549132ms id=01GNH44X841X3PA2YF67AE4CVB err="create index header reader: write index header: new index reader: get object attributes of 01GNH44X841X3PA2YF67AE4CVB/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548551713Z caller=bucket.go:652 msg="loading block failed" elapsed=3.520884ms id=01HG5MJHV6536D5NKFMN6Z3HD7 err="create index header reader: write index header: new index reader: get object attributes of 01HG5MJHV6536D5NKFMN6Z3HD7/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548600296Z caller=bucket.go:652 msg="loading block failed" elapsed=2.923082ms id=01HG695QEDVAGJ9MCPYEFM1B1E err="create index header reader: write index header: new index reader: get object attributes of 01HG695QEDVAGJ9MCPYEFM1B1E/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548964003Z caller=bucket.go:652 msg="loading block failed" elapsed=3.079476ms id=01GTJ8VHZJ70EXBHDF7R9A1MZ2 err="create index header reader: write index header: new index reader: get object attributes of 01GTJ8VHZJ70EXBHDF7R9A1MZ2/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.549556296Z caller=bucket.go:652 msg="loading block failed" elapsed=3.047629ms id=01HFV43YJX0SY19SSJNJ4Q05WE err="create index header reader: write index header: new index reader: get object attributes of 01HFV43YJX0SY19SSJNJ4Q05WE/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.55066445Z caller=bucket.go:652 msg="loading block failed" elapsed=3.244963ms id=01H7J41MEM1EZGEAY9Y695CM1T err="create index header reader: write index header: new index reader: get object attributes of 01H7J41MEM1EZGEAY9Y695CM1T/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.566415912Z caller=bucket.go:652 msg="loading block failed" elapsed=21.854809ms id=01H5V67MFXA5PZG4VCF1SW5REM err="create index header reader: write index header: new index reader: get object attributes of 01H5V67MFXA5PZG4VCF1SW5REM/index: The specified key does not exist."
The text was updated successfully, but these errors were encountered:
Thanos, Prometheus and Golang version used:
quay.io/thanos/thanos:v0.31.0
Object Storage Provider: own Object Storage with S3 endpoint
What happened:
At some point after a restart of a Thanos-store pod several blocks are no able to be loaded from our object storage , it is not a huge number but it still results in data loss
What you expected to happen:
The Thanos-store pod should be able to load all of the blocks.
Full logs to relevant components:
level=warn ts=2024-03-27T13:56:45.548274508Z caller=bucket.go:652 msg="loading block failed" elapsed=3.549132ms id=01GNH44X841X3PA2YF67AE4CVB err="create index header reader: write index header: new index reader: get object attributes of 01GNH44X841X3PA2YF67AE4CVB/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548551713Z caller=bucket.go:652 msg="loading block failed" elapsed=3.520884ms id=01HG5MJHV6536D5NKFMN6Z3HD7 err="create index header reader: write index header: new index reader: get object attributes of 01HG5MJHV6536D5NKFMN6Z3HD7/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548600296Z caller=bucket.go:652 msg="loading block failed" elapsed=2.923082ms id=01HG695QEDVAGJ9MCPYEFM1B1E err="create index header reader: write index header: new index reader: get object attributes of 01HG695QEDVAGJ9MCPYEFM1B1E/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.548964003Z caller=bucket.go:652 msg="loading block failed" elapsed=3.079476ms id=01GTJ8VHZJ70EXBHDF7R9A1MZ2 err="create index header reader: write index header: new index reader: get object attributes of 01GTJ8VHZJ70EXBHDF7R9A1MZ2/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.549556296Z caller=bucket.go:652 msg="loading block failed" elapsed=3.047629ms id=01HFV43YJX0SY19SSJNJ4Q05WE err="create index header reader: write index header: new index reader: get object attributes of 01HFV43YJX0SY19SSJNJ4Q05WE/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.55066445Z caller=bucket.go:652 msg="loading block failed" elapsed=3.244963ms id=01H7J41MEM1EZGEAY9Y695CM1T err="create index header reader: write index header: new index reader: get object attributes of 01H7J41MEM1EZGEAY9Y695CM1T/index: The specified key does not exist."
level=warn ts=2024-03-27T13:56:45.566415912Z caller=bucket.go:652 msg="loading block failed" elapsed=21.854809ms id=01H5V67MFXA5PZG4VCF1SW5REM err="create index header reader: write index header: new index reader: get object attributes of 01H5V67MFXA5PZG4VCF1SW5REM/index: The specified key does not exist."
The text was updated successfully, but these errors were encountered: