Ingester: Validate completed blocks

[joe-elliott]

What this PR does:
Adds a Validate() method to backend blocks and uses it in ingester startup to do basic confirmation of block health before reloading them. This prevents corrupt blocks from being loaded and eventually pushed to object storage.

Which issue(s) this PR fixes:
Fixes #4166 as near as we can tell. At least it covers the instance seen internally.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[javiermolinar]

I wonder if it would not be simpler trying to open the file directly with parquet-go:

	o := []parquet.FileOption{
		parquet.SkipBloomFilters(true),
		parquet.SkipPageIndex(true),
		parquet.FileSchema(parquetSchema),
		parquet.FileReadMode(parquet.ReadModeAsync),
	}

	reader := NewBackendReaderAt(ctx, b.r, DataFileName, b.meta)
	_, err := parquet.OpenFile(reader, int64(b.meta.Size_), o...)
	if err != nil {
		return fmt.Errorf("failed to read parquet fike: %w", err)
	}

OpenFile seems to do essentially the same plus some other validations. Is this difference noticeable enough? Since this is at startup time maybe it isn't.

[joe-elliott]

OpenFile also unmarshals the thrift footer which is more costly. I was trying to avoid paying those allocs on startup.

[zalegrala]

Is there a way to reassemble this information from data on disk and still avoid paying the allocs on startup like you mention above?

[joe-elliott]

possibly? what do you mean by "this information"?

[zalegrala]

Looks good to me.