[Bug]:[most urgent]segment can not loaded beacuse of index_null_offset not existed. #39889

melon1705 · 2025-02-14T07:01:25Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version:2.5.4
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):kafka
- SDK version(e.g. pymilvus v2.0.0rc2):2.5.4
- OS(Ubuntu or CentOS): CentOS
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Step1. Using Milvus 2.5.0, a collection was created with 500 partitions. This collection includes an IVF_FLAT type index. The collection schema is as follows:
`schema = MilvusClient.create_schema(
auto_id=False,
enable_dynamic_field=True,
)

schema.add_field(field_name="image_id", datatype=DataType.VARCHAR, is_primary=True, max_length=50, auto_id=False)
schema.add_field(field_name="image_vector", datatype=DataType.FLOAT_VECTOR, dim=1024)
schema.add_field(field_name="image_desc", datatype=DataType.VARCHAR, max_length=150, enable_analyzer=True, enable_match=True, analyzer_params={"type": "chinese"})
schema.add_field(field_name="index", datatype=DataType.VARCHAR, max_length=50)

index_params = client.prepare_index_params()

index_params.add_index(
field_name="image_vector",
index_type="IVF_FLAT",
index_name="image_vector_index",
metric_type="COSINE"
)`
Step2.The program started normally, but after running for a month, it failed to load the partition, which caused the vector search to be unavailable.the reason likes as follows:
2025/02/14 06:44:40.286 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]

Expected Behavior

The issue persists even after exporting the data from version 2.5.0 to the directory of version 2.5.4.I tried deleting the index and recreating it, but the issue was not resolved.

Steps To Reproduce

Created a collection in v2.5.0 or lower using the schema above and then upgrade Milvus to v2.5.4. Check if the collection can be loaded.

Milvus Log

I20250214 06:44:39.216392 646 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_LOAD][]inverted index remove path:
[2025/02/14 06:44:39.248 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
[2025/02/14 06:44:39.248 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
[2025/02/14 06:44:39.289 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
I20250214 06:44:40.195726 646 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 100 with num_rows 1
I20250214 06:44:40.195737 675 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 101 with num_rows 1
I20250214 06:44:40.195974 646 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 100 task to thread pool
I20250214 06:44:40.195771 643 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 104 with num_rows 816
I20250214 06:44:40.195722 548 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 104 with num_rows 1
I20250214 06:44:40.195744 587 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 104 with num_rows 3
I20250214 06:44:40.196064 548 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 104 task to thread pool
I20250214 06:44:40.196076 587 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 104 task to thread pool
I20250214 06:44:40.195799 244 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 100 with num_rows 3
I20250214 06:44:40.196004 675 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 101 task to thread pool
I20250214 06:44:40.195820 586 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 100 with num_rows 816
I20250214 06:44:40.195833 553 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 103 with num_rows 1
I20250214 06:44:40.196048 643 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 104 task to thread pool
I20250214 06:44:40.196403 586 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 100 task to thread pool
I20250214 06:44:40.196481 244 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 100 task to thread pool
I20250214 06:44:40.196491 553 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 103 task to thread pool
I20250214 06:44:40.196635 646 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 100 mmap false done
I20250214 06:44:40.196679 675 IndexConfigGenerator.cpp:71] [SERVER][VecIndexConfig][CGO_LOAD][]VecIndexConfig: origin_index_type=IVF_FLAT, index_type=IVF_FLAT_CC, metric_type=COSINE, config={"metric_type":"COSINE","nlist":"128","ssize":"48"}
I20250214 06:44:40.196704 675 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 101 mmap false done
I20250214 06:44:40.196782 587 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 104 mmap false done
I20250214 06:44:40.196796 675 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 0 with num_rows 1
I20250214 06:44:40.196877 646 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 102 with num_rows 816
I20250214 06:44:40.197628 675 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 0 task to thread pool
I20250214 06:44:40.196887 548 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 104 mmap false done
I20250214 06:44:40.197664 646 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 102 task to thread pool
I20250214 06:44:40.197796 548 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 0 with num_rows 816
I20250214 06:44:40.197830 548 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 0 task to thread pool
I20250214 06:44:40.197836 675 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 0 mmap false done
I20250214 06:44:40.197031 244 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 100 mmap false done
I20250214 06:44:40.197440 586 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 100 mmap false done
I20250214 06:44:40.197953 587 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 101 with num_rows 3
I20250214 06:44:40.197986 587 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 101 task to thread pool
I20250214 06:44:40.197553 553 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 103 mmap false done
I20250214 06:44:40.197990 586 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 1 with num_rows 1
I20250214 06:44:40.198027 586 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 1 task to thread pool
I20250214 06:44:40.198040 244 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 0 with num_rows 3
I20250214 06:44:40.198071 244 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 0 task to thread pool
I20250214 06:44:40.198122 553 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 1 with num_rows 816
I20250214 06:44:40.198141 553 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 1 task to thread pool
I20250214 06:44:40.198184 643 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 104 mmap false done
I20250214 06:44:40.198313 675 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 103 with num_rows 3
I20250214 06:44:40.198355 675 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 103 task to thread pool
I20250214 06:44:40.198370 244 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 0 mmap false done
I20250214 06:44:40.198444 643 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 103 with num_rows 816
I20250214 06:44:40.198463 587 IndexConfigGenerator.cpp:71] [SERVER][VecIndexConfig][CGO_LOAD][]VecIndexConfig: origin_index_type=IVF_FLAT, index_type=IVF_FLAT_CC, metric_type=COSINE, config={"metric_type":"COSINE","nlist":"128","ssize":"48"}
I20250214 06:44:40.198508 587 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 101 mmap false done
I20250214 06:44:40.198534 586 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 1 mmap false done
I20250214 06:44:40.198565 244 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 1 with num_rows 3
I20250214 06:44:40.198590 643 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 103 task to thread pool
I20250214 06:44:40.198640 244 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 1 task to thread pool
I20250214 06:44:40.198699 553 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 1 mmap false done
I20250214 06:44:40.198450 548 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 0 mmap false done
I20250214 06:44:40.198683 586 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 102 with num_rows 1
I20250214 06:44:40.198815 586 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 submits load field 102 task to thread pool
I20250214 06:44:40.198827 675 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 103 mmap false done
I20250214 06:44:40.198916 675 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 101 with num_rows 816
I20250214 06:44:40.198941 675 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 submits load field 101 task to thread pool
I20250214 06:44:40.198982 244 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 1 mmap false done
I20250214 06:44:40.199023 646 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 102 mmap false done
I20250214 06:44:40.199025 548 ChunkedSegmentSealedImpl.cpp:268] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 102 with num_rows 3
I20250214 06:44:40.199105 548 ChunkedSegmentSealedImpl.cpp:282] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 submits load field 102 task to thread pool
I20250214 06:44:40.199169 586 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831877730 loads field 102 mmap false done
I20250214 06:44:40.199206 643 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 103 mmap false done
=> invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51

I20250214 06:44:40.199466 643 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_LOAD][]inverted index remove path:
I20250214 06:44:40.199766 548 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175827916719 loads field 102 mmap false done
=> invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51

I20250214 06:44:40.203465 548 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_LOAD][]inverted index remove path:
I20250214 06:44:40.206964 675 IndexConfigGenerator.cpp:71] [SERVER][VecIndexConfig][CGO_LOAD][]VecIndexConfig: origin_index_type=IVF_FLAT, index_type=IVF_FLAT_CC, metric_type=COSINE, config={"metric_type":"COSINE","nlist":"128","ssize":"48"}
I20250214 06:44:40.207391 675 ChunkedSegmentSealedImpl.cpp:293] [SERVER][LoadFieldData][CGO_LOAD][]segment 455305175831852528 loads field 101 mmap false done
=> invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51

I20250214 06:44:40.214983 675 InvertedIndexTantivy.cpp:113] [SERVER][~InvertedIndexTantivy][CGO_LOAD][]inverted index remove path:
[2025/02/14 06:44:40.246 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175831877731/0/455210079704244207/455305175831877730/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
[2025/02/14 06:44:40.286 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175831852529/0/455210079704241976/455305175831852528/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]
[2025/02/14 06:44:40.286 +00:00] [ERROR] [funcutil/parallel.go:89] [loadSegmentFunc] [error="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n"] [errorVerbose="At LoadSegment: => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\n(1) attached stack trace\n -- stack trace:\n | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func5\n | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:348\n | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n | \t/workspace/source/pkg/util/funcutil/parallel.go:87\n | runtime.goexit\n | \t/go/pkg/mod/golang.org/[email protected]/src/runtime/asm_amd64.s:1695\nWraps: (2) At LoadSegment\nWraps: (3) => invalid local path:/var/lib/milvus/data/index_files/455305175827916720/0/455210079704244993/455305175827916719/index_null_offset at /workspace/source/internal/core/src/storage/LocalChunkManager.cpp:51\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:89"]

Anything else?

Are there any other ways to temporarily avoid this problem? The inability to load data is causing the database to be unable to perform query operations, which has a significant impact.

yanliang567 · 2025-02-14T07:33:20Z

@melon0803 please attach the milvus logs for investigation.
If you install Milvus with k8s, please refer this doc to export the whole Milvus logs.
If you install Milvus with docker-compose, please use docker-compose logs > milvus.log to export the logs.

/assign @melon0803

melon1705 · 2025-02-14T08:03:56Z

logs.tar.gzThis is the k8s pod log I exported using export-milvus-log.sh.

joenzx · 2025-02-14T08:40:14Z

I have similar problems, waiting for solutions

smellthemoon · 2025-02-14T10:20:06Z

From this log, we can see all the index files were missed. Text index will be build if you set **enable_match = true**, which can not be rebuild now. Are you rebuilding on a vector column? @melon0803

smellthemoon · 2025-02-14T10:23:19Z

It related with #39635. It try to load on dropped segment, which has gced index binlog.

smellthemoon · 2025-02-14T10:24:22Z

you can try to release it and load collection again. @melon0803

smellthemoon · 2025-02-14T10:26:36Z

/assign

Andy6132024 · 2025-02-14T10:32:34Z

@smellthemoon release and load didn't work for us. It still got stuck at loading. Also drop/recreate indexes didn't help either.

melon1705 · 2025-02-14T10:41:14Z

From this log, we can see all the index files were missed. Text index will be build if you set enable_match = true, which can not be rebuild now. Are you rebuilding on a vector column? @melon0803

yes, you are right. I have two fields.

The first one is image_desc, with the schema defined as follows:
schema.add_field(field_name="image_desc", datatype=DataType.VARCHAR, max_length=150, enable_analyzer=True, enable_match=True, analyzer_params={"type": "chinese"})
The second field is image_vector, with the schema defined as:
schema.add_field(field_name="image_vector", datatype=DataType.FLOAT_VECTOR, dim=1024)
index_params.add_index(
field_name="image_vector",
index_type="IVF_FLAT",
index_name="image_vector_index",
metric_type="COSINE"
)

melon1705 · 2025-02-14T10:43:11Z

It related with #39635. Try to load on dropped segment, which has gced index binlog.
can you tell me how to load data onto a dropped segment?

smellthemoon · 2025-02-14T10:45:41Z

It related with #39635. Try to load on dropped segment, which has gced index binlog.
can you tell me how to load data onto a dropped segment?

sorry. I meant the root cause of this issue is that load on dropped segment.

smellthemoon · 2025-02-14T10:51:26Z

@smellthemoon release and load didn't work for us. It still got stuck at loading. Also drop/recreate indexes didn't help either.

drop/recreate vector index will do no effect, because the index is build in scalar field. And text index can not be rebuild now. If release collection and load collection not work to you, could you help to provide your log for investigation?

melon1705 · 2025-02-14T10:57:22Z

@smellthemoon release and load didn't work for us. It still got stuck at loading. Also drop/recreate indexes didn't help either.

drop/recreate vector index will do no effect, because the index is build in scalar field. And text index can not be rebuild now. If release collection and load collection not work to you, could you help to provide your log?

logs.tar.gz this is my log in k8s，log only for release collection and load collection.

melon1705 · 2025-02-14T11:00:19Z

@smellthemoon release and load didn't work for us. It still got stuck at loading. Also drop/recreate indexes didn't help either.

drop/recreate vector index will do no effect, because the index is build in scalar field. And text index can not be rebuild now. If release collection and load collection not work to you, could you help to provide your log for investigation?

but index_null_offset param, if normal, the value is IVF_FLAT file. what this means?
example: folder /var/lib/milvus/data/index_files/455305175829023992/0/455115947100595870/455305175829023991/IVF_FLAT

smellthemoon · 2025-02-14T11:11:43Z

@smellthemoon release and load didn't work for us. It still got stuck at loading. Also drop/recreate indexes didn't help either.

drop/recreate vector index will do no effect, because the index is build in scalar field. And text index can not be rebuild now. If release collection and load collection not work to you, could you help to provide your log for investigation?

but index_null_offset param, if normal, the value is IVF_FLAT file. what this means?

If a field set enable_match=True, will create a text index which is built on "image_desc" field. IVF_FLAT is built on "image_vector". Now we can not drop the text index by DropIndex request. when create text index, will generate index_null_offset param. IVF_FLAT files will be genereated when build vector vector. They are two separate columns.

melon1705 · 2025-02-14T11:27:43Z

@smellthemoon release and load didn't work for us. It still got stuck at loading. Also drop/recreate indexes didn't help either.

drop/recreate vector index will do no effect, because the index is build in scalar field. And text index can not be rebuild now. If release collection and load collection not work to you, could you help to provide your log for investigation?

but index_null_offset param, if normal, the value is IVF_FLAT file. what this means?

If a field set enable_match=True, will create a text index which is built on "image_desc" field. IVF_FLAT is built on "image_vector". Now we can not drop the text index by DropIndex request. when create text index, will generate index_null_offset param. IVF_FLAT files will be genereated when build vector vector. They are two separate columns.

Thank you. I understand. But index_null_offset is not created, how to solve the problem? Can you share your thoughts?

xiaofan-luan · 2025-02-15T04:41:08Z

@yanliang567
We need to reproduce an fixed it in 2.5.5

yanliang567 · 2025-02-17T03:55:53Z

@melon0803 could you please refer to this doc: https://github.com/milvus-io/birdwatcher to backup etcd backup with birdwatcher, and attache the birdwatcher backup file for investigation?

congqixia · 2025-02-17T04:07:58Z

@melon0803 quick question
do you have any other index except vector and text match field?

melon1705 · 2025-02-17T05:36:37Z

@melon0803 quick question do you have any other index except vector and text match field?
I only have 4 fields in a collection, like this:
schema.add_field(field_name="image_id", datatype=DataType.VARCHAR, is_primary=True, max_length=50, auto_id=False)
schema.add_field(field_name="image_vector", datatype=DataType.FLOAT_VECTOR, dim=1024)
schema.add_field(field_name="image_desc", datatype=DataType.VARCHAR, max_length=150, enable_analyzer=True, enable_match=True, analyzer_params={"type": "chinese"})
schema.add_field(field_name="index", datatype=DataType.VARCHAR, max_length=50)

melon1705 · 2025-02-17T05:38:55Z

@melon0803 could you please refer to this doc: https://github.com/milvus-io/birdwatcher to backup etcd backup with birdwatcher, and attache the birdwatcher backup file for investigation?

Ok, I'll try

melon1705 · 2025-02-17T06:24:31Z

@melon0803 could you please refer to this doc: https://github.com/milvus-io/birdwatcher to backup etcd backup with birdwatcher, and attache the birdwatcher backup file for investigation?

like this picture, when i execute go install , $GOPATH/bin/birdwatcher binary file is not generate success.

congqixia · 2025-02-17T06:27:27Z

@melon0803 you could just pre-compiled binary in this link https://github.com/milvus-io/birdwatcher/releases/tag/v1.0.7

melon1705 · 2025-02-17T06:45:45Z

@melon0803 could you please refer to this doc: https://github.com/milvus-io/birdwatcher to backup etcd backup with birdwatcher, and attache the birdwatcher backup file for investigation?

hello, backup file has uploaded.

bw_etcd_ALL.250217-064143.bak.gz

congqixia · 2025-02-17T10:45:49Z

@melon1705 after some digging of the backup file, we found that index_null_offset for text match index used wrong prefix and got GCed unintentionally.
We shall fix this bug in next release.
As for your case, you could try compact our collection with this issue, in which could trigger regeneration of index file

melon1705 · 2025-02-18T03:27:06Z

@melon1705 after some digging of the backup file, we found that index_null_offset for text match index used wrong prefix and got GCed unintentionally. We shall fix this bug in next release. As for your case, you could try compact our collection with this issue, in which could trigger regeneration of index file

Thank you for your answer. but how to compact the specified collection？

yanliang567 · 2025-02-18T04:59:46Z

collection.compact() will issue a manual compaction request to the collection.

@melon1705 after some digging of the backup file, we found that index_null_offset for text match index used wrong prefix and got GCed unintentionally. We shall fix this bug in next release. As for your case, you could try compact our collection with this issue, in which could trigger regeneration of index file

Thank you for your answer. but how to compact the specified collection？

melon1705 · 2025-02-18T05:44:45Z

collection.compact() will issue a manual compaction request to the collection.

@melon1705 after some digging of the backup file, we found that index_null_offset for text match index used wrong prefix and got GCed unintentionally. We shall fix this bug in next release. As for your case, you could try compact our collection with this issue, in which could trigger regeneration of index file

Thank you for your answer. but how to compact the specified collection？

sorry, I did it(collection.compact()), but no index rebuild was triggered

melon1705 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 14, 2025

melon1705 assigned yanliang567 Feb 14, 2025

melon1705 mentioned this issue Feb 14, 2025

[Bug]: [Urgent] Failed to LoadSegment because index file index_null_offset is missing #39881

Open

1 task

sre-ci-robot assigned melon1705 Feb 14, 2025

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. triage/needs-information Indicates an issue needs more information in order to work on it. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Feb 14, 2025

sre-ci-robot assigned smellthemoon Feb 14, 2025

xiaofan-luan added this to the 2.5.5 milestone Feb 15, 2025

yanliang567 added priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. and removed triage/needs-information Indicates an issue needs more information in order to work on it. labels Feb 17, 2025

melon1705 closed this as completed Feb 17, 2025

congqixia reopened this Feb 17, 2025

congqixia mentioned this issue Feb 17, 2025

[Bug]: index_null_offset of TextMatchIndex using wrong prefix and could be GCed unintentionally #39933

Closed

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]:[most urgent]segment can not loaded beacuse of index_null_offset not existed. #39889

[Bug]:[most urgent]segment can not loaded beacuse of index_null_offset not existed. #39889

melon1705 commented Feb 14, 2025

yanliang567 commented Feb 14, 2025

melon1705 commented Feb 14, 2025

joenzx commented Feb 14, 2025

smellthemoon commented Feb 14, 2025

smellthemoon commented Feb 14, 2025 •

edited

Loading

smellthemoon commented Feb 14, 2025

smellthemoon commented Feb 14, 2025

Andy6132024 commented Feb 14, 2025 •

edited

Loading

melon1705 commented Feb 14, 2025

melon1705 commented Feb 14, 2025

smellthemoon commented Feb 14, 2025

smellthemoon commented Feb 14, 2025 •

edited

Loading

melon1705 commented Feb 14, 2025

melon1705 commented Feb 14, 2025 •

edited

Loading

smellthemoon commented Feb 14, 2025

melon1705 commented Feb 14, 2025 •

edited

Loading

xiaofan-luan commented Feb 15, 2025

yanliang567 commented Feb 17, 2025

congqixia commented Feb 17, 2025

melon1705 commented Feb 17, 2025 •

edited

Loading

melon1705 commented Feb 17, 2025

melon1705 commented Feb 17, 2025 •

edited

Loading

congqixia commented Feb 17, 2025

melon1705 commented Feb 17, 2025

congqixia commented Feb 17, 2025

melon1705 commented Feb 18, 2025

yanliang567 commented Feb 18, 2025

melon1705 commented Feb 18, 2025

[Bug]:[most urgent]segment can not loaded beacuse of index_null_offset not existed. #39889

[Bug]:[most urgent]segment can not loaded beacuse of index_null_offset not existed. #39889

Comments

melon1705 commented Feb 14, 2025

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

yanliang567 commented Feb 14, 2025

melon1705 commented Feb 14, 2025

joenzx commented Feb 14, 2025

smellthemoon commented Feb 14, 2025

smellthemoon commented Feb 14, 2025 • edited Loading

smellthemoon commented Feb 14, 2025

smellthemoon commented Feb 14, 2025

Andy6132024 commented Feb 14, 2025 • edited Loading

melon1705 commented Feb 14, 2025

melon1705 commented Feb 14, 2025

smellthemoon commented Feb 14, 2025

smellthemoon commented Feb 14, 2025 • edited Loading

melon1705 commented Feb 14, 2025

melon1705 commented Feb 14, 2025 • edited Loading

smellthemoon commented Feb 14, 2025

melon1705 commented Feb 14, 2025 • edited Loading

xiaofan-luan commented Feb 15, 2025

yanliang567 commented Feb 17, 2025

congqixia commented Feb 17, 2025

melon1705 commented Feb 17, 2025 • edited Loading

melon1705 commented Feb 17, 2025

melon1705 commented Feb 17, 2025 • edited Loading

congqixia commented Feb 17, 2025

melon1705 commented Feb 17, 2025

congqixia commented Feb 17, 2025

melon1705 commented Feb 18, 2025

yanliang567 commented Feb 18, 2025

melon1705 commented Feb 18, 2025

smellthemoon commented Feb 14, 2025 •

edited

Loading

Andy6132024 commented Feb 14, 2025 •

edited

Loading

smellthemoon commented Feb 14, 2025 •

edited

Loading

melon1705 commented Feb 14, 2025 •

edited

Loading

melon1705 commented Feb 14, 2025 •

edited

Loading

melon1705 commented Feb 17, 2025 •

edited

Loading

melon1705 commented Feb 17, 2025 •

edited

Loading