HnswDensevector SafeTensor Generator #2515

Panizghi · 2024-06-02T05:06:59Z

Linked issue : castorini/ura-projects#31 (comment)
@17Melissa will provide the flow command below :)

17Melissa · 2024-06-02T05:27:14Z

Setup for NFCorpus Indexing with Safetensors

To efficiently perform NFCorpus indexing using Safetensors, follow this setup workflow:

Download and Unzip Collections
- Begin by downloading the necessary collections and unzipping them. For instance:
  wget https://rgw.cs.uwaterloo.ca/pyserini/data/beir-v1.0.0-bge-base-en-v1.5.tar -P collections/tar xvf collections/beir-v1.0.0-bge-base-en-v1.5.tar -C collections/
Prepare the Environment
- Navigate to the Safetensors directory within the Anserini project
  cd /anserini/src/main/python/safetensors
- Install the required Python packages:
  pip install -r requirements.txt
- Activate the virtual environment
  python3 -m venv venv
  source venv/bin/activate
Convert JSON to Safetensors Format
- Use the provided script to convert JSON files to Safetensors format
  python3 -m json_to_bin
- the script will create the following files in the target directory
  - Saved vectors to ../../../../target/safetensors/vectors/part00_vectors.safetensors
  - Saved docids to ../../../../target/safetensors/docids/part00_docids.safetensors
  - Saved docid_to_idx mapping to ../../../../target/safetensors/docid_to_idx/part00_docid_to_idx.json

Indexing Procedure

To build HNSWSafetensors indexes, use the following sample command:

bin/run.sh io.anserini.index.SafeTensorsIndexCollection \
  -collection JsonDenseVectorCollection \
  -input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus  \
  -index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ \
  -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator \
  -threads 9 -storePositions -storeDocvectors -storeRaw \
  -vectorsDirectory target\safetesnors\vectors \
  -docidsDirectory  target\safetesnors\docids \
  -docidToIdxDirectory  target\safetesnors\docid_to_idx \
>& logs/log.beir-v1.0.0-bge-base-en-v1.5 &

Ensure all paths and parameters are adjusted according to your setup and directory structure.

lintool · 2024-06-02T11:45:00Z

Can you make the safetensors collection go into collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/, alongside the original? So all files should go into collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/.

We also shouldn't need a new indexer. The indexing command should be similar to https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-nfcorpus-bge-base-en-v1.5-hnsw.md

e.g.,

bin/run.sh io.anserini.index.IndexHnswDenseVectors \
  -collection JsonDenseVectorCollection \
  -input /path/to/beir-v1.0.0-bge-base-en-v1.5 \
  -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator \
  -index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus-bge-base-en-v1.5/ \
  -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge \
  >& logs/log.beir-v1.0.0-bge-base-en-v1.5 &

With the only exception being a different -generator.

17Melissa · 2024-06-15T15:44:11Z

Updated Workflow for Safetensors Conversion and Indexing Process

Create Directory: Create the safetensors folder collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus
Run Conversion Script: Execute the python script json_to_bin.py from the root directory using the command:
python src/main/python/safetensors/json_to_bin.py
Execute Indexing Command: Following the indexing command below, which you will run after the conversion script completes

bin/run.sh io.anserini.index.IndexHnswDenseVectors -collection JsonDenseVectorCollection -input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus  -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator -index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.5 &

...in/java/io/anserini/index/generator/HnswJsonWithSafeTensorsDenseVectorDocumentGenerator.java

src/main/python/safetensors/json_to_bin.py

Panizghi · 2024-07-09T04:43:49Z

Updates

Removed hardcoded path.
Removed indexer arguments and updated the path hierarchy.
Internal mapping of the docid and vectors.
Updated argument for Python input and output.

Updated commands

Python

python src/main/python/safetensors/json_to_bin.py --input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/vectors.part00.jsonl --output collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus

Java

bin/run.sh io.anserini.index.IndexHnswDenseVectors -collection JsonDenseVectorCollection -input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus -generator HnswJsonWithSafeTensorsDenseVectorDocumentGenerator -index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ -threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.112 &

Melissa

Panizghi · 2024-07-15T07:19:44Z

updated command :

bin/run.sh io.anserini.index.IndexHnswDenseVectors 
-collection SafeTensorsDenseVectorCollection 
-input collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus   
-generator SafeTensorsDenseVectorDocumentGenerator 
-index indexes/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/ 
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge  >& logs/log.beir-v1.0.0-nq.bge-base-en-v1.5 &

src/main/java/io/anserini/collection/SafeTensorsDenseVectorCollection.java

...ava/io/anserini/index/generator/HnswJsonWithSafeTensorsDenseVectorDocumentGeneratorTest.java

lintool · 2024-08-23T11:29:01Z

@Panizghi if I'm reading your code correctly, you're assuming that there's only one vector file per directory, right? This is not necessary the case.

For example, for robust04:

$ ls robust04/
vectors.part00.jsonl.gz  vectors.part01.jsonl.gz  vectors.part02.jsonl.gz  vectors.part03.jsonl.gz  vectors.part04.jsonl.gz  vectors.part05.jsonl.gz

lintool · 2024-08-23T11:34:28Z

@Panizghi on your branch, running:

$ python src/main/python/safetensors/json_to_bin.py \
--input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/vectors.part00.jsonl.gz \
--output collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus

Works fine. However, I would like some progress indication... e.g., using tqdm?

Also, what do I do if there is more than one vector part?

However, more compact, as excepted, which is good.

$ du -h collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/
22M	collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/
$ du -h collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/
84M	collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/

lintool · 2024-08-23T11:42:16Z

Running indexing command:

bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-collection SafeTensorsDenseVectorCollection \
-input collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus \
-generator SafeTensorsDenseVectorDocumentGenerator \
-index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/ \
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge

Something's not right... get an exception:

2024-08-23 07:40:33,960 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:205) - Setting log level to INFO
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:208) - ============ Loading Index Configuration ============
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:209) - AbstractIndexer settings:
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:210) -  + DocumentCollection path: collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:211) -  + CollectionClass: SafeTensorsDenseVectorCollection
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:212) -  + Index path: indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/
2024-08-23 07:40:33,964 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:213) -  + Threads: 16
2024-08-23 07:40:33,964 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:214) -  + Optimize (merge segments)? false
Aug 23, 2024 7:40:34 AM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
INFO: Using MemorySegmentIndexInput with Java 21; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2024-08-23 07:40:34,217 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:149) - HnswIndexer settings:
2024-08-23 07:40:34,217 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:150) -  + Generator: SafeTensorsDenseVectorDocumentGenerator
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:151) -  + M: 16
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:152) -  + efC: 100
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:153) -  + Store document vectors? false
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:154) -  + Int8 quantization? false
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:155) -  + Codec: Lucene99
2024-08-23 07:40:34,219 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:156) -  + MemoryBuffer: 65536
2024-08-23 07:40:34,219 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:157) -  + MaxThreadMemoryBeforeFlush: 2047
2024-08-23 07:40:34,219 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:160) -  + MergePolicy: NoMerge
2024-08-23 07:40:34,219 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:238) - ============ Indexing Collection ============
2024-08-23 07:40:34,222 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:247) - Thread pool with 16 threads initialized.
2024-08-23 07:40:34,222 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:248) - 2 files found in collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus
2024-08-23 07:40:34,222 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:249) - Starting to index...
2024-08-23 07:40:34,225 INFO  [pool-2-thread-1] generator.SafeTensorsDenseVectorDocumentGenerator (SafeTensorsDenseVectorDocumentGenerator.java:48) - Processing document ID: MED-10 with thread: pool-2-thread-1
2024-08-23 07:40:34,225 WARN  [pool-2-thread-2] generator.SafeTensorsDenseVectorDocumentGenerator (SafeTensorsDenseVectorDocumentGenerator.java:43) - Document ID: MED-10 is already being processed by another thread.
java.lang.NullPointerException
	at java.base/java.util.Objects.requireNonNull(Objects.java:233)
	at java.base/java.util.ImmutableCollections$List12.<init>(ImmutableCollections.java:563)
	at java.base/java.util.List.of(List.java:937)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1837)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1476)
	at io.anserini.index.AbstractIndexer$IndexerThread.run(AbstractIndexer.java:135)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2024-08-23 07:40:34,229 ERROR [pool-2-thread-2] index.AbstractIndexer$IndexerThread (AbstractIndexer.java:179) - pool-2-thread-2: Unexpected Exception:
2024-08-23 07:40:34,235 INFO  [pool-2-thread-1] generator.SafeTensorsDenseVectorDocumentGenerator (SafeTensorsDenseVectorDocumentGenerator.java:56) - Vector length: 768 for document ID: MED-10
Aug 23, 2024 7:40:34 AM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>
INFO: Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
2024-08-23 07:40:34,277 INFO  [pool-2-thread-1] generator.SafeTensorsDenseVectorDocumentGenerator (SafeTensorsDenseVectorDocumentGenerator.java:64) - Document created for ID: MED-10
20

Panizghi · 2024-08-25T04:33:10Z

@Panizghi if I'm reading your code correctly, you're assuming that there's only one vector file per directory, right? This is not necessary the case.

For example, for robust04:
$ ls robust04/
vectors.part00.jsonl.gz  vectors.part01.jsonl.gz  vectors.part02.jsonl.gz  vectors.part03.jsonl.gz  vectors.part04.jsonl.gz  vectors.part05.jsonl.gz

Yes that is correct on the early discussion we kep it only for nfcorpus with single file, I will update the code for the multiple file handling

Panizghi · 2024-08-25T04:34:27Z

Running indexing command:

bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-collection SafeTensorsDenseVectorCollection \
-input collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus \
-generator SafeTensorsDenseVectorDocumentGenerator \
-index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/ \
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge

Something's not right... get an exception:

2024-08-23 07:40:33,960 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:205) - Setting log level to INFO
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:208) - ============ Loading Index Configuration ============
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:209) - AbstractIndexer settings:
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:210) -  + DocumentCollection path: collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:211) -  + CollectionClass: SafeTensorsDenseVectorCollection
2024-08-23 07:40:33,963 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:212) -  + Index path: indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/
2024-08-23 07:40:33,964 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:213) -  + Threads: 16
2024-08-23 07:40:33,964 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:214) -  + Optimize (merge segments)? false
Aug 23, 2024 7:40:34 AM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
INFO: Using MemorySegmentIndexInput with Java 21; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
2024-08-23 07:40:34,217 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:149) - HnswIndexer settings:
2024-08-23 07:40:34,217 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:150) -  + Generator: SafeTensorsDenseVectorDocumentGenerator
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:151) -  + M: 16
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:152) -  + efC: 100
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:153) -  + Store document vectors? false
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:154) -  + Int8 quantization? false
2024-08-23 07:40:34,218 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:155) -  + Codec: Lucene99
2024-08-23 07:40:34,219 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:156) -  + MemoryBuffer: 65536
2024-08-23 07:40:34,219 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:157) -  + MaxThreadMemoryBeforeFlush: 2047
2024-08-23 07:40:34,219 INFO  [main] index.IndexHnswDenseVectors (IndexHnswDenseVectors.java:160) -  + MergePolicy: NoMerge
2024-08-23 07:40:34,219 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:238) - ============ Indexing Collection ============
2024-08-23 07:40:34,222 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:247) - Thread pool with 16 threads initialized.
2024-08-23 07:40:34,222 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:248) - 2 files found in collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus
2024-08-23 07:40:34,222 INFO  [main] index.AbstractIndexer (AbstractIndexer.java:249) - Starting to index...
2024-08-23 07:40:34,225 INFO  [pool-2-thread-1] generator.SafeTensorsDenseVectorDocumentGenerator (SafeTensorsDenseVectorDocumentGenerator.java:48) - Processing document ID: MED-10 with thread: pool-2-thread-1
2024-08-23 07:40:34,225 WARN  [pool-2-thread-2] generator.SafeTensorsDenseVectorDocumentGenerator (SafeTensorsDenseVectorDocumentGenerator.java:43) - Document ID: MED-10 is already being processed by another thread.
java.lang.NullPointerException
	at java.base/java.util.Objects.requireNonNull(Objects.java:233)
	at java.base/java.util.ImmutableCollections$List12.<init>(ImmutableCollections.java:563)
	at java.base/java.util.List.of(List.java:937)
	at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1837)
	at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1476)
	at io.anserini.index.AbstractIndexer$IndexerThread.run(AbstractIndexer.java:135)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
2024-08-23 07:40:34,229 ERROR [pool-2-thread-2] index.AbstractIndexer$IndexerThread (AbstractIndexer.java:179) - pool-2-thread-2: Unexpected Exception:
2024-08-23 07:40:34,235 INFO  [pool-2-thread-1] generator.SafeTensorsDenseVectorDocumentGenerator (SafeTensorsDenseVectorDocumentGenerator.java:56) - Vector length: 768 for document ID: MED-10
Aug 23, 2024 7:40:34 AM org.apache.lucene.internal.vectorization.PanamaVectorizationProvider <init>
INFO: Java vector incubator API enabled; uses preferredBitSize=256; FMA enabled
2024-08-23 07:40:34,277 INFO  [pool-2-thread-1] generator.SafeTensorsDenseVectorDocumentGenerator (SafeTensorsDenseVectorDocumentGenerator.java:64) - Document created for ID: MED-10
20

This should be fixed now and work with the same command

Panizghi · 2024-08-25T04:55:39Z

@Panizghi on your branch, running:
$ python src/main/python/safetensors/json_to_bin.py \
--input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/vectors.part00.jsonl.gz \
--output collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus
Works fine. However, I would like some progress indication... e.g., using tqdm?

Also, what do I do if there is more than one vector part?

However, more compact, as excepted, which is good.
$ du -h collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/
22M	collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/
$ du -h collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/
84M	collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/

tqdm is added there is --overwrite in arguments which also you can use if the file already exists

command:

python src/main/python/safetensors/json_to_bin.py \
--input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/vectors.part00.jsonl \
--output collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus --overwrite

sample output :

Processing lines: 100%|█████████████████████████████████████████████████| 3633/3633 [00:01<00:00, 3347.56it/s]
2024-08-25 00:53:02,642 - INFO - Saved vectors to collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/vectors.part00_vectors.safetensors
2024-08-25 00:53:02,643 - INFO - Saved docids to collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/vectors.part00_docids.safetensors
2024-08-25 00:53:02,643 - INFO - Loaded vectors from collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/vectors.part00_vectors.safetensors
2024-08-25 00:53:02,644 - INFO - Loaded document IDs (ASCII) from collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus/vectors.part00_docids.safetensors

For vector parts are we considering a case like this?

{
  "docid": "MED-10",
  "vector_1": [0.00344, 0.00231, ...],
  "vector_2": [0.00112, 0.00456, ...]
}

lintool

Please update Submodule tools in your branch to bring up to date w/ master.

lintool · 2024-08-25T13:13:03Z

Okay, I can now run these commands:

python src/main/python/safetensors/json_to_bin.py \
--input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/vectors.part00.jsonl.gz \
--output collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus --overwrite

bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-collection SafeTensorsDenseVectorCollection \
-input collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus \
-generator SafeTensorsDenseVectorDocumentGenerator \
-index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/ \
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge

After I build the index, I should be able to switch to retrieval, here: https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-nfcorpus.bge-base-en-v1.5.hnsw.onnx.md

The retrieval command is this:

bin/run.sh io.anserini.search.SearchHnswDenseVectors \
  -index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/ \
  -topics tools/topics-and-qrels/topics.beir-v1.0.0-nfcorpus.test.tsv.gz \
  -topicReader TsvString \
  -output runs/run.beir-v1.0.0-nfcorpus.bge-base-en-v1.5.bge-hnsw-onnx.topics.beir-v1.0.0-nfcorpus.test.txt \
  -generator VectorQueryGenerator -topicField title -removeQuery -threads 16 -hits 1000 -efSearch 1000 -encoder BgeBaseEn15

But the eval command generates errors:

$ bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-nfcorpus.test.txt runs/run.beir-v1.0.0-nfcorpus.bge-base-en-v1.5.bge-hnsw-onnx.topics.beir-v1.0.0-nfcorpus.test.txt
WARNING: Using incubator modules: jdk.incubator.vector
trec_eval.form_res_qrels: duplicate docs MED-1000trec_eval: Can't calculate measure 'ndcg_cut'

From here:

$ head runs/run.beir-v1.0.0-nfcorpus.bge-base-en-v1.5.bge-hnsw-onnx.topics.beir-v1.0.0-nfcorpus.test.txt
PLAIN-1008 Q0 MED-2036 1 0.776562 Anserini
PLAIN-1008 Q0 MED-2036 2 0.776562 Anserini
PLAIN-1008 Q0 MED-5135 3 0.775252 Anserini
PLAIN-1008 Q0 MED-5135 4 0.775252 Anserini
PLAIN-1008 Q0 MED-4694 5 0.774549 Anserini
PLAIN-1008 Q0 MED-4694 6 0.774549 Anserini
PLAIN-1008 Q0 MED-3865 7 0.773869 Anserini
PLAIN-1008 Q0 MED-3865 8 0.773869 Anserini
PLAIN-1008 Q0 MED-3316 9 0.771660 Anserini
PLAIN-1008 Q0 MED-3316 10 0.771660 Anserini

I appear to be getting duplicates of docs, e.g., MED-2036. Are you somehow indexing everything twice?

Panizghi · 2024-08-25T13:56:02Z

Okay, I can now run these commands:

python src/main/python/safetensors/json_to_bin.py \
--input collections/beir-v1.0.0/bge-base-en-v1.5/nfcorpus/vectors.part00.jsonl.gz \
--output collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus --overwrite

bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-collection SafeTensorsDenseVectorCollection \
-input collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus \
-generator SafeTensorsDenseVectorDocumentGenerator \
-index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/ \
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge

After I build the index, I should be able to switch to retrieval, here: https://github.com/castorini/anserini/blob/master/docs/regressions/regressions-beir-v1.0.0-nfcorpus.bge-base-en-v1.5.hnsw.onnx.md

The retrieval command is this:

bin/run.sh io.anserini.search.SearchHnswDenseVectors \
  -index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/ \
  -topics tools/topics-and-qrels/topics.beir-v1.0.0-nfcorpus.test.tsv.gz \
  -topicReader TsvString \
  -output runs/run.beir-v1.0.0-nfcorpus.bge-base-en-v1.5.bge-hnsw-onnx.topics.beir-v1.0.0-nfcorpus.test.txt \
  -generator VectorQueryGenerator -topicField title -removeQuery -threads 16 -hits 1000 -efSearch 1000 -encoder BgeBaseEn15

But the eval command generates errors:

$ bin/trec_eval -c -m ndcg_cut.10 tools/topics-and-qrels/qrels.beir-v1.0.0-nfcorpus.test.txt runs/run.beir-v1.0.0-nfcorpus.bge-base-en-v1.5.bge-hnsw-onnx.topics.beir-v1.0.0-nfcorpus.test.txt
WARNING: Using incubator modules: jdk.incubator.vector
trec_eval.form_res_qrels: duplicate docs MED-1000trec_eval: Can't calculate measure 'ndcg_cut'

From here:

$ head runs/run.beir-v1.0.0-nfcorpus.bge-base-en-v1.5.bge-hnsw-onnx.topics.beir-v1.0.0-nfcorpus.test.txt
PLAIN-1008 Q0 MED-2036 1 0.776562 Anserini
PLAIN-1008 Q0 MED-2036 2 0.776562 Anserini
PLAIN-1008 Q0 MED-5135 3 0.775252 Anserini
PLAIN-1008 Q0 MED-5135 4 0.775252 Anserini
PLAIN-1008 Q0 MED-4694 5 0.774549 Anserini
PLAIN-1008 Q0 MED-4694 6 0.774549 Anserini
PLAIN-1008 Q0 MED-3865 7 0.773869 Anserini
PLAIN-1008 Q0 MED-3865 8 0.773869 Anserini
PLAIN-1008 Q0 MED-3316 9 0.771660 Anserini
PLAIN-1008 Q0 MED-3316 10 0.771660 Anserini

I appear to be getting duplicates of docs, e.g., MED-2036. Are you somehow indexing everything twice?

That was initially the reason I swapped to single thread and having critical section testing the fix right now

Panizghi · 2024-09-01T00:23:37Z

Updated command :

python src/main/python/safetensors/json_to_bin.py --input collections/robust04 --output collections/robust04.safetensors/ --overwrite

bin/run.sh io.anserini.index.IndexHnswDenseVectors \
-collection SafeTensorsDenseVectorCollection \
-input collections/beir-v1.0.0/bge-base-en-v1.5.safetensors/nfcorpus \
-generator SafeTensorsDenseVectorDocumentGenerator \
-index indexes/lucene-hnsw.beir-v1.0.0-nfcorpus.bge-base-en-v1.5/ \
-threads 16 -M 16 -efC 100 -memoryBuffer 65536 -noMerge

Indexing Performance:

8 threads: Indexed 337,860 documents in 00:01:25
1 thread: Indexed 337,860 documents in 00:05:50

File Sizes:

Original JSONL Files (Total: 3.6 GB):
- vectors.part00.jsonl.gz: 683 MB
- vectors.part01.jsonl.gz: 683 MB
- vectors.part02.jsonl.gz: 682 MB
- vectors.part03.jsonl.gz: 683 MB
- vectors.part04.jsonl.gz: 683 MB
- vectors.part05.jsonl.gz: 192 MB
Converted Safetensor Files (Total: 3.1 GB):
- Vector Files:
  - vectors.part00_vectors.safetensors: 586 MB
  - vectors.part01_vectors.safetensors: 586 MB
  - vectors.part02_vectors.safetensors: 586 MB
  - vectors.part03_vectors.safetensors: 586 MB
  - vectors.part04_vectors.safetensors: 586 MB
  - vectors.part05_vectors.safetensors: 165 MB
- DocID Files:
  - vectors.part00_docids.safetensors: 13 MB
  - vectors.part01_docids.safetensors: 10 MB
  - vectors.part02_docids.safetensors: 10 MB
  - vectors.part03_docids.safetensors: 13 MB
  - vectors.part04_docids.safetensors: 10 MB
  - vectors.part05_docids.safetensors: 2.8 MB

lintool · 2024-09-10T19:18:13Z

Superseded by #2582

Panizghi added 6 commits June 1, 2024 01:33

not issue

85fe2e0

update

e0a7937

test working

b486073

fix paths

b1b4ef0

fix

07f777c

delete extra files

2d84a38

Panizghi mentioned this pull request Jun 2, 2024

Anserini: replace verbose json-based vector format with more compact binary encoding castorini/ura-projects#31

Open

file ext fix

03dfdde

r4meng and others added 3 commits June 11, 2024 00:15

refactor code based on feedback

9e40ec3

python fix

7187238

hardcoded path

bbc8a20

lintool requested changes Jun 15, 2024

View reviewed changes

Panizghi added 2 commits June 16, 2024 15:53

indent fix

008864e

path python fix

23c90b8

Panizghi force-pushed the master branch from 8935d28 to 086ef9c Compare June 16, 2024 22:24

java

59165a6

Panizghi force-pushed the master branch 2 times, most recently from 62cd3c7 to ff75047 Compare July 9, 2024 05:01

Panizghi closed this Jul 9, 2024

Panizghi force-pushed the master branch from 257edd6 to 3885b5c Compare July 9, 2024 05:10

Panizghi added 4 commits July 9, 2024 00:19

Merge pull request #1 from Panizghi/Melissa

699f42e

Melissa

Update HnswJsonWithSafeTensorsDenseVectorDocumentGenerator.java

9a8bbff

Update AbstractIndexer.java

2094e24

Update json_to_bin.py

4868d00

Panizghi reopened this Jul 9, 2024

Delete src/main/java/io/anserini/index/SafeTensorsIndexCollection.java

bca9754

Panizghi added 2 commits July 15, 2024 00:53

test

92987dd

fixed collection and indexer

01b1c51

Panizghi added 2 commits July 16, 2024 23:30

update

1fcd094

added comments

897fac7

lintool reviewed Aug 23, 2024

View reviewed changes

src/main/java/io/anserini/collection/SafeTensorsDenseVectorCollection.java Show resolved Hide resolved

lintool reviewed Aug 23, 2024

View reviewed changes

...ava/io/anserini/index/generator/HnswJsonWithSafeTensorsDenseVectorDocumentGeneratorTest.java Outdated Show resolved Hide resolved

Panizghi added 2 commits August 25, 2024 00:28

fixed thread exception

7551989

remove extra files

20ab166

bar progress and high level logging

229331c

Panizghi and others added 3 commits August 25, 2024 00:57

Merge branch 'castorini:master' into master

a2ec450

update header

d50c552

fix space

381e5cd

lintool requested changes Aug 25, 2024

View reviewed changes

Panizghi added 3 commits August 26, 2024 11:21

testing refactored

2e56f19

multiple jsonl handeling

7429ed8

remove test

50272dd

Panizghi and others added 2 commits August 31, 2024 20:28

Delete src/main/python/safetensors/test.py

f17420b

update tools

bc68411

lintool closed this Sep 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HnswDensevector SafeTensor Generator #2515

HnswDensevector SafeTensor Generator #2515

Panizghi commented Jun 2, 2024 •

edited

Loading

17Melissa commented Jun 2, 2024

lintool commented Jun 2, 2024

17Melissa commented Jun 15, 2024

Panizghi commented Jul 9, 2024

Panizghi commented Jul 15, 2024

lintool commented Aug 23, 2024

lintool commented Aug 23, 2024

lintool commented Aug 23, 2024

Panizghi commented Aug 25, 2024

Panizghi commented Aug 25, 2024

Panizghi commented Aug 25, 2024

lintool left a comment

lintool commented Aug 25, 2024

Panizghi commented Aug 25, 2024

Panizghi commented Sep 1, 2024

lintool commented Sep 10, 2024

HnswDensevector SafeTensor Generator #2515

HnswDensevector SafeTensor Generator #2515

Conversation

Panizghi commented Jun 2, 2024 • edited Loading

17Melissa commented Jun 2, 2024

Setup for NFCorpus Indexing with Safetensors

Indexing Procedure

lintool commented Jun 2, 2024

17Melissa commented Jun 15, 2024

Updated Workflow for Safetensors Conversion and Indexing Process

Panizghi commented Jul 9, 2024

Updates

Updated commands

Python

Java

Panizghi commented Jul 15, 2024

lintool commented Aug 23, 2024

lintool commented Aug 23, 2024

lintool commented Aug 23, 2024

Panizghi commented Aug 25, 2024

Panizghi commented Aug 25, 2024

Panizghi commented Aug 25, 2024

lintool left a comment

Choose a reason for hiding this comment

lintool commented Aug 25, 2024

Panizghi commented Aug 25, 2024

Panizghi commented Sep 1, 2024

Indexing Performance:

File Sizes:

lintool commented Sep 10, 2024

Panizghi commented Jun 2, 2024 •

edited

Loading