Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add /rerank and /import sections to README #82

Merged
merged 4 commits into from
Oct 22, 2024
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 132 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -579,6 +579,74 @@ func main() {
}
```

### Import vectors from object storage

You can now [import vectors en masse](https://docs.pinecone.io/guides/data/understanding-imports) from object
storage. `Import` is a long-running, asynchronous operation that imports large numbers of records into a Pinecone
serverless index.

In order to import vectors from object storage, they must be stored in Parquet files and adhere to the necessary
[file format](https://docs.pinecone.io/guides/data/understanding-imports#parquet-file-format). Your object storage
must also adhere to the necessary [directory structure](https://docs.pinecone.io/guides/data/understanding-imports#directory-structure).

The following example imports vectors from an Amazon S3 bucket into a Pinecone serverless index:

```go
ctx := context.Background()

clientParams := pinecone.NewClientParams{
ApiKey: os.Getenv("PINECONE_API_KEY"),
}

pc, err := pinecone.NewClient(clientParams)

if err != nil {
log.Fatalf("Failed to create Client: %v", err)
} else {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Let's drop the else here, I'd like to cut down on some of the verbosity in our Go documentation and stuff like this isn't really necessary.

I don't think we need the success messages for any of the errors below either.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okie doke

fmt.Println("Successfully created a new Client object!")
}

indexName := "sample-index"

idx, err := pc.CreateServerlessIndex(ctx, &pinecone.CreateServerlessIndexRequest{
Name: indexName,
Dimension: 3,
Metric: pinecone.Cosine,
Cloud: pinecone.Aws,
Region: "us-east-1",
})

if err != nil {
log.Fatalf("Failed to create serverless index: %v", err)
} else {
fmt.Printf("Successfully created serverless index: %s", idx.Name)
}

idx, err = pc.DescribeIndex(ctx, "pinecone-index")

if err != nil {
log.Fatalf("Failed to describe index \"%v\": %v", idx.Name, err)
}

idxConnection, err := pc.Index(pinecone.NewIndexConnParams{Host: idx.Host})
if err != nil {
log.Fatalf("Failed to create IndexConnection for Host: %v: %v", idx.Host, err)
}

storageURI := "s3://my-bucket/my-directory/"

errorMode := "abort" // Will abort if error encountered; other option: "continue"

importRes, err := idxConnection.StartImport(ctx, storageURI, nil, (*pinecone.ImportErrorMode)(&errorMode))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The need to cast this errorMode pointer feels messy. I think updating the argument type so it's a bit easier to work with would be good.

You don't need to do anything here, and this is fine and accurate for now. Just made me realize - I added a ticket.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good!


if err != nil {
log.Fatalf("Failed to start import: %v", err)
}

fmt.Printf("import started with ID: %s", importRes.Id)
```
You can [start, cancel, and check the status](https://docs.pinecone.io/guides/data/import-data) of all or one import operation(s).

### Query an index

#### Query by vector values
Expand Down Expand Up @@ -1307,13 +1375,17 @@ func main() {

## Inference

The `Client` object has an `Inference` namespace which allows interacting with Pinecone's [Inference API](https://docs.pinecone.io/reference/api/2024-07/inference/generate-embeddings). The Inference API is a service that gives you access to embedding models hosted on Pinecone's infrastructure. Read more at [Understanding Pinecone Inference](https://docs.pinecone.io/guides/inference/understanding-inference).
The `Client` object has an `Inference` namespace which allows interacting with
Pinecone's [Inference API](https://docs.pinecone.io/reference/api/2024-07/inference/generate-embeddings). The Inference
API is a service that gives you access to embedding models hosted on Pinecone's infrastructure. Read more
at [Understanding Pinecone Inference](https://docs.pinecone.io/guides/inference/understanding-inference).

**Notes:**

Models currently supported:

- [multilingual-e5-large](https://docs.pinecone.io/guides/inference/understanding-inference#embedding-models)
- Embedding: [multilingual-e5-large](https://docs.pinecone.io/guides/inference/understanding-inference#embedding-models)
- Reranking: [bge-reranker-v2-m3](https://docs.pinecone.io/models/bge-reranker-v2-m3)

### Create Embeddings

Expand Down Expand Up @@ -1368,11 +1440,67 @@ Send text to Pinecone's inference API to generate embeddings for documents and q
}
fmt.Printf("query embedding response: %+v", queryEmbeddingsResponse)

// << Send query to Pinecone to retrieve similar documents >>
// << Send query to Pinecone to retrieve similar documents >>
```

### Rerank documents

Rerank documents in descending relevance-order against a query.

**Note:** The `score` represents the absolute measure of relevance of a given query and passage pair. Normalized
between [0, 1], the `score` represents how closely relevant a specific item and query are, with scores closer to 1
indicating higher relevance.

```go
ctx := context.Background()

pc, err := pinecone.NewClient(pinecone.NewClientParams{
ApiKey: "YOUR-API-KEY"
})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Indentation may be off here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good on my screen; it's aligned with pc. What are you seeing?
Screenshot 2024-10-21 at 4 48 22 PM

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's just gh formatting, nevermind.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay cool!


if err != nil {
log.Fatalf("Failed to create Client: %v", err)
}

rerankModel := "bge-reranker-v2-m3"
query := "What are some good Turkey dishes for Thanksgiving?"

documents := []pinecone.Document{
{"title": "Turkey Sandwiches", "body": "Turkey is a classic meat to eat at American Thanksgiving."},
{"title": "Lemon Turkey", "body": "A lemon brined Turkey with apple sausage stuffing is a classic Thanksgiving main course."},
{"title": "Thanksgiving", "body": "My favorite Thanksgiving dish is pumpkin pie"},
{"title": "Protein Sources", "body": "Turkey is a great source of protein."},
}

// Optional arguments
topN := 3
returnDocuments := false
rankFields := []string{"body"}
modelParams := map[string]string{
"truncate": "END",
}

rerankRequest := pinecone.RerankRequest{
Model: rerankModel,
Query: query,
Documents: documents,
TopN: &topN,
ReturnDocuments: &returnDocuments,
RankFields: &rankFields,
Parameters: &modelParams,
}

rerankResponse, err := pc.Inference.Rerank(ctx, &rerankRequest)

if err != nil {
log.Fatalf("Failed to rerank documents: %v", err)
}

fmt.Printf("rerank response: %+v", rerankResponse)
```

## Support

To get help using go-pinecone you can file an issue on [GitHub](https://github.com/pinecone-io/go-pinecone/issues), visit the [community forum](https://community.pinecone.io/),
To get help using go-pinecone you can file an issue on [GitHub](https://github.com/pinecone-io/go-pinecone/issues),
visit the [community forum](https://community.pinecone.io/),
or reach out to [email protected].
Loading