Skip to content

Commit

Permalink
Merge pull request #22 from deepset-ai/update-import-instructions
Browse files Browse the repository at this point in the history
Docs: Update instructions for uploading components through a release
agnieszka-m authored Dec 12, 2024
2 parents 9248711 + 1522474 commit 57223dc
Showing 1 changed file with 43 additions and 21 deletions.
64 changes: 43 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Deepset Cloud Custom Component Template
# deepset Cloud Custom Component Template

This repository contains a template for creating custom components for your deepset Cloud pipelines. Components are Python code snippets that perform specific tasks within your pipeline. This template will guide you through all the necessary elements your custom component must include.
This template contains two sample components which are ready to be used:
@@ -21,7 +21,7 @@ See also our tutorial for [creating a custom RegexBooster component](https://doc
- Python v3.10 or v3.11
- `hatch` package manager

### Hatch: A Python Package Manager
### Hatch: A Python package manager

We use `hatch` to manage our Python packages. Install it with pip:

@@ -53,9 +53,11 @@ For more information on `hatch`, please refer to the [official Hatch documentati
| `/src/dc_custom_component/__about__.py` | Your custom components' version. deepset Cloud always uses the latest version. Bump the version every time you update your component before uploading it to deepset Cloud. |
| `/pyproject.toml` | Information about the project. If needed, add your components' dependencies in this file in the `dependencies` section. |

Note that the location of your custom component implementation defines your component's `type` to be used in pipeline YAML. For example, the sample components have the following types because of their location:
The directory where your custom component is stored determines the name of the component group in Pipeline Builder. For example, the `CharacterSplitter` component would appear in the `Preprocessors` group, while the `KeywordBooster` component would be listed in the `Rankers` group. You can drag these components onto the canvas to use them.

When working with YAML, the location of your custom component implementation defines your component's `type`. For example, the sample components have the following types because of their location:
- `dc_custom_component.example_components.preprocessors.character_splitter.CharacterSplitter`
- `dc_custom_component.example_components.rankers.keyword_booster.KeyWordBooster`
- `dc_custom_component.example_components.rankers.keyword_booster.KeywordBooster`

Here is how you would add them to a pipeline:
```yaml
@@ -66,6 +68,14 @@ components:
...

```
### Working on your component

1. Fork this repository.
2. Navigate to the `/src/dc_custom_component/components/` folder.
3. Add your custom components following the examples.
4. Update the components' version in `/src/__about__.py`.
5. Format your code using the `hatch run code-quality:all` command. (Note that hatch commands work from the project root directory only.)

### Formatting
We defined a suite of formatting tools. To format your code, run:

@@ -82,29 +92,41 @@ It's crucial to thoroughly test your custom component before uploading it to dee

## 3. Uploading your custom component

1. Fork this repository.
2. Navigate to the `/src/dc_custom_component/components/` folder.
3. Add your custom components following the examples.
4. Update the components' version in `/src/__about__.py`.
5. Format your code using the `hatch run code-quality:all` command. (Note that hatch commands work from the project root directory only.)
6. Set your [deepset Cloud API key](https://docs.cloud.deepset.ai/docs/generate-api-key).
You can upload in one of two ways:
- By releasing your forked directory.
- By zipping the forked repository and uploading it with commands.

### Uploading by releasing your forked repository

We use GitHub Actions to build and push custom components to deepset Cloud. The action runs the tests and code quality checks before pushing the component code to deepset Cloud. Create a tag to trigger the build and the push job. This method helps you keep track of the changes and investigate the code deployed to deepset Cloud.

After forking or cloning this repository:

1. Push all your changes to the forked repository.
2. Add the `DEEPSET_CLOUD_API_KEY` [secret to your repository](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions). This is your deepset Cloud API key.
(To add a secret, go to your repository and choose _Settings > Secrets and variables > Actions > New repository secret_.)
3. Enable workflows for your repository by going to _Actions > Enable workflows_.
4. (Optional) Adjust the workflow file in `.github/workflows/publish_on_tag.yaml` as needed.
5. Create a new release with a tag to trigger the GitHub Actions workflow. The workflow builds and pushes the custom component to deepset Cloud with the tag as version. For help, see [GitHub documentation](https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository).

> **Warning:** When using this GitHub Actions workflow, the version specified in the `__about__` file will be overwritten by the tag value. Make sure your tag matches the desired version number.
You can check the upload status in the `Actions` tab of your forked repository.

### Uploading a zipped repository with commands

In this method, you run commands to zip and push the repository to deepset Cloud.

1. Set your [deepset Cloud API key](https://docs.cloud.deepset.ai/docs/generate-api-key).
- On Linux and macOS: `export API_KEY=<TOKEN>`
- On Windows: `set API_KEY=<TOKEN>`
7. Upload your project by running the following command from inside of this project:
2. Upload your project by running the following command from inside of this project:
- On Linux and macOS: `hatch run dc:build-and-push`
- On Windows: `hatch run dc:build-windows` and `hatch run dc:push-windows`
This creates a zip file called `custom_component.zip` in the `dist` directory and uploads it to deepset Cloud.
This creates a ZIP file called `custom_component.zip` in the `dist` directory and uploads it to deepset Cloud.


For detailed instructions, refer to our documentation on [Creating a Custom Component](https://docs.cloud.deepset.ai/docs/create-a-custom-component).

### GitHub Actions

We use GitHub Actions to build and push custom components to deepset Cloud. Create a tag to trigger the build and push job.
After forking or cloning this repository:

1. Add the `DEEPSET_CLOUD_API_KEY` [secret to your repository](https://docs.github.com/en/actions/security-for-github-actions/security-guides/using-secrets-in-github-actions). This is your deepset Cloud API key.
(To add a secret, go to your repository and choose _Settings > Secrets and variables > Actions > New repository secret_.)
2. (Optional) Adjust the workflow file in `.github/workflows/publish_on_tag.yaml` as needed.
3. Create a tag to trigger the GitHub Actions workflow. The workflow builds and pushes the custom component to deepset Cloud with the tag as version.

> **Warning:** When using this GitHub Actions workflow, the version specified in the `__about__` file will be overwritten by the tag value. Make sure your tag matches the desired version number.

0 comments on commit 57223dc

Please sign in to comment.