Skip to content

Commit

Permalink
Add docs via github pages (#79)
Browse files Browse the repository at this point in the history
  • Loading branch information
joecorall authored Jan 31, 2025
1 parent a7bc9b4 commit 8ae41d9
Show file tree
Hide file tree
Showing 9 changed files with 226 additions and 129 deletions.
1 change: 1 addition & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,3 +5,4 @@ scyllaridae
gha-creds-*.json
ci
fixtures
docs
44 changes: 44 additions & 0 deletions .github/workflows/deploy-docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
name: Deploy Docs

on:
push:
branches:
- 'main'

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: "pages"
cancel-in-progress: false

jobs:
build:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4

- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5
with:
python-version: '3.12'
cache: 'pip'

- run: pip install -r requirements.txt
working-directory: docs

- run: mkdocs build
working-directory: docs

- name: Setup Pages
uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5

- name: Upload artifact
uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
with:
path: './docs/site'

- name: Deploy to GitHub Pages
id: deployment
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4
2 changes: 2 additions & 0 deletions .github/workflows/lint-test-build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ on:
paths-ignore:
- '**/*.md'
- 'ci/**'
- 'docs/**'
- '.github/workflows/deploy-docs.yml'
- '.github/workflows/delete.yml'
- 'renovate.json5'
tags:
Expand Down
130 changes: 1 addition & 129 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,135 +2,7 @@

Any command that takes a file as input and prints a result as output can use scyllaridae.


## Basic overview

This service reads a file, and pipes the file's stream as `stdin` to a command. The `stdout` from that command is then returned as an HTTP response.

Both `GET` and `POST` requests are supported by any scyllaridea service.

`GET` supports Islandora's alpaca/event spec, which sends the URL of a file as an HTTP header `Apix-Ldp-Resource` and prints the result. e.g. to create a VTT file from an audio file:

```
$ curl -H "Apix-Ldp-Resource: https://github.com/ggerganov/whisper.cpp/raw/master/samples/jfk.wav" http://localhost:8080
WEBVTT
00:00:00.000 --> 00:00:11.000
And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
```

`POST` supports directly uploading a file to the service, being sure to include the mimetype of the file in the `Content-Type` HTTP header

```
$ curl \
-H "Content-Type: audio/x-wav" \
--data-binary "@output.wav" \
http://localhost:8080/
WEBVTT
00:00:00.000 --> 00:00:02.960
Lehigh University Library Technology.
```

You can see several example implementations using this framework in [examples](./examples). Some examples send the file's contents directly to `stdin` if the command supports reading from that stream e.g. [fits](./examples/fits/scyllaridae.yml). For other commands that do not support reading directly from stdin, and instead requiring specifying a file path on disk, a bash script is implemented to act as a wrapper around the command. e.g. [libreoffice](./examples/libreoffice/cmd.sh)

## Adding a new microservice

### Create the microservice

Define the service's behavior in `scyllaridae.yml`.

You can specify which mimetypes the service can act on in `allowedMimeTypes`

And specify different commands for different mimetypes in `cmdByMimeType`, or set the default command to run for all mimetypes with the `default` key.

```yaml
allowedMimeTypes:
- "*"
cmdByMimeType:
default:
cmd: "curl"
args:
- "-X"
- "POST"
# read the source media file in from stdin
- "-F"
- "datafile=@-"
# send the media file to FITS which will return the XML to stdout
- "http://fits:8080/fits/examine"
```
<sup><sub>Here's another more [complex example YML](./scyllaridae.complex.yml).</sub></sup>
Define the `Dockerfile` to run your microservice. Your service will run the main `scyllaridae` program which is an http service configured by your `scyllaridae.yml`. You just need to install the binaries your yml specifies to ensure the command is in the container when it runs.

```dockerfile
FROM lehighlts/scyllaridae:main
RUN apk update && \
apk add --no-cache curl==8.5.0-r0
COPY scyllaridae.yml /app/scyllaridae.yml
```


### Deploy your new microservice

Update your [ISLE docker-compose.yml](https://github.com/Islandora-Devops/isle-site-template/blob/main/docker-compose.yml) to deploy the service's docker image defined above

```yaml
fits-dev: &fits
<<: [*dev, *common]
image: ${DOCKER_REPOSITORY}/scyllaridae-fits:main
networks:
default:
aliases:
- fits
fits-prod: &fits-prod
<<: [*prod, *fits]
```

#### Kubernetes deployment

The demo kubernetes manifests in this repo uses the `ingress-nginx` controller to route requests from a single domain to the proper microservice.

##### Create a the TLS secret

```
kubectl create secret tls CHANGE-ME --key key.pem --cert cert.pem
```
#### Apply the kubernetes manifests
Now you can apply your kube manifests, being sure to replace `__DOMAIN__` and `__DOCKER_REPOSITORY__` with the proper valuyes
```
DOMAIN=CHANGE-ME.bar.com
DOCKER_REPOSITORY=lehighlts
KUBE_TLS_SECRET=CHANGE-ME
sed -e "s|__DOMAIN__|$DOMAIN|" \
-e "s|__DOCKER_REPOSITORY__|$DOCKER_REPOSITORY|" \
-e "s|__KUBE_TLS_SECRET__|$KUBE_TLS_SECRET|" \
*.yaml \
| kubectl apply -f -
```
### Configure alpaca and Drupal
Until we define a subscription spec for Islandora Events in this repo, you'll also need to:
3. Update alpaca.properties
```
derivative.QUEUE-NAME.enabled=true
derivative.QUEUE-NAME.in.stream=queue:islandora-connector-QUEUE-NAME
derivative.QUEUE-NAME.service.url=http://QUEUE-NAME:8080/
derivative.QUEUE-NAME.concurrent-consumers=-1
derivative.QUEUE-NAME.max-concurrent-consumers=-1
derivative.QUEUE-NAME.async-consumer=true
```
4. Add an action in Islandora with your `islandora-connector-QUEUE-NAME` specified
5. Add a context to trigger the action you created in step 4
Documentation is available at https://lehigh-university-libraries.github.io/scyllaridae/

## Attribution

Expand Down
16 changes: 16 additions & 0 deletions docs/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
FROM python:3.13-alpine@sha256:f9d772b2b40910ee8de2ac2b15ff740b5f26b37fc811f6ada28fce71a2542b0e

RUN mkdir -p /docs

WORKDIR /docs

COPY requirements.txt .
RUN apk add --no-cache cairo \
&& pip install -r requirements.txt

COPY . .
RUN mkdocs build

# serve the docs via nginx
FROM nginx:1.27@sha256:0a399eb16751829e1af26fea27b20c3ec28d7ab1fb72182879dcae1cca21206a
COPY --from=0 /docs/site /usr/share/nginx/html
12 changes: 12 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# scyllaridae documentation

Viewable at https://lehigh-university-libraries.github.io/scyllaridae/

## Local docs development

```
docker build -t docs:main .
docker run -p 8080:80 docs:main
```

You should be able to view the docs in your web browser at http://localhost:8080
133 changes: 133 additions & 0 deletions docs/docs/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,133 @@
# scyllaridae

Any command that takes a file as input and prints a result as output can use scyllaridae.


## Basic overview

This service reads a file, and pipes the file's stream as `stdin` to a command. The `stdout` from that command is then returned as an HTTP response.

Both `GET` and `POST` requests are supported by any scyllaridea service.

`GET` supports Islandora's alpaca/event spec, which sends the URL of a file as an HTTP header `Apix-Ldp-Resource` and prints the result. e.g. to create a VTT file from an audio file:

```
$ curl -H "Apix-Ldp-Resource: https://github.com/ggerganov/whisper.cpp/raw/master/samples/jfk.wav" http://localhost:8080
WEBVTT
00:00:00.000 --> 00:00:11.000
And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country.
```

`POST` supports directly uploading a file to the service, being sure to include the mimetype of the file in the `Content-Type` HTTP header

```
$ curl \
-H "Content-Type: audio/x-wav" \
--data-binary "@output.wav" \
http://localhost:8080/
WEBVTT
00:00:00.000 --> 00:00:02.960
Lehigh University Library Technology.
```

You can see several example implementations using this framework in [examples](./examples). Some examples send the file's contents directly to `stdin` if the command supports reading from that stream e.g. [fits](./examples/fits/scyllaridae.yml). For other commands that do not support reading directly from stdin, and instead requiring specifying a file path on disk, a bash script is implemented to act as a wrapper around the command. e.g. [libreoffice](./examples/libreoffice/cmd.sh)

## Adding a new microservice

### Create the microservice

Define the service's behavior in `scyllaridae.yml`.

You can specify which mimetypes the service can act on in `allowedMimeTypes`

And specify different commands for different mimetypes in `cmdByMimeType`, or set the default command to run for all mimetypes with the `default` key.

```yaml
allowedMimeTypes:
- "*"
cmdByMimeType:
default:
cmd: "curl"
args:
- "-X"
- "POST"
# read the source media file in from stdin
- "-F"
- "datafile=@-"
# send the media file to FITS which will return the XML to stdout
- "http://fits:8080/fits/examine"
```
<sup><sub>Here's another more [complex example YML](./scyllaridae.complex.yml).</sub></sup>
Define the `Dockerfile` to run your microservice. Your service will run the main `scyllaridae` program which is an http service configured by your `scyllaridae.yml`. You just need to install the binaries your yml specifies to ensure the command is in the container when it runs.

```dockerfile
FROM lehighlts/scyllaridae:main
RUN apk update && \
apk add --no-cache curl==8.5.0-r0
COPY scyllaridae.yml /app/scyllaridae.yml
```


### Deploy your new microservice

Update your [ISLE docker-compose.yml](https://github.com/Islandora-Devops/isle-site-template/blob/main/docker-compose.yml) to deploy the service's docker image defined above

```yaml
fits-dev: &fits
<<: [*dev, *common]
image: ${DOCKER_REPOSITORY}/scyllaridae-fits:main
networks:
default:
aliases:
- fits
fits-prod: &fits-prod
<<: [*prod, *fits]
```

#### Kubernetes deployment

The demo kubernetes manifests in this repo uses the `ingress-nginx` controller to route requests from a single domain to the proper microservice.

##### Create a the TLS secret

```
kubectl create secret tls CHANGE-ME --key key.pem --cert cert.pem
```
#### Apply the kubernetes manifests
Now you can apply your kube manifests, being sure to replace `__DOMAIN__` and `__DOCKER_REPOSITORY__` with the proper valuyes
```
DOMAIN=CHANGE-ME.bar.com
DOCKER_REPOSITORY=lehighlts
KUBE_TLS_SECRET=CHANGE-ME
sed -e "s|__DOMAIN__|$DOMAIN|" \
-e "s|__DOCKER_REPOSITORY__|$DOCKER_REPOSITORY|" \
-e "s|__KUBE_TLS_SECRET__|$KUBE_TLS_SECRET|" \
*.yaml \
| kubectl apply -f -
```
### Configure alpaca and Drupal
Until we define a subscription spec for Islandora Events in this repo, you'll also need to:
3. Update alpaca.properties
```
derivative.QUEUE-NAME.enabled=true
derivative.QUEUE-NAME.in.stream=queue:islandora-connector-QUEUE-NAME
derivative.QUEUE-NAME.service.url=http://QUEUE-NAME:8080/
derivative.QUEUE-NAME.concurrent-consumers=-1
derivative.QUEUE-NAME.max-concurrent-consumers=-1
derivative.QUEUE-NAME.async-consumer=true
```
4. Add an action in Islandora with your `islandora-connector-QUEUE-NAME` specified
5. Add a context to trigger the action you created in step 4
14 changes: 14 additions & 0 deletions docs/mkdocs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
site_name: scyllaridae
site_url: https://lehigh-university-libraries.github.io/scyllaridae/

repo_url: https://github.com/lehigh-university-libraries/scyllaridae
edit_uri: 'edit/main/docs/'

theme: material
plugins:
- search
- social
- tags

nav:
- Home: index.md
3 changes: 3 additions & 0 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
mkdocs==1.6.1
mkdocs-material==9.6.1
mkdocs-material[imaging]==9.6.1

0 comments on commit 8ae41d9

Please sign in to comment.