-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
9 changed files
with
226 additions
and
129 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,3 +5,4 @@ scyllaridae | |
gha-creds-*.json | ||
ci | ||
fixtures | ||
docs |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,44 @@ | ||
name: Deploy Docs | ||
|
||
on: | ||
push: | ||
branches: | ||
- 'main' | ||
|
||
permissions: | ||
contents: read | ||
pages: write | ||
id-token: write | ||
|
||
concurrency: | ||
group: "pages" | ||
cancel-in-progress: false | ||
|
||
jobs: | ||
build: | ||
runs-on: ubuntu-24.04 | ||
steps: | ||
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4 | ||
|
||
- uses: actions/setup-python@0b93645e9fea7318ecaed2b359559ac225c90a2b # v5 | ||
with: | ||
python-version: '3.12' | ||
cache: 'pip' | ||
|
||
- run: pip install -r requirements.txt | ||
working-directory: docs | ||
|
||
- run: mkdocs build | ||
working-directory: docs | ||
|
||
- name: Setup Pages | ||
uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5 | ||
|
||
- name: Upload artifact | ||
uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3 | ||
with: | ||
path: './docs/site' | ||
|
||
- name: Deploy to GitHub Pages | ||
id: deployment | ||
uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,16 @@ | ||
FROM python:3.13-alpine@sha256:f9d772b2b40910ee8de2ac2b15ff740b5f26b37fc811f6ada28fce71a2542b0e | ||
|
||
RUN mkdir -p /docs | ||
|
||
WORKDIR /docs | ||
|
||
COPY requirements.txt . | ||
RUN apk add --no-cache cairo \ | ||
&& pip install -r requirements.txt | ||
|
||
COPY . . | ||
RUN mkdocs build | ||
|
||
# serve the docs via nginx | ||
FROM nginx:1.27@sha256:0a399eb16751829e1af26fea27b20c3ec28d7ab1fb72182879dcae1cca21206a | ||
COPY --from=0 /docs/site /usr/share/nginx/html |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
# scyllaridae documentation | ||
|
||
Viewable at https://lehigh-university-libraries.github.io/scyllaridae/ | ||
|
||
## Local docs development | ||
|
||
``` | ||
docker build -t docs:main . | ||
docker run -p 8080:80 docs:main | ||
``` | ||
|
||
You should be able to view the docs in your web browser at http://localhost:8080 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,133 @@ | ||
# scyllaridae | ||
|
||
Any command that takes a file as input and prints a result as output can use scyllaridae. | ||
|
||
|
||
## Basic overview | ||
|
||
This service reads a file, and pipes the file's stream as `stdin` to a command. The `stdout` from that command is then returned as an HTTP response. | ||
|
||
Both `GET` and `POST` requests are supported by any scyllaridea service. | ||
|
||
`GET` supports Islandora's alpaca/event spec, which sends the URL of a file as an HTTP header `Apix-Ldp-Resource` and prints the result. e.g. to create a VTT file from an audio file: | ||
|
||
``` | ||
$ curl -H "Apix-Ldp-Resource: https://github.com/ggerganov/whisper.cpp/raw/master/samples/jfk.wav" http://localhost:8080 | ||
WEBVTT | ||
00:00:00.000 --> 00:00:11.000 | ||
And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country. | ||
``` | ||
|
||
`POST` supports directly uploading a file to the service, being sure to include the mimetype of the file in the `Content-Type` HTTP header | ||
|
||
``` | ||
$ curl \ | ||
-H "Content-Type: audio/x-wav" \ | ||
--data-binary "@output.wav" \ | ||
http://localhost:8080/ | ||
WEBVTT | ||
00:00:00.000 --> 00:00:02.960 | ||
Lehigh University Library Technology. | ||
``` | ||
|
||
You can see several example implementations using this framework in [examples](./examples). Some examples send the file's contents directly to `stdin` if the command supports reading from that stream e.g. [fits](./examples/fits/scyllaridae.yml). For other commands that do not support reading directly from stdin, and instead requiring specifying a file path on disk, a bash script is implemented to act as a wrapper around the command. e.g. [libreoffice](./examples/libreoffice/cmd.sh) | ||
|
||
## Adding a new microservice | ||
|
||
### Create the microservice | ||
|
||
Define the service's behavior in `scyllaridae.yml`. | ||
|
||
You can specify which mimetypes the service can act on in `allowedMimeTypes` | ||
|
||
And specify different commands for different mimetypes in `cmdByMimeType`, or set the default command to run for all mimetypes with the `default` key. | ||
|
||
```yaml | ||
allowedMimeTypes: | ||
- "*" | ||
cmdByMimeType: | ||
default: | ||
cmd: "curl" | ||
args: | ||
- "-X" | ||
- "POST" | ||
# read the source media file in from stdin | ||
- "-F" | ||
- "datafile=@-" | ||
# send the media file to FITS which will return the XML to stdout | ||
- "http://fits:8080/fits/examine" | ||
``` | ||
<sup><sub>Here's another more [complex example YML](./scyllaridae.complex.yml).</sub></sup> | ||
Define the `Dockerfile` to run your microservice. Your service will run the main `scyllaridae` program which is an http service configured by your `scyllaridae.yml`. You just need to install the binaries your yml specifies to ensure the command is in the container when it runs. | ||
|
||
```dockerfile | ||
FROM lehighlts/scyllaridae:main | ||
RUN apk update && \ | ||
apk add --no-cache curl==8.5.0-r0 | ||
COPY scyllaridae.yml /app/scyllaridae.yml | ||
``` | ||
|
||
|
||
### Deploy your new microservice | ||
|
||
Update your [ISLE docker-compose.yml](https://github.com/Islandora-Devops/isle-site-template/blob/main/docker-compose.yml) to deploy the service's docker image defined above | ||
|
||
```yaml | ||
fits-dev: &fits | ||
<<: [*dev, *common] | ||
image: ${DOCKER_REPOSITORY}/scyllaridae-fits:main | ||
networks: | ||
default: | ||
aliases: | ||
- fits | ||
fits-prod: &fits-prod | ||
<<: [*prod, *fits] | ||
``` | ||
|
||
#### Kubernetes deployment | ||
|
||
The demo kubernetes manifests in this repo uses the `ingress-nginx` controller to route requests from a single domain to the proper microservice. | ||
|
||
##### Create a the TLS secret | ||
|
||
``` | ||
kubectl create secret tls CHANGE-ME --key key.pem --cert cert.pem | ||
``` | ||
#### Apply the kubernetes manifests | ||
Now you can apply your kube manifests, being sure to replace `__DOMAIN__` and `__DOCKER_REPOSITORY__` with the proper valuyes | ||
``` | ||
DOMAIN=CHANGE-ME.bar.com | ||
DOCKER_REPOSITORY=lehighlts | ||
KUBE_TLS_SECRET=CHANGE-ME | ||
sed -e "s|__DOMAIN__|$DOMAIN|" \ | ||
-e "s|__DOCKER_REPOSITORY__|$DOCKER_REPOSITORY|" \ | ||
-e "s|__KUBE_TLS_SECRET__|$KUBE_TLS_SECRET|" \ | ||
*.yaml \ | ||
| kubectl apply -f - | ||
``` | ||
### Configure alpaca and Drupal | ||
Until we define a subscription spec for Islandora Events in this repo, you'll also need to: | ||
3. Update alpaca.properties | ||
``` | ||
derivative.QUEUE-NAME.enabled=true | ||
derivative.QUEUE-NAME.in.stream=queue:islandora-connector-QUEUE-NAME | ||
derivative.QUEUE-NAME.service.url=http://QUEUE-NAME:8080/ | ||
derivative.QUEUE-NAME.concurrent-consumers=-1 | ||
derivative.QUEUE-NAME.max-concurrent-consumers=-1 | ||
derivative.QUEUE-NAME.async-consumer=true | ||
``` | ||
4. Add an action in Islandora with your `islandora-connector-QUEUE-NAME` specified | ||
5. Add a context to trigger the action you created in step 4 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
site_name: scyllaridae | ||
site_url: https://lehigh-university-libraries.github.io/scyllaridae/ | ||
|
||
repo_url: https://github.com/lehigh-university-libraries/scyllaridae | ||
edit_uri: 'edit/main/docs/' | ||
|
||
theme: material | ||
plugins: | ||
- search | ||
- social | ||
- tags | ||
|
||
nav: | ||
- Home: index.md |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
mkdocs==1.6.1 | ||
mkdocs-material==9.6.1 | ||
mkdocs-material[imaging]==9.6.1 |