Skip to content

Commit

Permalink
Merge branch 'release37'
Browse files Browse the repository at this point in the history
  • Loading branch information
nytamin committed Sep 3, 2021
2 parents 388746a + ea14840 commit cd2ebe7
Show file tree
Hide file tree
Showing 146 changed files with 6,673 additions and 2,710 deletions.
3 changes: 3 additions & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,7 @@ module.exports = {
"**/dist/**/*",
"**/__tests__/**/*",
],
"rules": {
"no-console": "warn"
}
};
79 changes: 68 additions & 11 deletions FOR_DEVELOPERS.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,20 @@

This document contains documentation intended for developers of this repo.


# Key concepts

![System overview](./images/System-overview.png "System overview")
![System overview](./images/System-overview.png 'System overview')

## Workforce

*Note: There can be only one (1) Workforce in a setup.*
_Note: There can be only one (1) Workforce in a setup._

The Workforce keeps track of which `ExpectationManagers` and `Workers` are online, and mediates the contact between the two.

_Future functionality: The Workforce is responsible for tracking the total workload and spin up/down workers accordingly._


## Package Manager

*Note: There can be multiple Package Managers in a setup*
_Note: There can be multiple Package Managers in a setup_

The Package Manager receives [Packages](#packages) from [Sofie Core](https://github.com/nrkno/tv-automation-server-core) and generates [Expectations](#expectations) from them.

Expand All @@ -35,22 +32,20 @@ A typical lifetime of an Expectation is:
4. `WORKING`: Intermediary state while the Worker is working.
5. `FULFILLED`: From time-to-time, the Expectation is re-checked if it still is `FULFILLED`


## Worker

*Note: There can be multiple Workers in a setup*
_Note: There can be multiple Workers in a setup_

The Worker is the process which actually does the work.
It exposes an API with methods for the `ExpectationManager` to call in order to check status of Packages and perform the work.

The Worker is (almost completely) **stateless**, which allows it to expose a lambda-like API. This allows for there being a pool of Workers where the workload can be easilly shared between the Workers.


_Future functionality: There are multiple different types of Workers. Some are running on a Windows-machine with direct access to that maching. Some are running in Linux/Docker containers, or even off-premise._

### ExpectationHandlers & AccessorHandlers

![Expectation and Accessor handlers](./images/handlers.png "Expectation and Accessor handlers")
![Expectation and Accessor handlers](./images/handlers.png 'Expectation and Accessor handlers')

Internally, the Worker is structured by separating the `ExpectationHandlers` and the `AccessorHandlers`.

Expand All @@ -59,7 +54,6 @@ The `ExpectationHandlers` handles the high-level functionality required for a ce
For example, when [copying a file](shared/packages/worker/src/worker/workers/windowsWorker/expectationHandlers/fileCopy.ts) from one folder to another, the `ExpectationHandler` will handle things like "check if the source package exists", "check if the target package already exists" but it's never touching the files directly, only talking to the the `AccessorHandler`.
The `AccessorHandler` [exposes a few generic methods](./shared/packages/worker/src/worker/accessorHandlers/genericHandle.ts), like "check if we can read from a Package" (ie does a file exist), etc.


## HTTP-server

The HTTP-server application is a simple HTTP-server which allows for uploading and downloading of Packages with a RESTful API.
Expand Down Expand Up @@ -94,3 +88,66 @@ A PackageContainer is separated from an **Accessor**, which is the "way to acces
_See [expectationApi.ts](shared/packages/api/src/expectationApi.ts)._

An Expectation is what the PackageManager uses to figure out what should be done and how to do it. One example is "Copy a file from a source into a target".

# Examples:

Below are some examples of the data:

## Example A, a file is to be copied

```javascript
// The input to Package Manager (from Sofie-Core):

const packageContainers = {
source0: {
label: 'Source 0',
accessors: {
local: {
type: 'local_folder',
folderPath: 'D:\\media\\source0',
allowRead: true,
},
},
},
target0: {
label: 'Target 0',
accessors: {
local: {
type: 'local_folder',
folderPath: 'D:\\media\\target0',
allowRead: true,
allowWrite: true,
},
},
},
}
const expectedPackages = [
{
type: 'media_file',
_id: 'test',
contentVersionHash: 'abc1234',
content: {
filePath: 'myFocalFolder\\amb.mp4',
},
version: {},
sources: [
{
containerId: 'source0',
accessors: {
local: {
type: 'local_folder',
filePath: 'amb.mp4',
},
},
},
],
layers: ['target0'],
sideEffect: {
previewContainerId: null,
previewPackageSettings: null,
thumbnailContainerId: null,
thumbnailPackageSettings: null,
},
},
]
```
24 changes: 13 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,24 +10,26 @@ The packages in [shared/packages](shared/packages) are helper libraries, used by

The packages in [apps/](apps/) can be run as individual applications.

The packages in [tests/](tests/) contain unit/integration tests.

### Applications

| Name | Location | Description |
| ----- | -------- | ----------- |
| **Workforce** | [apps/workforce/app](apps/workforce/app) | Mediates connections between the Workers and the Package Managers. _(Later: Will handle spin-up/down of workers according to the current need.)_ |
| Name | Location | Description |
| ------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Workforce** | [apps/workforce/app](apps/workforce/app) | Mediates connections between the Workers and the Package Managers. _(Later: Will handle spin-up/down of workers according to the current need.)_ |
| **Package Manager** | [apps/package-manager/app](apps/package-manager/app) | The Package Manager receives `expectedPackages` from a [Sofie Core](https://github.com/nrkno/tv-automation-server-core), converts them into `Expectations`. Keeps track of work statues and distributes the work to the Workers. |
| **Worker** | [apps/worker/app](apps/worker/app) | Executes work orders from the Package Manager |
| **HTTP-server** | [apps/http-server/app](apps/http-server/app) | A simple HTTP server, where files can be uploaded to and served from. (Often used for thumbnails & previews) |
| **Single-app** | [apps/single-app/app](apps/single-app/app) | Runs one of each of the above in a single application. |
| **Worker** | [apps/worker/app](apps/worker/app) | Executes work orders from the Package Manager |
| **HTTP-server** | [apps/http-server/app](apps/http-server/app) | A simple HTTP server, where files can be uploaded to and served from. (Often used for thumbnails & previews) |
| **Single-app** | [apps/single-app/app](apps/single-app/app) | Runs one of each of the above in a single application. |

### Packages (Libraries)

| Name | Location | Description |
| -- | -- | -- |
| **API** | [shared/packages/api](shared/packages/api) | Various interfaces used by the other libraries |
| Name | Location | Description |
| ---------------------- | ------------------------------------------------------------------------ | ----------------------------------------------------------------------- |
| **API** | [shared/packages/api](shared/packages/api) | Various interfaces used by the other libraries |
| **ExpectationManager** | [shared/packages/expectationManager](shared/packages/expectationManager) | The ExpectationManager class is used by the Package Manager application |
| **Worker** | [shared/packages/worker](shared/packages/worker) | The Worker class is used by the Worker application |
| **Workforce** | [shared/packages/Workforce](shared/packages/Workforce) | The Workforce class is used by the Worker application |
| **Worker** | [shared/packages/worker](shared/packages/worker) | The Worker class is used by the Worker application |
| **Workforce** | [shared/packages/Workforce](shared/packages/Workforce) | The Workforce class is used by the Worker application |

## For Developers

Expand Down
4 changes: 2 additions & 2 deletions apps/_boilerplate/app/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
"scripts": {
"build": "rimraf dist && yarn build:main",
"build:main": "tsc -p tsconfig.json",
"build-win32": "mkdir deploy & rimraf deploy/boilerplate.exe && nexe dist/index.js -t windows-x64-12.18.1 -o deploy/boilerplate.exe && node scripts/copy-natives.js win32-x64",
"build-win32": "mkdir deploy & node ../../../scripts/build-win32.js boilerplate.exe && node ../../../scripts/copy-natives.js win32-x64",
"__test": "jest",
"start": "node dist/index.js",
"precommit": "lint-staged"
Expand All @@ -23,7 +23,7 @@
},
"prettier": "@sofie-automation/code-standard-preset/.prettierrc.json",
"engines": {
"node": ">=12.3.0"
"node": ">=12.20.0"
},
"lint-staged": {
"*.{js,css,json,md,scss}": [
Expand Down
33 changes: 0 additions & 33 deletions apps/_boilerplate/app/scripts/copy-natives.js

This file was deleted.

1 change: 1 addition & 0 deletions apps/_boilerplate/app/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
import { startProcess } from '@boilerplate/generic'
/* eslint-disable no-console */

startProcess().catch(console.error)
4 changes: 2 additions & 2 deletions apps/_boilerplate/packages/generic/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
},
"prettier": "@sofie-automation/code-standard-preset/.prettierrc.json",
"engines": {
"node": ">=12.3.0"
"node": ">=12.20.0"
},
"lint-staged": {
"*.{js,css,json,md,scss}": [
Expand All @@ -29,4 +29,4 @@
"eslint"
]
}
}
}
1 change: 1 addition & 0 deletions apps/_boilerplate/packages/generic/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
// boilerplate
export async function startProcess(): Promise<void> {
// eslint-disable-next-line no-console
console.log('hello world!')
}
2 changes: 2 additions & 0 deletions apps/appcontainer-node/app/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
log.log
deploy/*
File renamed without changes.
36 changes: 36 additions & 0 deletions apps/appcontainer-node/app/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"name": "@appcontainer-node/app",
"version": "1.0.0",
"description": "AppContainer-Node.js",
"private": true,
"scripts": {
"build": "rimraf dist && yarn build:main",
"build:main": "tsc -p tsconfig.json",
"build-win32": "mkdir deploy & node ../../../scripts/build-win32.js appContainer-node.exe && node ../../../scripts/copy-natives.js win32-x64",
"__test": "jest",
"start": "node dist/index.js",
"precommit": "lint-staged"
},
"devDependencies": {
"nexe": "^3.3.7",
"lint-staged": "^7.2.0"
},
"dependencies": {
"@appcontainer-node/generic": "*"
},
"peerDependencies": {
"@sofie-automation/blueprints-integration": "*"
},
"prettier": "@sofie-automation/code-standard-preset/.prettierrc.json",
"engines": {
"node": ">=12.20.0"
},
"lint-staged": {
"*.{js,css,json,md,scss}": [
"prettier"
],
"*.{ts,tsx}": [
"eslint"
]
}
}
7 changes: 7 additions & 0 deletions apps/appcontainer-node/app/src/__tests__/test.spec.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
describe('tmp', () => {
test('tmp', () => {
// Note: To enable tests in this package, ensure that the "test" script is present in package.json
expect(1).toEqual(1)
})
})
export {}
5 changes: 5 additions & 0 deletions apps/appcontainer-node/app/src/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
import { startProcess } from '@appcontainer-node/generic'
/* eslint-disable no-console */

console.log('process started') // This is a message all Sofie processes log upon startup
startProcess().catch(console.error)
File renamed without changes.
8 changes: 8 additions & 0 deletions apps/appcontainer-node/packages/generic/jest.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
const base = require('../../../../jest.config.base')
const packageJson = require('./package')

module.exports = {
...base,
name: packageJson.name,
displayName: packageJson.name,
}
35 changes: 35 additions & 0 deletions apps/appcontainer-node/packages/generic/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
{
"name": "@appcontainer-node/generic",
"version": "1.0.0",
"private": true,
"main": "dist/index.js",
"types": "dist/index.d.ts",
"scripts": {
"build": "rimraf dist && yarn build:main",
"build:main": "tsc -p tsconfig.json",
"__test": "jest",
"precommit": "lint-staged"
},
"peerDependencies": {
"@sofie-automation/blueprints-integration": "*"
},
"dependencies": {
"@shared/api": "*",
"@shared/worker": "*"
},
"devDependencies": {
"lint-staged": "^7.2.0"
},
"prettier": "@sofie-automation/code-standard-preset/.prettierrc.json",
"engines": {
"node": ">=12.20.0"
},
"lint-staged": {
"*.{js,css,json,md,scss}": [
"prettier"
],
"*.{ts,tsx}": [
"eslint"
]
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
describe('Temp', () => {
// This is just a placeholder, to be replaced with real tests later on
test('basic math', () => {
expect(1 + 1).toEqual(2)
})
})
Loading

0 comments on commit cd2ebe7

Please sign in to comment.