From 141f001226cfff3ea73f549f95d33dda62730fcc Mon Sep 17 00:00:00 2001 From: Nicolas Thouvenin Date: Thu, 4 Jul 2024 18:06:17 +0200 Subject: [PATCH] =?UTF-8?q?docs:=20=E2=9C=8F=EF=B8=8F=20update?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/plugin-basics.md | 33 ++++++++++++++++++++++ docs/plugin-conditor.md | 34 +++++++++++++++++++++++ docs/plugin-core.md | 55 +++++++++++++++++++++++++------------ packages/basics/README.md | 33 ++++++++++++++++++++++ packages/conditor/README.md | 34 +++++++++++++++++++++++ packages/core/README.md | 55 +++++++++++++++++++++++++------------ 6 files changed, 208 insertions(+), 36 deletions(-) diff --git a/docs/plugin-basics.md b/docs/plugin-basics.md index bcebd642..29709450 100644 --- a/docs/plugin-basics.md +++ b/docs/plugin-basics.md @@ -20,6 +20,7 @@ npm install @ezs/basics - [CSVParse](#csvparse) - [CSVString](#csvstring) - [FILELoad](#fileload) +- [FILEMerge](#filemerge) - [FILESave](#filesave) - [INIString](#inistring) - [JSONParse](#jsonparse) @@ -251,6 +252,37 @@ Output: Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** +### FILEMerge + +Take `Object` or `Buffer` and throw only one document + +```json +[ fi1e1.csv, file2.csv ] +``` + +Script: + +```ini +[use] +plugin = basics + +[FILELoad] +[FILEMerge] +[replace] +path = contentOfFile1AndFile2 +value = self() +``` + +Output: + +```json +[ +(...) +] +``` + +Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** + ### FILESave Take data, convert it to buffer and append it to file @@ -591,6 +623,7 @@ It returns to the output stream - `path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** Regex to select the files to extract (optional, default `"**\/*.json"`) - `json` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** Parse as JSON the content of each file (optional, default `true`) +- `text` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)** The content of each file is converted to a string (otherwise it remains a buffer) (optional, default `true`) - `compress` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)** Enable gzip compression (optional, default `false`) Returns **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)<{id: [String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String), value: [String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)}>** diff --git a/docs/plugin-conditor.md b/docs/plugin-conditor.md index 6b1775bb..7d39c505 100644 --- a/docs/plugin-conditor.md +++ b/docs/plugin-conditor.md @@ -56,6 +56,7 @@ Sachant qu'on appauvrit (casse, accents, tiret, apostrophe) tous les champs. - [compareRnsr](#comparernsr) - [conditorScroll](#conditorscroll) - [CORHALFetch](#corhalfetch) +- [CORHALFetch](#corhalfetch-1) - [getRnsr](#getrnsr) - [getRnsrInfo](#getrnsrinfo) - [WOSFetch](#wosfetch) @@ -226,6 +227,39 @@ Output: Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** +### CORHALFetch + +Take `String` as URL, throw each chunk from the result + +Input: + +```json +[ + { q: "toto" }, +] +``` + +Script: + +```ini +[CORHALFetch] +url = https://corhal-api.inist.fr +``` + +Output: + +```json +[{...}, {"a": "b"}, {"a": "c" }] +``` + +#### Parameters + +- `url` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** corhal api url +- `timeout` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** Timeout in milliseconds (optional, default `1000`) +- `retries` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** The maximum amount of times to retry the connection (optional, default `5`) + +Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** + ### getRnsr Find the RNSR identifier(s) matching the `address` and the publication `year` diff --git a/docs/plugin-core.md b/docs/plugin-core.md index 3f99e930..d13a8b5c 100644 --- a/docs/plugin-core.md +++ b/docs/plugin-core.md @@ -12,13 +12,13 @@ npm install @ezs/core Plusieurs instructions permettent de créer des sous flux (sub pipeline), à partir d'un fichier d’instructions ou d'instructions imbriquées. Si elles s'utilisent toutes de la même manière (avec les mêmes paramètres) centaines peuvent apparaître comme similaires mais leur fonctionnement est différent : -- [delegate] : 1 sous flux pour tous les éléments -- [swing] : 1 sous flux pour tous les éléments filtrés selon une condition -- [spaw] : 1 sous flux par élément -- [loop] : 1 sous flux par élément -- [expand] : 1 sous flux pour N éléments (N = size), seul le champ sélectionné est envoyé dans le pipeline -- [combine] : 1 sous flux pour tous les éléments, seul le champ sélectionné est comparé avec le résultat du sous flux -- [singleton] : 1 sous flux pour le premier élément +- [delegate] : 1 sous flux pour tous les éléments +- [swing] : 1 sous flux pour tous les éléments filtrés selon une condition +- [spaw] : 1 sous flux par élément +- [loop] : 1 sous flux par élément +- [expand] : 1 sous flux pour N éléments (N = size), seul le champ sélectionné est envoyé dans le pipeline +- [combine] : 1 sous flux pour tous les éléments, seul le champ sélectionné est comparé avec le résultat du sous flux +- [singleton] : 1 sous flux pour le premier élément ## usage @@ -32,7 +32,7 @@ Plusieurs instructions permettent de créer des sous flux (sub pipeline), à par - [debug](#debug) - [dedupe](#dedupe) - [delegate](#delegate) -- [dispatch](#dispatch) +- [delegate](#delegate-1) - [dump](#dump) - [env](#env) - [exchange](#exchange) @@ -64,6 +64,7 @@ Plusieurs instructions permettent de créer des sous flux (sub pipeline), à par - [truncate](#truncate) - [ungroup](#ungroup) - [unpack](#unpack) +- [use](#use) - [validate](#validate) ### assign @@ -251,23 +252,19 @@ Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G ### delegate -Delegate processing to an external pipeline. - -> **Note**: works like [spawn](#spawn), but each chunk share the same external pipeline. +Break the stream if the control file cannot be checked #### Parameters -- `file` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a file -- `script` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a string of characters -- `commands` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a object -- `command` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a URL-like command -- `logger` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** A dedicaded pipeline described in a file to trap or log errors +- `fusible` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** file to check Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** -### dispatch +### delegate + +Delegate processing to an external pipeline. -Dispatch processing to an external pipeline on one or more servers. +> **Note**: works like [spawn](#spawn), but each chunk share the same external pipeline. #### Parameters @@ -275,6 +272,7 @@ Dispatch processing to an external pipeline on one or more servers. - `script` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a string of characters - `commands` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a object - `command` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a URL-like command +- `logger` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** A dedicaded pipeline described in a file to trap or log errors Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** @@ -491,6 +489,7 @@ fork the current pipeline - `commands` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a object - `command` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a URL-like command - `logger` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** A dedicaded pipeline described in a file to trap or log errors +- `target` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** choose the key to set with the forked request identifier (optional, default `x-request-id`) Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** @@ -1305,6 +1304,26 @@ Take `String`s or `Buffer`s and throw `Object` builded by JSON.parse on each lin Returns **[object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** +### use + +Take all `String`, concat them and throw just one. + +Script: + +```ini +[use] +plugin = basics +plugin = analytics +``` + +#### Parameters + +- `beginWith` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** Add value at the begin +- `joinWith` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** use value to join 2 chunk +- `endWith` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** Add value at the end + +Returns **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** + ### validate From an `Object`, throw the same object if all rules pass diff --git a/packages/basics/README.md b/packages/basics/README.md index bcebd642..29709450 100644 --- a/packages/basics/README.md +++ b/packages/basics/README.md @@ -20,6 +20,7 @@ npm install @ezs/basics - [CSVParse](#csvparse) - [CSVString](#csvstring) - [FILELoad](#fileload) +- [FILEMerge](#filemerge) - [FILESave](#filesave) - [INIString](#inistring) - [JSONParse](#jsonparse) @@ -251,6 +252,37 @@ Output: Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** +### FILEMerge + +Take `Object` or `Buffer` and throw only one document + +```json +[ fi1e1.csv, file2.csv ] +``` + +Script: + +```ini +[use] +plugin = basics + +[FILELoad] +[FILEMerge] +[replace] +path = contentOfFile1AndFile2 +value = self() +``` + +Output: + +```json +[ +(...) +] +``` + +Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** + ### FILESave Take data, convert it to buffer and append it to file @@ -591,6 +623,7 @@ It returns to the output stream - `path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** Regex to select the files to extract (optional, default `"**\/*.json"`) - `json` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** Parse as JSON the content of each file (optional, default `true`) +- `text` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)** The content of each file is converted to a string (otherwise it remains a buffer) (optional, default `true`) - `compress` **[Boolean](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Boolean)** Enable gzip compression (optional, default `false`) Returns **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Array)<{id: [String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String), value: [String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)}>** diff --git a/packages/conditor/README.md b/packages/conditor/README.md index 6b1775bb..7d39c505 100644 --- a/packages/conditor/README.md +++ b/packages/conditor/README.md @@ -56,6 +56,7 @@ Sachant qu'on appauvrit (casse, accents, tiret, apostrophe) tous les champs. - [compareRnsr](#comparernsr) - [conditorScroll](#conditorscroll) - [CORHALFetch](#corhalfetch) +- [CORHALFetch](#corhalfetch-1) - [getRnsr](#getrnsr) - [getRnsrInfo](#getrnsrinfo) - [WOSFetch](#wosfetch) @@ -226,6 +227,39 @@ Output: Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** +### CORHALFetch + +Take `String` as URL, throw each chunk from the result + +Input: + +```json +[ + { q: "toto" }, +] +``` + +Script: + +```ini +[CORHALFetch] +url = https://corhal-api.inist.fr +``` + +Output: + +```json +[{...}, {"a": "b"}, {"a": "c" }] +``` + +#### Parameters + +- `url` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** corhal api url +- `timeout` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** Timeout in milliseconds (optional, default `1000`) +- `retries` **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Number)** The maximum amount of times to retry the connection (optional, default `5`) + +Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** + ### getRnsr Find the RNSR identifier(s) matching the `address` and the publication `year` diff --git a/packages/core/README.md b/packages/core/README.md index 3f99e930..d13a8b5c 100644 --- a/packages/core/README.md +++ b/packages/core/README.md @@ -12,13 +12,13 @@ npm install @ezs/core Plusieurs instructions permettent de créer des sous flux (sub pipeline), à partir d'un fichier d’instructions ou d'instructions imbriquées. Si elles s'utilisent toutes de la même manière (avec les mêmes paramètres) centaines peuvent apparaître comme similaires mais leur fonctionnement est différent : -- [delegate] : 1 sous flux pour tous les éléments -- [swing] : 1 sous flux pour tous les éléments filtrés selon une condition -- [spaw] : 1 sous flux par élément -- [loop] : 1 sous flux par élément -- [expand] : 1 sous flux pour N éléments (N = size), seul le champ sélectionné est envoyé dans le pipeline -- [combine] : 1 sous flux pour tous les éléments, seul le champ sélectionné est comparé avec le résultat du sous flux -- [singleton] : 1 sous flux pour le premier élément +- [delegate] : 1 sous flux pour tous les éléments +- [swing] : 1 sous flux pour tous les éléments filtrés selon une condition +- [spaw] : 1 sous flux par élément +- [loop] : 1 sous flux par élément +- [expand] : 1 sous flux pour N éléments (N = size), seul le champ sélectionné est envoyé dans le pipeline +- [combine] : 1 sous flux pour tous les éléments, seul le champ sélectionné est comparé avec le résultat du sous flux +- [singleton] : 1 sous flux pour le premier élément ## usage @@ -32,7 +32,7 @@ Plusieurs instructions permettent de créer des sous flux (sub pipeline), à par - [debug](#debug) - [dedupe](#dedupe) - [delegate](#delegate) -- [dispatch](#dispatch) +- [delegate](#delegate-1) - [dump](#dump) - [env](#env) - [exchange](#exchange) @@ -64,6 +64,7 @@ Plusieurs instructions permettent de créer des sous flux (sub pipeline), à par - [truncate](#truncate) - [ungroup](#ungroup) - [unpack](#unpack) +- [use](#use) - [validate](#validate) ### assign @@ -251,23 +252,19 @@ Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G ### delegate -Delegate processing to an external pipeline. - -> **Note**: works like [spawn](#spawn), but each chunk share the same external pipeline. +Break the stream if the control file cannot be checked #### Parameters -- `file` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a file -- `script` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a string of characters -- `commands` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a object -- `command` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a URL-like command -- `logger` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** A dedicaded pipeline described in a file to trap or log errors +- `fusible` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** file to check Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** -### dispatch +### delegate + +Delegate processing to an external pipeline. -Dispatch processing to an external pipeline on one or more servers. +> **Note**: works like [spawn](#spawn), but each chunk share the same external pipeline. #### Parameters @@ -275,6 +272,7 @@ Dispatch processing to an external pipeline on one or more servers. - `script` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a string of characters - `commands` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a object - `command` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a URL-like command +- `logger` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** A dedicaded pipeline described in a file to trap or log errors Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** @@ -491,6 +489,7 @@ fork the current pipeline - `commands` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a object - `command` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** the external pipeline is described in a URL-like command - `logger` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** A dedicaded pipeline described in a file to trap or log errors +- `target` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** choose the key to set with the forked request identifier (optional, default `x-request-id`) Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** @@ -1305,6 +1304,26 @@ Take `String`s or `Buffer`s and throw `Object` builded by JSON.parse on each lin Returns **[object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)** +### use + +Take all `String`, concat them and throw just one. + +Script: + +```ini +[use] +plugin = basics +plugin = analytics +``` + +#### Parameters + +- `beginWith` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** Add value at the begin +- `joinWith` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** use value to join 2 chunk +- `endWith` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)?** Add value at the end + +Returns **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** + ### validate From an `Object`, throw the same object if all rules pass