Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cleanup analytics docs and test #388

Merged
merged 30 commits into from
Feb 7, 2024
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
640c8e3
refactor: first draft of a small docs and test cleanup
AlasDiablo Dec 14, 2023
7deb2a0
test (analytics): add different data set to tune
AlasDiablo Dec 14, 2023
ae66698
test (analytics): fix typo
AlasDiablo Dec 14, 2023
93f1d31
Merge branch 'master' into docs-tests-analytics
AlasDiablo Jan 24, 2024
75f3521
docs (analytics): update tune doc
AlasDiablo Jan 24, 2024
eb7b3a3
docs (analytics): fix typo
AlasDiablo Jan 24, 2024
f8969d1
Merge branch 'master' into docs-tests-analytics
AlasDiablo Jan 24, 2024
5797e4f
docs (analytics): generify tests
AlasDiablo Jan 24, 2024
e95c6d3
docs (analytics): generated doc
AlasDiablo Jan 24, 2024
3084e28
test (analytics): fix test
AlasDiablo Jan 24, 2024
41d61e2
refactor (analytics): update value
AlasDiablo Jan 25, 2024
4182d47
docs (analytics): fix typo in json example
AlasDiablo Jan 25, 2024
e5399fe
docs (analytics): fix typo
AlasDiablo Jan 25, 2024
df570cd
refactor (analytics): update summing doc and add test
AlasDiablo Jan 25, 2024
dc721a2
test (analytics): add test how send badly structured data
AlasDiablo Jan 25, 2024
8ddd539
docs (analytics): add sort and statistics doc and fix some typo
AlasDiablo Jan 26, 2024
b93f5e4
Merge branch 'master' into docs-tests-analytics
AlasDiablo Jan 29, 2024
77bf346
docs (analytics): add slice doc
AlasDiablo Jan 29, 2024
d0f5093
docs: add Engine scope jsdoc type
AlasDiablo Jan 29, 2024
edb6743
docs (analytics): reformat segment doc
AlasDiablo Jan 29, 2024
25a8f62
docs (core): add missing private
AlasDiablo Jan 29, 2024
a776fe3
docs (analytics): fix typo
AlasDiablo Jan 30, 2024
164c119
docs (analytics): typo
AlasDiablo Jan 30, 2024
e0c1920
docs (analytics): add reducing documentation
AlasDiablo Jan 30, 2024
cff6018
docs (analytics): reformat pluck doc and fix reducing doc
AlasDiablo Jan 31, 2024
eaa00a5
docs (analytics): fix nitpick typo
AlasDiablo Jan 31, 2024
2947063
docs (analytics): reformat pair doc
AlasDiablo Jan 31, 2024
91d2410
refactor (analytics): remove code added for no reason
AlasDiablo Jan 31, 2024
67ee08d
refactor (analytics): reformat output doc
AlasDiablo Jan 31, 2024
a26ce90
Merge branch 'master' into docs-tests-analytics
touv Feb 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
67 changes: 55 additions & 12 deletions docs/plugin-analytics.md
Original file line number Diff line number Diff line change
Expand Up @@ -1504,34 +1504,77 @@ Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G

### tune

Take all `Object` and sort them with selected field
Create and replace the id with a unify id that can be used with [sort](#sort)

```json
[{
}]
```
Créer et remplacer l'identifiant par un identifiant unifié qui peut être utilisé avec [sort](#sort)

Script:
#### Example / Exemple

##### Script / Scénario

```ini
; Import analytics plugin required to use tune
; Importation du plugin analytique nécessaire pour utiliser tune
[use]
plugin = analytics

; Using "tune" with default settings
; Utilisation de "tune" avec les paramètres par défaut
[tune]
```

Output:
##### Input / Entrée

```json
[
]
[
{
"id": 1,
"value": 1
},
{
"id": 2,
"value": 2
}
]
```

#### Parameters
##### Output / Sortie

```json
[
{
"id": "0000000000000000001.00000000000000000000",
"value": {
"id": 1,
"value": 1,
"label": "static value"
}
},
{
"id": "0000000000000000002.00000000000000000000",
"value": {
"id": 2,
"value": 2,
"label": "static value"
}
}
]
```

- `path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** path to use for the sort key (optional, default `id`)
#### Parameters

Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)**
- `path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** <ul><li>path of the element used to create the unified identifier</li></ul>
<ul><li>chemin de l'élément utilisé pour créer l'identifiant unifié</li></ul> (optional, default `id`)
- `method` **(`"natural"` \| `"levenshtein"` \| `"numerical"`)** <ul><li>method used to create the unified identifier</li></ul>
<ul><ul><li>natural - Create a normalised identifier that is set to a fixed length</li></ul></ul>
<ul><ul><li>levenshtein - Create an identifier based on the Levenshtein algorithm</li></ul></ul>
<ul><ul><li>numerical - Create an identifier based on a numeric value</li></ul></ul>
<ul><li>méthode utilisée pour créer l'identifiant unifié</li></ul>
<ul><ul><li>natural - Crée un identifiant normalisé de longueur fixe</li></ul></ul>
<ul><ul><li>levenshtein - Crée un identifiant basé sur l'algorithme de Levenshtein</li></ul></ul>
<ul><ul><li>numerical - Crée un identifiant basé sur une valeur numérique</li></ul></ul> (optional, default `natural`)

Returns **{id: [String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String), value: [Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)}**

### value

Expand Down
67 changes: 55 additions & 12 deletions packages/analytics/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1504,34 +1504,77 @@ Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G

### tune

Take all `Object` and sort them with selected field
Create and replace the id with a unify id that can be used with [sort](#sort)

```json
[{
}]
```
Créer et remplacer l'identifiant par un identifiant unifié qui peut être utilisé avec [sort](#sort)

Script:
#### Example / Exemple

##### Script / Scénario

```ini
; Import analytics plugin required to use tune
; Importation du plugin analytique nécessaire pour utiliser tune
[use]
plugin = analytics

; Using "tune" with default settings
; Utilisation de "tune" avec les paramètres par défaut
[tune]
```

Output:
##### Input / Entrée

```json
[
]
[
{
"id": 1,
"value": 1
},
{
"id": 2,
"value": 2
}
]
```

#### Parameters
##### Output / Sortie

```json
[
{
"id": "0000000000000000001.00000000000000000000",
"value": {
"id": 1,
"value": 1,
"label": "static value"
}
},
{
"id": "0000000000000000002.00000000000000000000",
"value": {
"id": 2,
"value": 2,
"label": "static value"
}
}
]
```

- `path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** path to use for the sort key (optional, default `id`)
#### Parameters

Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)**
- `path` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** <ul><li>path of the element used to create the unified identifier</li></ul>
<ul><li>chemin de l'élément utilisé pour créer l'identifiant unifié</li></ul> (optional, default `id`)
- `method` **(`"natural"` \| `"levenshtein"` \| `"numerical"`)** <ul><li>method used to create the unified identifier</li></ul>
<ul><ul><li>natural - Create a normalised identifier that is set to a fixed length</li></ul></ul>
<ul><ul><li>levenshtein - Create an identifier based on the Levenshtein algorithm</li></ul></ul>
<ul><ul><li>numerical - Create an identifier based on a numeric value</li></ul></ul>
<ul><li>méthode utilisée pour créer l'identifiant unifié</li></ul>
<ul><ul><li>natural - Crée un identifiant normalisé de longueur fixe</li></ul></ul>
<ul><ul><li>levenshtein - Crée un identifiant basé sur l'algorithme de Levenshtein</li></ul></ul>
<ul><ul><li>numerical - Crée un identifiant basé sur une valeur numérique</li></ul></ul> (optional, default `natural`)

Returns **{id: [String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String), value: [Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)}**

### value

Expand Down
156 changes: 116 additions & 40 deletions packages/analytics/src/tune.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,11 @@ import clone from 'lodash.clone';
import { levenshteinDistance } from './algorithms';
import core from './core';

/**
* @private
* @param s {string | number | unknown}
* @returns {string}
*/
export const normalize = (s) => {
if (typeof s === 'string') {
return String(s).normalize('NFD').replace(/[\u0300-\u036f]/g, '').padEnd(40, '~');
Expand All @@ -13,7 +18,20 @@ export const normalize = (s) => {
return String(s);
};

/**
* @private
* @param x {string | number | unknown}
* @param y {string | number | unknown}
* @returns {*}
*/
const levenshtein = (x, y) => levenshteinDistance(normalize(x), normalize(y));

/**
* @private
* @param x {number}
* @param y {number}
* @returns {number}
*/
const numerical = (x, y) => (x + 1) / (y + 1);

const methods = {
Expand All @@ -23,41 +41,20 @@ const methods = {
const allMethods = Object.keys(methods).join(',');

/**
* Take all `Object` and sort them with selected field
*
* ```json
* [{
* }]
* ```
*
* Script:
*
* ```ini
* [use]
* plugin = analytics
*
* [tune]
*
* ```
*
* Output:
*
* ```json
* [
* ]
* ```
*
* @name tune
* @param {String} [path=id] path to use for the sort key
* @returns {Object}
* Tune function see documenation at the end.
* This part of the doc is use for jsdoc typing
* @private
* @param data {unknown}
* @param feed
* @param ctx
*/
export default function tune(data, feed) {
if (this.isLast()) {
const tune = (data, feed, ctx) => {
if (ctx.isLast()) {
feed.close();
return;
}
const path = this.getParam('path', 'id');
const method = this.getParam('method', 'natural');
const path = ctx.getParam('path', 'id');
const method = ctx.getParam('method', 'natural');
const fields = Array.isArray(path) ? path : [path];
const currentValue = fields
.filter((k) => typeof k === 'string')
Expand All @@ -73,21 +70,100 @@ export default function tune(data, feed) {
return;
}

if (!this.previousValue) {
this.previousValue = currentValue;
this.previousDistance = 1;
if (!ctx.previousValue) {
ctx.previousValue = currentValue;
ctx.previousDistance = 1;
feed.send(core(1, data));
return;
}

const similarity = methods[method](this.previousValue, currentValue);
const score = similarity === 0 ? Math.max(this.previousValue.length, currentValue.length) : similarity;
const currentDistance = this.previousDistance / score;
const similarity = methods[method](ctx.previousValue, currentValue);
const score = similarity === 0 ? Math.max(ctx.previousValue.length, currentValue.length) : similarity;
const currentDistance = ctx.previousDistance / score;

this.previousValue = clone(currentValue);
this.previousDistance = clone(currentDistance);
ctx.previousValue = clone(currentValue);
ctx.previousDistance = clone(currentDistance);
feed.send(core(currentDistance, data));
}
};

/**
* Create and replace the id with a unify id that can be used with [sort](#sort)
*
* Créer et remplacer l'identifiant par un identifiant unifié qui peut être utilisé avec [sort](#sort)
*
* ### Example / Exemple
*
* #### Script / Scénario
*
* ```ini
* ; Import analytics plugin required to use tune
* ; Importation du plugin analytique nécessaire pour utiliser tune
* [use]
* plugin = analytics
*
* ; Using "tune" with default settings
* ; Utilisation de "tune" avec les paramètres par défaut
* [tune]
*
* ```
*
* #### Input / Entrée
*
* ```json
* [
* {
* "id": 1,
* "value": 1
* },
* {
* "id": 2,
* "value": 2
* }
* ]
* ```
*
* #### Output / Sortie
*
* ```json
* [
* {
* "id": "0000000000000000001.00000000000000000000",
* "value": {
* "id": 1,
* "value": 1,
* "label": "static value"
* }
* },
* {
* "id": "0000000000000000002.00000000000000000000",
* "value": {
* "id": 2,
* "value": 2,
* "label": "static value"
* }
* }
* ]
* ```
*
* @name tune
* @param {String} [path=id]
* <ul><li>path of the element used to create the unified identifier</li></ul>
* <ul><li>chemin de l'élément utilisé pour créer l'identifiant unifié</li></ul>
* @param {'natural' | 'levenshtein' | 'numerical'} [method=natural]
* <ul><li>method used to create the unified identifier</li></ul>
* <ul><ul><li>natural - Create a normalised identifier that is set to a fixed length</li></ul></ul>
* <ul><ul><li>levenshtein - Create an identifier based on the Levenshtein algorithm</li></ul></ul>
* <ul><ul><li>numerical - Create an identifier based on a numeric value</li></ul></ul>
* <ul><li>méthode utilisée pour créer l'identifiant unifié</li></ul>
* <ul><ul><li>natural - Crée un identifiant normalisé de longueur fixe</li></ul></ul>
* <ul><ul><li>levenshtein - Crée un identifiant basé sur l'algorithme de Levenshtein</li></ul></ul>
* <ul><ul><li>numerical - Crée un identifiant basé sur une valeur numérique</li></ul></ul>
* @returns {{
* id: String,
* value: Object
* }}
*/
export default tune;

/*

Expand Down
23 changes: 23 additions & 0 deletions packages/analytics/test-utils/runEzs.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
import from from 'from';

/**
* Helper function use to call ezs in each tests
* @param ezsRuntime {(name: string, options: any, environment?: unknown) => NodeJS.WritableStream}
* @param dataSet {Array<unknown>}
* @param functionName {string}
* @param [options] {any}
* @returns {Promise<Array<unknown>>}
*/
const runEzs = (ezsRuntime, dataSet, functionName, options) => new Promise((resolve) => {
const result = [];
from(dataSet)
.pipe(ezsRuntime(functionName, options))
.on('data', (chunk) => {
result.push(chunk);
})
.on('end', () => {
resolve(result);
});
});

export default runEzs;
Loading