Merge branch 'main' into tests

README.md

@@ -6,8 +6,6 @@ LD Workbench is a Command Line Interface (CLI) application. LD Workbench is test
 
 This project is currently in a Proof-of-Concept phase. Feel free to watch our progress, but please do not use this project in a production setting.
 
-
-
 ## Approach
 
 A *pipeline* is the sequence of *stages*.
@@ -18,15 +16,59 @@ The *iterator* component is configured by a SPARQL Select query. This query bind
 
 Every binding for variable `$this` is used to parameterize a SPARQL Construct query; this is the *generator* component. Parameterization follows [SPARQL pre-binding](https://www.w3.org/TR/shacl/#pre-binding) according to the SHACL standard. Each SPARQL Construct query returns RDF triples that are part of the transformed result.
 
-
-
 ## Configuration
 
 An LD Workbench pipeline is defined with a YAML configuration file. The configuration is validated by a JSON Schema. The schema is part of this repository ([link](https://github.com/netwerk-digitaal-erfgoed/ld-workbench/blob/main/static/ld-workbench.schema.json)). The YAML and JSON Schema combination is tested to work in the VSCode editor.
 
 A pipeline must have a name, one or more stages, and optionally a description. Multiple pipelines can be configured as long as they have unique names. See the [example configuration file](https://github.com/netwerk-digitaal-erfgoed/ld-workbench/blob/main/static/example/config.yml) for a boilerplate configuration file. A visualization of the schema gives more insights on required and optional properties can be [found here](https://json-schema.app/view/%23?url=https%3A%2F%2Fraw.githubusercontent.com%2Fnetwerk-digitaal-erfgoed%2Fld-workbench%2Fmain%2Fstatic%2Fld-workbench.schema.json).
 
+### Example YAML File For Configuration Options
+
+```yaml
+name: MyPipeline
+description: Example pipeline configuration
+destination: output/result.ttl
+stages:
+  - name: Stage1
+    iterator:
+      query: "SELECT ?s ?p ?o WHERE { ?s ?p ?o } LIMIT 100"
+      endpoint: "http://example.com/sparql-endpoint"
+      delay: "150 ms"
+    generator:
+      - query: "CONSTRUCT { ?s ?p ?o } WHERE { ?s ?p ?o }"
+        batchSize: 50
+    destination: output/stage1-result.ttl
+  - name: Stage2
+    iterator:
+      query: file://queries/iteratorQuery.rq
+      endpoint: "http://example.com/sparql-endpoint-1"
+      batchSize: 200
+    generator:
+      - query: file://queries/generator1Query.rq
+        endpoint: "http://example.com/sparql-endpoint-1"
+        batchSize: 200
+      - query: file://queries/generator2Query.rq
+        endpoint: "http://example.com/sparql-endpoint-2"
+        batchSize: 100
+    destination: output/stage2-result.ttl
+```
 
+### Configuration Options Table
+
+| Section                          | Variable           | Description                                                                                                         | Required |
+|----------------------------------|--------------------|---------------------------------------------------------------------------------------------------------------------|----------|
+| General Configuration File       | name               | The name of your pipeline, it must be unique over all your configurations.                                          | Yes      |
+|                                  | description        | An optional description for your pipeline.                                                                          | No       |
+|                                  | destination        | The file where the final result of your pipeline is saved.                                                          | No       |
+| Stage                            | name               | The name of your pipeline step, it must be unique within one configuration.                                         | Yes      |
+|                                  | destination        | The file where the results are saved. This is not a required property; if omitted, a temporary file will be created automatically. | No       |
+| Iterator                         | query              | Path (prefixed with "file://") of SPARQL Query `.rq` file or SPARQL Query string that makes the iterator using SPARQL select.                        | Yes      |
+|                                  | endpoint           | The SPARQL endpoint for the iterator. If it starts with "file://", a local RDF file is queried. If omitted, the result of the previous stage is used. | No       |
+|                                  | batchSize          | Overrule the iterator's behavior of fetching 10 results per request, regardless of any limits in your query.       | No       |
+|                                  | delay              | Human-readable time delay for the iterator's SPARQL endpoint requests (e.g., '5ms', '100 milliseconds', '1s').   | No       |
+| Generator                        | query              | Path (prefixed with "file://") of SPARQL Query `.rq` file or SPARQL Query string that makes the generator using SPARQL construct.                    | Yes      |
+|                                  | endpoint           | The SPARQL endpoint for the generator. If it starts with "file://", a local RDF file is queried. If omitted, the endpoint of the Iterator is used. | No       |
+|                                  | batchSize          | Overrule the generator's behavior of fetching results for 10 bindings of $this per request.                          | No       |
 
 ## Installation
 
@@ -42,13 +84,11 @@ A pipeline must have a name, one or more stages, and optionally a description. M
 2. Install LD Workbench:
 
    ```sh
-   npx @netwerk-digitaal-ergoed/ld-workbench --init
+   npx @netwerk-digitaal-erfgoed/ld-workbench --init
    ```
 
    Your workbench is now ready for use.
 
-
-
 ## Usage
 
 Once installed, an example workbench is present that can be run with the following command:
@@ -57,14 +97,13 @@ Once installed, an example workbench is present that can be run with the followi
 npx @netwerkdigitaalergoed/ld-workbench
 ```
 
-
 ### Configuring a workbench pipeline
 
 To keep your workbench workspace clean, create a folder for each pipeline that contains the configuration and the SPARQL Select and Construct queries. Use the `static` directory for this.
 
 Here is an example of how your file structure may look:
 
-```
+```sh
 ld-workbench
 |-- static
 |   |-- my-pipeline
@@ -73,8 +112,6 @@ ld-workbench
 |   |   |-- construct.rq
 ```
 
-
-
 ## Development
 
 For local development, the following command should get you going:
@@ -95,3 +132,21 @@ npm run ld-workbench -- --configDir static/example
 Since this project is written in Typescript, your code needs to be transpiled to Javascript before you can run it (using `npm run build`). With `npm run dev` the transpiler will watch changes in the Typescript code an transpiles on each change.
 
 The configuration of this project is validated and defined by [JSON Schema](https://json-schema.org). The schema is located in `./static/ld-workbench-schema.json`. To create the types from this schema, run `npm run util:json-schema-to-typescript`. This will regenerate `./src/types/LDWorkbenchConfiguration.d.ts`, do not modify this file by hand.
+
+## Publishing to NPM
+
+To trigger a published release event in GitHub Actions, you need to create a release on your GitHub repository. Here are the general steps:
+
+1. Navigate to the [LDWorkbench GitHub repository](https://github.com/netwerk-digitaal-erfgoed/ld-workbench).
+
+2. Click on the "Releases" tab or visit <https://github.com/netwerk-digitaal-erfgoed/ld-workbench/releases>.
+
+3. Click the "Draft a new release" button if you haven't created a release yet.
+
+4. Fill in the necessary information for your release, such as the tag version, release title, and release description.
+
+5. Optionally, attach any assets (e.g. release notes).
+
+6. Click the "Publish release" button.
+
+By publishing the release, you will trigger the "release" event in GitHub Actions, which can then execute the workflow specified [here](https://github.com/netwerk-digitaal-erfgoed/ld-workbench/blob/main/.github/workflows/npmjs.yml).

package.json

@@ -94,6 +94,7 @@
     "millify": "^6.1.0",
     "n3": "^1.17.2",
     "ora": "^7.0.1",
+    "parse-duration": "^1.1.0",
     "pretty-ms": "^8.0.0",
     "sparqljs": "^3.7.1"
   }

src/lib/Generator.class.ts

@@ -50,7 +50,6 @@ class Generator extends EventEmitter {
       this.iterationsIncoming = count
         for (const $this of this.$thisList) {
           this.run($this, this.$thisList.length)
-
       }
     })
 
@@ -99,4 +98,5 @@ class Generator extends EventEmitter {
   }
 }
 
+
 export default Generator

src/lib/Iterator.class.ts

@@ -10,6 +10,7 @@ import getEndpoint from "../utils/getEndpoint.js";
 import type { Endpoint, QueryEngine } from "./types.js";
 import getEngine from "../utils/getEngine.js";
 import getEngineSource from "../utils/getEngineSource.js";
+import parse from 'parse-duration'
 
 const DEFAULT_LIMIT = 10;
 declare interface Iterator {
@@ -26,65 +27,72 @@ class Iterator extends EventEmitter {
   private readonly query: SelectQuery;
   public readonly endpoint: Endpoint;
   private readonly engine: QueryEngine;
+  private readonly delay: number | undefined
   private source: string = "";
   private $offset = 0;
-  private totalResults = 0;
+  public totalResults = 0;
 
   constructor(stage: Stage) {
     super();
     this.query = getSPARQLQuery(stage.configuration.iterator.query, "select");
     this.query.limit =
-      this.query.limit ??
       stage.configuration.iterator.batchSize ??
+      this.query.limit ??
       DEFAULT_LIMIT;
     this.endpoint = getEndpoint(stage);
     this.engine = getEngine(this.endpoint);
+    if (stage.configuration.iterator.delay !== undefined){
+      const delay = parse(stage.configuration.iterator.delay)
+      if (delay === undefined) throw new Error(`Error in stage \`${stage.configuration.name}\`: incorrect delay format was provided.`)
+      this.delay = delay
+    }
   }
 
   public run(): void {
-    let resultsPerPage = 0;
-    if (this.source === "") this.source = getEngineSource(this.endpoint);
-    this.query.offset = this.$offset;
-    const queryString = getSPARQLQueryString(this.query);
-    const error = (e: any): Error => new Error(
-      `The Iterator did not run succesfully, it could not get the results from the endpoint ${this.source} (offset: ${this.$offset}, limit ${this.query.limit}): ${(e as Error).message}`
-    )
-    this.engine
-      .queryBindings(queryString, {
-        sources: [this.source],
-      })
-      .then((stream) => {
-        stream.on("data", (binding: Bindings) => {
-          resultsPerPage++;
-          if (!binding.has("this"))
-            throw new Error("Missing binding $this in the Iterator result.");
-          const $this = binding.get("this")!;
-          if ($this.termType !== "NamedNode") {
-            throw new Error(
-              `Binding $this in the Iterator result must be an Iri/NamedNode, but it is of type ${$this.termType}.`
-            );
-          } else {
-            this.emit("data", $this);
-          }
-        });
-
-        stream.on("end", () => {
-          this.totalResults += resultsPerPage;
-          this.$offset += this.query.limit!;
-          if (resultsPerPage < this.query.limit!) {
-            this.emit("end", this.totalResults);
-          } else {
-            this.run();
-          }
-        });
-
-        stream.on('error', (e) => {
-          this.emit("error", error(e))
+    setTimeout(() => {                
+      let resultsPerPage = 0;
+      if (this.source === "") this.source = getEngineSource(this.endpoint);
+      this.query.offset = this.$offset;
+      const queryString = getSPARQLQueryString(this.query);
+      const error = (e: any): Error => new Error(
+        `The Iterator did not run succesfully, it could not get the results from the endpoint ${this.source} (offset: ${this.$offset}, limit ${this.query.limit}): ${(e as Error).message}`
+      )
+      this.engine
+        .queryBindings(queryString, {
+          sources: [this.source],
+        })
+        .then((stream) => {
+          stream.on("data", (binding: Bindings) => {
+            resultsPerPage++;
+            if (!binding.has("this"))
+              throw new Error("Missing binding $this in the Iterator result.");
+            const $this = binding.get("this")!;
+            if ($this.termType !== "NamedNode") {
+              throw new Error(
+                `Binding $this in the Iterator result must be an Iri/NamedNode, but it is of type ${$this.termType}.`
+              );
+            } else {
+              this.emit("data", $this);
+            }
+          });
+          stream.on("end", () => {
+            this.totalResults += resultsPerPage;
+            this.$offset += this.query.limit!;
+            if (resultsPerPage < this.query.limit!) {
+              this.emit("end", this.totalResults);
+            } else {
+              this.run();
+            }
+          });
+
+          stream.on('error', (e) => {
+            this.emit("error", error(e))
+          })
         })
-      })
-      .catch((e) => {
-        this.emit("error", error(e))
-      });
+        .catch((e) => {
+          this.emit("error", error(e))
+        });
+    }, this.delay ?? 0)
   }
 }
 

src/lib/LDWorkbenchConfiguration.d.ts

@@ -49,6 +49,10 @@ export interface LDWorkbenchConfiguration {
          * Overrule the iterator's behaviour of fetching 10 results per request, regardless of any limit's in your query.
          */
         batchSize?: number;
+        /**
+         * Human readable time delay for the iterator's SPARQL endpoint requests (e.g. '5ms', '100 milliseconds', '1s').
+         */
+        delay?: string;
       };
       /**
        * @minItems 1
@@ -117,6 +121,10 @@ export interface LDWorkbenchConfiguration {
          * Overrule the iterator's behaviour of fetching 10 results per request, regardless of any limit's in your query.
          */
         batchSize?: number;
+        /**
+         * Human readable time delay for the iterator's SPARQL endpoint requests (e.g. '5ms', '100 milliseconds', '1s').
+         */
+        delay?: string;
       };
       /**
        * @minItems 1

src/lib/Pipeline.class.ts

@@ -38,6 +38,11 @@ class Pipeline {
     const destinationFile =
       this.configuration.destination ??
       `file://${path.join(this.dataDir, "statements.nt")}`;
+    const actualPath = destinationFile.replace(/^file:\/\//, '')
+    if (fs.existsSync(actualPath)) {
+      // removing destintation if it already exists
+        fs.unlinkSync(actualPath)
+      }
     if (
       !isFilePathString(destinationFile) &&
       !isTriplyDBPathString(destinationFile)
@@ -177,9 +182,9 @@ class Pipeline {
     let iterationsProcessed = 0
     if (!(this.opts?.silent === true)) spinner.start();
     await new Promise<void>((resolve, reject) => {
-      stage.on("iteratorResult", (_$this) => {
+      stage.on("iteratorResult", (_$this, quadsGenerated) => {
         iterationsProcessed++
-        if (!(this.opts?.silent === true)) spinner.text = `Running ${stage.name}:\n\n  Elements processed: ${millify(iterationsProcessed)} \n  Duration: ${formatDuration(startTime, performance.now())} `;
+        if (!(this.opts?.silent === true)) spinner.text = `Running ${stage.name}:\n\n  Processed elements: ${millify(iterationsProcessed)}\n  Generated quads: ${millify(quadsGenerated)}\n  Duration: ${formatDuration(startTime, performance.now())} `;
       });
       stage.on("error", (e) => {
         spinner.fail();