Add an initial set of benchmark tasks (#2)

* Add 2 new tasks

* Add the stream-compression task

* Add the end-to-end streaming tasks

* Add the previously uncommited task

tasks/stream-compression/index.md

@@ -0,0 +1,33 @@
+A benchmark task measuring the compression efficiency of serializations for grouped RDF streams.
+
+## Methodology
+
+### Data
+
+Stream distributions of any dataset in the [`stream` category](../../categories/stream/index.md) of RiverBench may be used for this task.
+
+### Workload
+
+The task consists of serializing RDF data in a grouped form (that is, as a stream of RDF graphs or RDF datasets) to bytes and measuring the size of the obtained representation.
+
+In this task, the time taken to serialize and deserialize the data is not considered – see the [`stream-serialization-throughput`](../stream-serialization-throughput/index.md) and [`stream-deserialization-throughput`](../stream-deserialization-throughput/index.md) tasks for that aspect.
+
+### Metrics
+
+- The primary metric is the serialized representation size of the RDF data, in bytes.
+- Additionally, the compression ratio can be calculated as the ratio of the reference size to the compressed size. The reference size is the size of the same data serialized using a baseline method, e.g., the N-Triples serialization format.
+    - In the RDF literature, the "compression ratio" is often defined as the inverse of the above definition and expressed as a percentage. For example, a compression ratio of (50%) means that the compressed data is half the size of the reference data.
+
+## Results
+
+There are no results with RiverBench available for this task yet.
+
+## Examples and references
+
+- In the paper about the Jelly streaming protocol, such a benchmark is performed in Section IV.C. The authors have measured the output size of several methods. The presented "Compression ratio" metric there refers to the ratio between the compressed data size and the reference data size, with N-Triples used as the reference.
+    - Sowiński, P., Wasielewska-Michniewska, K., Ganzha, M., & Paprzycki, M. (2022, October). Efficient RDF streaming for the edge-cloud continuum. In 2022 IEEE 8th World Forum on Internet of Things (WF-IoT) (pp. 1-8). IEEE.
+    - https://doi.org/10.1109/WF-IoT54382.2022.10152225
+
+## See also
+
+- Version of this task for flat RDF streams: [`flat-compression`](../flat-compression/index.md)

tasks/stream-compression/metadata.ttl

@@ -0,0 +1,22 @@
+@prefix : <https://w3id.org/riverbench/temp#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix foaf: <http://xmlns.com/foaf/0.1/> .
+@prefix rb: <https://w3id.org/riverbench/schema/metadata#> .
+@prefix rbdoc: <https://w3id.org/riverbench/schema/documentation#> .
+
+:task
+    # General information
+    a rb:Task ;
+    dcterms:conformsTo <https://w3id.org/riverbench/schema/metadata> ;
+    dcterms:identifier "stream-compression" ;
+    dcterms:title "Grouped RDF stream compression"@en ;
+    dcterms:description "A benchmark task measuring the compression efficiency of serializations for grouped RDF streams."@en ;
+
+    # Authors
+    dcterms:creator [
+        foaf:name "Piotr Sowiński" ;
+        foaf:nick "Ostrzyciel" ;
+        foaf:homepage <https://github.com/Ostrzyciel>, <https://orcid.org/0000-0002-2543-9461> ;
+        rbdoc:hasDocWeight 1 ;
+    ]
+.

tasks/stream-deserialization-throughput/index.md

@@ -0,0 +1,37 @@
+A benchmark task measuring the througput of deserializing a grouped RDF stream (that is, a stream in which the elements are either RDF graphs or RDF datasets) from a byte stream to memory.
+
+## Methodology
+
+### Data
+
+Stream distributions of any dataset in the [`stream` category](../../categories/stream/index.md) of RiverBench may be used for this task.
+
+### Workload
+
+The task consists of deserializing RDF data stored in a byte stream to memory in a grouped form (that is, as a stream of RDF graphs or RDF datasets). To isolate the performance of the deserializer itself, the following steps are taken:
+
+- The data (serialized RDF graphs/datasets in the tested serialization format) is preloaded into memory before the benchmark starts.
+- The deserialization process is repeated multiple times to amortize the cost of the initial setup, just-in-time code recompilation, and other external factors.
+- The deserialized statements are not inserted into any data structure or database, but just temporarily stored in memory and immediately discarded. This is to avoid the overhead of maintaining additional data structures.
+    - When possible, the deserialized data should NOT be inserted into a data structure dedicated for RDF graphs or datasets, as these typically include additional processing steps (e.g., updating indexes) that are not part of the deserialization process. Instead, the deserialized statements should ideally be simply iterated over and discarded or inserted into an array.
+
+### Metrics
+
+- Throughput of the deserialization process, measured in RDF statements (triples or quads) per second. This is calculated as the total number of RDF statements deserialized divided by the total time taken to deserialize them.
+- Additionally, the throughput may be measured in terms of stream elements (RDF graphs or RDF datasets) per second.
+
+## Results
+
+There are no results with RiverBench available for this task yet.
+
+## Examples and references
+
+- In the paper about the Jelly streaming protocol, such a benchmark is performed in Section IV.B. The corresponding task in the paper is named "Raw deserialization throughput" and the performance in measured in terms of the number of triples deserialized per second.
+    - Sowiński, P., Wasielewska-Michniewska, K., Ganzha, M., & Paprzycki, M. (2022, October). Efficient RDF streaming for the edge-cloud continuum. In 2022 IEEE 8th World Forum on Internet of Things (WF-IoT) (pp. 1-8). IEEE.
+    - https://doi.org/10.1109/WF-IoT54382.2022.10152225
+
+
+## See also
+
+- Version of this task for flat RDF streams: [`flat-deserialization-throughput`](../flat-deserialization-throughput/index.md)
+- The inverse task: [`stream-serialization-throughput`](../stream-serialization-throughput/index.md)

tasks/stream-deserialization-throughput/metadata.ttl

@@ -0,0 +1,22 @@
+@prefix : <https://w3id.org/riverbench/temp#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix foaf: <http://xmlns.com/foaf/0.1/> .
+@prefix rb: <https://w3id.org/riverbench/schema/metadata#> .
+@prefix rbdoc: <https://w3id.org/riverbench/schema/documentation#> .
+
+:task
+    # General information
+    a rb:Task ;
+    dcterms:conformsTo <https://w3id.org/riverbench/schema/metadata> ;
+    dcterms:identifier "stream-deserialization-throughput" ;
+    dcterms:title "Grouped streaming deserialization throughput"@en ;
+    dcterms:description "A benchmark task measuring the througput of deserializing a grouped RDF stream (that is, a stream in which the elements are either RDF graphs or RDF datasets) from a byte stream to memory."@en ;
+
+    # Authors
+    dcterms:creator [
+        foaf:name "Piotr Sowiński" ;
+        foaf:nick "Ostrzyciel" ;
+        foaf:homepage <https://github.com/Ostrzyciel>, <https://orcid.org/0000-0002-2543-9461> ;
+        rbdoc:hasDocWeight 1 ;
+    ]
+.

tasks/stream-latency-end-to-end/index.md

@@ -1,7 +1,54 @@
-A benchmark task measuring the latency of streaming elements in a grouped RDF stream over the network, including serialization and deserialization.
+A benchmark task measuring the latency of streaming elements in a grouped RDF stream over the network, including serialization, network communication, and deserialization.
 
-**[See the list of results for this task.](results.md)**
+## Methodology
 
-## Task description
+### Data
 
-**TODO**
+Stream distributions of any dataset in the [`stream` category](../../categories/stream/index.md) of RiverBench may be used for this task.
+
+### Workload
+
+The task consists of setting up two network applications: the **producer** and the **consumer**. The producer is tasked with serializing a grouped RDF stream and sending it over the network to the consumer. The consumer receives the serialized data and deserializes it. This benchmark task measures the time taken to serialize, transmit, and deserialize each individual element in the stream.
+
+The task may be considered in the constrained or unconstrained variant:
+
+- **Constrained** – the producer sends stream elements at a fixed rate, and the consumer processes them as they arrive. This simulates a scenario in which the stream is not saturated, found for example in IoT sensor networks.
+- **Unconstrained** – the producer sends stream elements as fast as possible, and the consumer processes them as they arrive. This simulates a scenario in which the stream is saturated, found for example in large cloud-based systems.
+
+In both cases, the producer should reactively slow down to not overload the consumer or the network. The producer may use various backpressure mechanisms to achieve this.
+
+To isolate the performance of the streaming process itself, the following steps are taken:
+
+- The data (RDF graphs/datasets) is preloaded into the memory of the producer before the benchmark starts.
+- The RDF data is stored by the producer in a data structure that is trivially iterable (e.g., an array of arrays or an array of sets).
+- The experiment is repeated multiple times to amortize the cost of the initial setup, just-in-time code recompilation, and other external factors.
+- The test stream should be large enough to reasonably emulate a real-world scenario. A run of at least 1000 stream elements should be enough to trigger any potential performance issues.
+- The consumer receives the serialized data and deserializes it, but the deserialized statements are not inserted into any data structure or database, but just temporarily stored in memory and immediately discarded. This is to avoid the overhead of maintaining additional data structures.
+    - When possible, the deserialized data should NOT be inserted into a data structure dedicated for RDF graphs or datasets, as these typically include additional processing steps (e.g., updating indexes) that are not part of the deserialization process. Instead, the deserialized statements should ideally be simply iterated over and discarded or inserted into an array.
+
+### Technical notes
+
+- Various streaming protocols may be used in the experiments, such as MQTT, Kafka, or gRPC. A common example of such a setup would be to send stream elements (RDF graphs) as Turtle files over MQTT.
+- The network communication may be simulated locally (e.g., using loopback interfaces) or over a real network connection. The latter may introduce additional variability in the results due to network latency, bandwidth, and other factors.
+    - Alternatively, specific network conditions may be simulated using tools like `tc` (Linux Traffic Control, NetEm kernel module) or network emulators.
+- When both the consumer and the producer are on the same physical machine, the impact of possible resource contention (e.g., CPU time, CPU cache, memory bandwidth) should be considered. The impact will be very small for modern, multi-core machines with lots of memory bandwidth and cache, but for small devices (e.g., IoT devices) it may be very significant.
+- When the producer and the consumer are on different physical machines, achieving sub-millisecond accuracy in latency measurement will be very challenging. Specialized timing hardware may be used for this purpose.
+    - Alternatively, the producer and the consumer may be placed on the same machine (or even in the same process) so that they can share a common monotonic clock for latency measurements. Such a clock is available, e.g., in Java as `System.nanoTime()`.
+
+### Metrics
+
+- Latency of streaming a stream element, measured typically in milliseconds. The measurement starts when the producer begins serializing the element and ends when the consumer finishes deserializing it.
+
+## Results
+
+There are no results with RiverBench available for this task yet.
+
+## Examples and references
+
+- In the paper about the Jelly streaming protocol, such a benchmark is performed in Section IV.E. The authors have measured the end-to-end latency of several methods (over Kafka and gRPC) in terms of the number of triples transmitted per second. In the experiment, both the producer and the consumer were running on the same physical machine (in the same Java Virtual Machine) and various network conditions were simulated using the NetEm Linux kernel module. The latency measurement was done using the machine's monotonic clock.
+    - Sowiński, P., Wasielewska-Michniewska, K., Ganzha, M., & Paprzycki, M. (2022, October). Efficient RDF streaming for the edge-cloud continuum. In 2022 IEEE 8th World Forum on Internet of Things (WF-IoT) (pp. 1-8). IEEE.
+    - https://doi.org/10.1109/WF-IoT54382.2022.10152225
+
+## See also
+
+- Version of this task focusing on stream throughput, not latency: [`stream-throughput-end-to-end`](../stream-throughput-end-to-end/index.md)

tasks/stream-latency-end-to-end/metadata.ttl

@@ -10,7 +10,7 @@
     dcterms:conformsTo <https://w3id.org/riverbench/schema/metadata> ;
     dcterms:identifier "stream-latency-end-to-end" ;
     dcterms:title "End-to-end streaming latency"@en ;
-    dcterms:description "A benchmark task measuring the latency of streaming elements in a grouped RDF stream over the network, including serialization and deserialization."@en ;
+    dcterms:description "A benchmark task measuring the latency of streaming elements in a grouped RDF stream over the network, including serialization, network communication, and deserialization."@en ;
 
     # Authors
     dcterms:creator [

tasks/stream-serialization-throughput/index.md

@@ -0,0 +1,38 @@
+A benchmark task measuring the througput of serializing a grouped RDF stream (that is, a stream in which the elements are either RDF graphs or RDF datasets).
+
+## Methodology
+
+### Data
+
+Stream distributions of any dataset in the [`stream` category](../../categories/stream/index.md) of RiverBench may be used for this task.
+
+### Workload
+
+The task consists of serializing RDF data stored in memory in a grouped form (that is, as a stream of RDF graphs or RDF datasets) to a byte stream. The data is stored in memory in the form of an array of RDF graphs or datasets, or similar. To isolate the performance of the serializer itself, the following steps are taken:
+
+- The data (RDF graphs/datasets) is preloaded into memory before the benchmark starts.
+- The RDF data is stored in a data structure that is trivially iterable (e.g., an array of arrays or an array of sets).
+- The serialization process is repeated multiple times to amortize the cost of the initial setup, just-in-time code recompilation, and other external factors.
+- The serialized data is typically not written to disk, but just temporarily stored in memory and immediately discarded. This is to avoid the overhead of disk I/O.
+    - Alternatively, in benchmarks that wish to evaluate the performance of writing to disk, especially considering the impact of different disk usage patterns (e.g., sequential vs. random access), the data may be written to disk.
+
+### Metrics
+
+- Throughput of the serialization process, measured in RDF statements (triples or quads) per second. This is calculated as the total number of RDF statements serialized divided by the total time taken to serialize them.
+- Additionally, the throughput may be measured in terms of stream elements (RDF graphs or RDF datasets) per second.
+
+## Results
+
+There are no results with RiverBench available for this task yet.
+
+## Examples and references
+
+- In the paper about the Jelly streaming protocol, such a benchmark is performed in Section IV.B. The corresponding task in the paper is named "Raw serialization throughput" and the performance in measured in terms of the number of triples serialized per second.
+    - Sowiński, P., Wasielewska-Michniewska, K., Ganzha, M., & Paprzycki, M. (2022, October). Efficient RDF streaming for the edge-cloud continuum. In 2022 IEEE 8th World Forum on Internet of Things (WF-IoT) (pp. 1-8). IEEE.
+    - https://doi.org/10.1109/WF-IoT54382.2022.10152225
+
+
+## See also
+
+- Version of this task for flat RDF streams: [`flat-serialization-throughput`](../flat-serialization-throughput/index.md)
+- The inverse task: [`stream-deserialization-throughput`](../stream-deserialization-throughput/index.md)

tasks/stream-serialization-throughput/metadata.ttl

@@ -0,0 +1,22 @@
+@prefix : <https://w3id.org/riverbench/temp#> .
+@prefix dcterms: <http://purl.org/dc/terms/> .
+@prefix foaf: <http://xmlns.com/foaf/0.1/> .
+@prefix rb: <https://w3id.org/riverbench/schema/metadata#> .
+@prefix rbdoc: <https://w3id.org/riverbench/schema/documentation#> .
+
+:task
+    # General information
+    a rb:Task ;
+    dcterms:conformsTo <https://w3id.org/riverbench/schema/metadata> ;
+    dcterms:identifier "stream-serialization-throughput" ;
+    dcterms:title "Grouped RDF stream serialization throughput"@en ;
+    dcterms:description "A benchmark task measuring the througput of serializing a grouped RDF stream (that is, a stream in which the elements are either RDF graphs or RDF datasets)."@en ;
+
+    # Authors
+    dcterms:creator [
+        foaf:name "Piotr Sowiński" ;
+        foaf:nick "Ostrzyciel" ;
+        foaf:homepage <https://github.com/Ostrzyciel>, <https://orcid.org/0000-0002-2543-9461> ;
+        rbdoc:hasDocWeight 1 ;
+    ]
+.