Skip to content

Home

Jump to bottom
Rishabh Chandaliya edited this page Apr 1, 2022 · 8 revisions

Semantic Engine

Objectives

We developed and implemented dedicated software components for Semantic Engine System as SEED, that is in charge to manage the semantic metadata, descriptions, queries, discoveries, retrieve, create and map descriptions and manipulate registries, federated queries and components interactions and interfaces. To make easier the interface and use of functionalities we present the external operations via APIs that are more agnostic and easy to use also for non-semantic experts.

semantic engine

Figure 1 – High Level Backplane Blocks Diagram

Building Blocks

building blocks

Figure 2 – High level Operations of the Semantic engine system

For the Semantic subsystems in charge of dealing with “Semantic Data Management” we can highlight parts like:

  • Semantic Data Storage: This component on receiving semantically annotated data push data into triple-store, for example, virtuoso. A Triple-store is database and responsible to store semantic data and process the SPARQL queries.

  • Semantic Data Storage: This component on receiving semantically annotated data push data into triple-store, for example, virtuoso. A Triple-store is database and responsible to store semantic data and process the SPARQL queries.

  • Semantic Mapping: This component does semantic mappings and transform data (JSON to RDF) received from API endpoints.

  • Vocabulary Management: This environment keeps and manages all of the vocabularies, defined as i3-Market Semantic model, used in different operations of the semantic engine.

  • Offering Registration: This component is basically REST APIs exposed as endpoints. Semantic engine exposes different endpoints for offering registration. Examples are:

    • register data provider,
    • register data offering of a data provider,
    • update data offerings,
    • deleting a data offering
    • query existing offerings, etc.

  • Offering Discovery: This component is basically REST APIs exposed as endpoints. Semantic engine exposes different endpoints for offering discoveries and retrieve. Examples are:

    • Retrieve list of Data Offerings,
    • discover data offerings by Providers,
    • discover data offerings by parameters,
    • search for particular metadata, etc.

The following picture shows a detailed landscape of the current set of microservices (cubes), API’s (little yellow rectangles), components (blue rectangles), and storages (white rectangles) on i3-MARKET.

services layout

Figure 3 – i3-MARKET services layout

How the above components interact with each other can be explained with the use of a scenario, for instance, registration of data provider. To complete this task, a data provider fills a template provided by offering registration component. The template as a JSON request is Semantically mapped by the Semantic Mapping component and is forwarded towards the Semantic Data Storage component where this annotated data is pushed into the triple-store. A JSON request is shown in the following example:

{
“description”: “nuig data provider”,
“name”: “iot-unit”,
“organization”: [
{
“address”: “IDA business park, Galway”,
“contactPoint”: “NUIG”,
“description”: “an institute of semantic web”,
“name”: “insight centre for data analytics”
}
],
“providerId”: “123a123”
}

After a Semantically annotation the above JSON request is transformed to RDF as Turtle format:

@prefix schema: <http://schema.org/> .
@prefix dc: <http://purl.org/dc/terms/> .

<http://i3-market.org/resource/123a123>
schema:sourceOrganization <http://i3-market.org/resource/organization/c6b2102a-f051-43cd-acd6-dc77784326b5> ;
schema:description “nuig data provider” ;
a <http://i3-market.org/core#Provider> ;
schema:name “iot-unit” .

<http://i3-market.org/resource/organization/c6b2102a-f051-43cd-acd6-dc77784326b5>
schema:contactPoint “NUIG” ;
a schema:Organization ;
schema:address “IDA business park, Galway” ;
schema:name “insight centre for data analytics” ;
dc:description “an institute of semantic web”.

Technical Requirements

For the Semantic Storage the following high-level capabilities have been defined.

Semantic Meta Data Management

The Semantic engine (SEED) relies on a local and distributed triple-store. All the information for instance, data provider, data offerings, consumer, and querying offering are stored as semantic data.

  • Name : Triple Storage

    Description : The Triple Storage is responsible to store semantic data and process the SPARQL queries. The triple storage should provide either the SPARQL endpoint or client connector so that other components can access to the data.

  • Name : Spatial and Text Data Storage

    Description : To support spatial and full-text search queries, the semantic data manager should be able to index spatial and fulltext data.

  • Name : Save Semantic Data

    Description : As a Subject, I want to save my semantic metadata so I can query and update it later Subject: Data Consumer, Data Provider.

Offering registration

The Semantic engine will expose APIs to register, query and update offering. A data provider can register offering, for instance, datasets and the price for data, etc.

  • Name : Offering Registration

    Description : Offering Registration is a component that allows the user to manage their data offering. More specifically, it provides the following functionalities:

    • Register the data offering
    • Retrieve all the offerings
    • Update/Delete offerings Subscribe to an offering

Offering Discovery

The Semantic engine will expose APIs to query the existing offerings in i3-Market backplane. A data consumer can query datasets, prices for any dataset, offering, etc.

  • Name : Offering Discovery

    Description : Offering Discovery is a component that allows the Data Consumers to search the offering data available on the marketplace. The Data Consumer has to specify the characteristics of the data they are looking for. The Offering Discovery module will then process the data request and returns a list of available offering data that meet their requirements. More specifically, the Offering Discovery should provide the following functionalities:

    • Register the consumer data request
    • Retrieve all the data request of a consumer
    • Process a data request
    • Update/Delete a data request
    • Subscribe to a data request.

Vocabulary Management

  • Name : Vocabulary Management: Semantic Model Management

    Description : Vocabulary Management is a component that is used to manage the i3- market semantic data model. More specifically, the Vocabulary Management should provide the following functionalities:

    • View, search the concepts of i3-Market
    • Allow user to propose a new concept
    • Allow the administrator to add a new concept
    • Allow the administrator to update/delete an existing concept.
Clone this wiki locally