v5.6.0

Version Summary

Version Number : v5.6.0 [tag: v5.6.0]
Date: 15-November-2024

Projects / Repos Included in this release: DX Resource Server, DX Deployment and installation, DX's Documentation

Release Summary :

Data Exchange Resource Server v5.6.0 is released with the following enhancements

• Included wiki with extended documentation for the following
  • Architecture
  • Usage guides
  • Configuration Mapping
  • Prerequisites
• Added JWT leeway to allow time synchronization
• Added ingestion API
• Improved Async API
• Software License:
  • With this release, all DX Softwares will be distributed under Apache 2.0 License

Test Reports:

Please find the release test details and reports here

API Docs

The api docs can be found here.

How to use the Release:

External Dependencies Installation

The Resource Server connects with various external dependencies namely

• ELK stack
• PostgreSQL
• RabbitMQ
• Redis
• AWS S3

Once the above setup is ready as mentioned in the deployment, you can install the DX Resource Server.

Docker based Installation

1. Install docker and docker-compose
2. Clone this repo
3. Build the images
   ./docker/build.sh
4. Modify the docker-compose.yml file to map the config file you just created
5. Start the server in production (prod) or development (dev) mode using docker-compose
   docker-compose up prod

Maven based Installation

1. Install java 11 and maven
2. Use the maven exec plugin based starter to start the server
   mvn clean compile exec:java@resource-server

JAR based Installation

1. Install java 11 and maven
2. Set Environment variables

export RS_URL=https://<rs-domain-name>
export LOG_LEVEL=INFO

3. Use maven to package the application as a JAR
   mvn clean package -Dmaven.test.skip=true
4. 2 JAR files would be generated in the target/ directory
   • iudx.resource.server-cluster-0.0.1-SNAPSHOT-fat.jar - clustered vert.x containing micrometer metrics
   • iudx.resource.server-dev-0.0.1-SNAPSHOT-fat.jar - non-clustered vert.x and does not contain micrometer metrics

Running the clustered JAR

Note: The clustered JAR requires Zookeeper to be installed. Refer here to learn more about how to set up Zookeeper. Additionally, the zookeepers key in the config being used needs to be updated with the IP address/domain of the system running Zookeeper.

The JAR requires 3 runtime arguments when running:

• --config/-c : path to the config file
• --hostname/-i : the hostname for clustering
• --modules/-m : comma separated list of module names to deploy

e.g. java -jar target/iudx.resource.server-cluster-0.0.1-SNAPSHOT-fat.jar --host $(hostname) -c configs/config.json -m iudx.resource.server.database.archives.DatabaseVerticle,iudx.resource.server.authenticator.AuthenticationVerticle ,iudx.resource.server.metering.MeteringVerticle,iudx.resource.server.database.postgres.PostgresVerticle

Use the --help/-h argument for more information. You may additionally append an RS_JAVA_OPTS environment variable containing any Java options to pass to the application.

e.g.

$ export RS_JAVA_OPTS="-Xmx4096m"
$ java $RS_JAVA_OPTS -jar target/iudx.resource.server-cluster-0.0.1-SNAPSHOT-fat.jar ...

Running the non-clustered JAR

The JAR requires 1 runtime argument when running:

• --config/-c : path to the config file

e.g. java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory -jar target/iudx.resource.server-dev-0.0.1-SNAPSHOT-fat.jar -c configs/config.json

Use the --help/-h argument for more information. You may additionally append an RS_JAVA_OPTS environment variable containing any Java options to pass to the application.

e.g.

$ export RS_JAVA_OPTS="-Xmx1024m"
$ java $RS_JAVA_OPTS -jar target/iudx.resource.server-dev-0.0.1-SNAPSHOT-fat.jar ...

Enabling Data Encryption in transit

All the count and search APIs have a feature to get encrypted data.
To get the data in encrypted format, the user could provide a publicKey in the header, with the value that is generated from lazySodium sealed box.
The header value should be in url-safe base64 format.
The encrypted data could be decrypted using the lazysodium sealed box by supplying the private and public key.

License

With this release, all DX Softwares will be distributed under Apache 2.0 License

Issues and Suggestions

Any Release related issues/suggestions, please raise here
For all other issues/suggestions related specific projects, please raise them to the respective project repo/issues.

What's Changed

• docs version update by @ankitmashu in #491
• collection veriosn updated by @ananjaykumar2 in #496
• Update flyway script by @shreelakshmijoshi in #498
• ogc audit table script by @ankitmashu in #500
• Update openapi by @shreelakshmijoshi in #502
• Update audit schema by @shreelakshmijoshi in #505
• Fix/git ignore by @ankitmashu in #511
• enhanced subs to revert changes in failure of pg by @ankitmashu in #508
• Fix/response fix by @ankitmashu in #514
• added primary key in postgres by @ankitmashu in #517
• git ignored by @ankitmashu in #516
• remove call back subscriptions by @ankitmashu in #519
• feat: string based entities search by @pranavrd in #522
• updated recordIssues attributes in jenkins file by @karthickp432001 in #525
• Separated docker tag workflow file by @sivanaikk in #527
• jwt leeway time added by @tanvi029 in #532
• ingestion entities api by @ankitmashu in #538
• async csv file by @ankitmashu in #544
• sync-with-main by @gopal-mahajan in #547
• publish adaptor , rmq push message change by @ankitmashu in #548
• disabled-logging-sensitive-info by @gopal-mahajan in #550
• ownerid and provider id check by @ankitmashu in #552
• config examples by @ankitmashu in #553
• unused callback databroker by @ankitmashu in #555
• updated docs and design docs for github by @ankitmashu in #556
• updated arch image by @ankitmashu in #557
• Make time limits for search queries configurable by @ananjaykumar2 in #559
• Update README.md by @swaminathanvasanth in #560
• Update Solution_Architecture.md by @swaminathanvasanth in #561
• Update Configurations.md by @swaminathanvasanth in #562
• Update Explanation.md by @swaminathanvasanth in #563
• Docs/documentation updation by @ankitmashu in #565
• final docs updates by @ankitmashu in #566
• rest assured fix for pipelines by @ankitmashu in #567

Full Changelog: v5.5.0...v5.6.0

v5.5.0

Version Summary

Version Number : v5.5.0 [tag: v5.5.0]
Date: 03-April-2024

Projects / Repos Included in this release: IUDX Resource Server, IUDX Deployment and installation, IUDX's Documentation

Release Summary :

India Urban Data Exchange (IUDX) | Resource Server v5.5.0 is released with enhanced features.

Highlights:

• Streaming Data Monitoring:
  • Resource Server will update the data consumed via "streaming"
  • Consumers can now refresh their "active" subscription using APIs
• End to End encryption
• Updated Software Client libraries
• Updated Auditing Flow
• Updated Software Testing
  • Updated Postman Collection
  • Included REST Assured based integration testing
  • Improved test coverage
• Software License:
  • With this release, all IUDX Softwares will be distributed under Apache 2.0 License

Test Reports:

Please find the release test details and reports here

API Docs

The api docs can be found here.

How to use the Release:

External Dependencies Installation

The Resource Server connects with various external dependencies namely

• ELK stack
• PostgreSQL
• RabbitMQ
• Redis
• AWS S3

Once the above setup is ready as mentioned in the deployment, you can install the IUDX Resource Server.

Docker based Installation

1. Install docker and docker-compose
2. Clone this repo
3. Build the images
   ./docker/build.sh
4. Modify the docker-compose.yml file to map the config file you just created
5. Start the server in production (prod) or development (dev) mode using docker-compose
   docker-compose up prod

Maven based Installation

1. Install java 11 and maven
2. Use the maven exec plugin based starter to start the server
   mvn clean compile exec:java@resource-server

JAR based Installation

1. Install java 11 and maven
2. Set Environment variables

export RS_URL=https://<rs-domain-name>
export LOG_LEVEL=INFO

3. Use maven to package the application as a JAR
   mvn clean package -Dmaven.test.skip=true
4. 2 JAR files would be generated in the target/ directory
   • iudx.resource.server-cluster-0.0.1-SNAPSHOT-fat.jar - clustered vert.x containing micrometer metrics
   • iudx.resource.server-dev-0.0.1-SNAPSHOT-fat.jar - non-clustered vert.x and does not contain micrometer metrics

Running the clustered JAR

Note: The clustered JAR requires Zookeeper to be installed. Refer here to learn more about how to set up Zookeeper. Additionally, the zookeepers key in the config being used needs to be updated with the IP address/domain of the system running Zookeeper.

The JAR requires 3 runtime arguments when running:

• --config/-c : path to the config file
• --hostname/-i : the hostname for clustering
• --modules/-m : comma separated list of module names to deploy

e.g. java -jar target/iudx.resource.server-cluster-0.0.1-SNAPSHOT-fat.jar --host $(hostname) -c configs/config.json -m iudx.resource.server.database.archives.DatabaseVerticle,iudx.resource.server.authenticator.AuthenticationVerticle ,iudx.resource.server.metering.MeteringVerticle,iudx.resource.server.database.postgres.PostgresVerticle

Use the --help/-h argument for more information. You may additionally append an RS_JAVA_OPTS environment variable containing any Java options to pass to the application.

e.g.

$ export RS_JAVA_OPTS="-Xmx4096m"
$ java $RS_JAVA_OPTS -jar target/iudx.resource.server-cluster-0.0.1-SNAPSHOT-fat.jar ...

Running the non-clustered JAR

The JAR requires 1 runtime argument when running:

• --config/-c : path to the config file

e.g. java -Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.Log4j2LogDelegateFactory -jar target/iudx.resource.server-dev-0.0.1-SNAPSHOT-fat.jar -c configs/config.json

Use the --help/-h argument for more information. You may additionally append an RS_JAVA_OPTS environment variable containing any Java options to pass to the application.

e.g.

$ export RS_JAVA_OPTS="-Xmx1024m"
$ java $RS_JAVA_OPTS -jar target/iudx.resource.server-dev-0.0.1-SNAPSHOT-fat.jar ...

Enabling Data Encryption in transit

All the count and search APIs have a feature to get encrypted data.
To get the data in encrypted format, the user could provide a publicKey in the header, with the value that is generated from lazySodium sealed box.
The header value should be in url-safe base64 format.
The encrypted data could be decrypted using the lazysodium sealed box by supplying the private and public key.

License

With this release, all IUDX Softwares will be distributed under Apache 2.0 License

Issues and Suggestions

Any Release related issues/suggestions, please raise here
For all other issues/suggestions related specific projects, please raise them to the respective project repo/issues.

What's Changed

• modify append subscription & subscription auditing flow by @ananjaykumar2 in #456
• auditing consent log by @ankitmashu in #463
• fix: attribute query logic, fix: temporal params for async by @pranavrd in #464
• Jenkins CI image tag update by @karun-singh in #459
• Create DMP Audit table by @shreelakshmijoshi in #469
• Schema/async request details by @ananjaykumar2 in #471
• SDM_code_change by @ankitmashu in #467
• Add Rest-Assured integration tests for Spatial Search APIs in the Resource Server along with the Rest Assure config file. by @Kranthi-Guribilli in #461
• Rest Assured Tests for Resource Server by @DivyaSreeMunagavalasa in #474
• Alterschema by @ankitmashu in #475
• Update delete subscription script to fetch the actual exchange from db by @Kranthi-Guribilli in #476
• Resolve issue while appending subscritptions when there is only 1 bin… by @Kranthi-Guribilli in #477
• Software Updates and Update RestAssured testHost on depl by fetching from config by @DivyaSreeMunagavalasa in #478
• License information update by @sushanthakumar in #480
• Solved bug and corrected pmd and checkstyle by @ankitmashu in #481

New Contributors

• @Kranthi-Guribilli made their first contribution in #461
• @sushanthakumar made their first contribution in #480

Full Changelog: v5.0.0...v5.5.0

Resource Server v5.0.0

New Features

• End to end data encryption extension to APIs to ensure data integrity and trust
  Multi-attribute support for search queries to support on-the-fly filtration using APIs
• Introduced new auditing and summary APIs to enhance the User Interface
• Added support for multiple file formats such as json and csv for file download from async API

Enhancement

• Updated the ID structure to Universally Unique IDentifiers (UUIDs) for improved security.
• Improved and enhanced auditing service workflow for better QOS
• Multi-Tenant based backend

Testing

• Improved Unit, Integration, Performance, and Security testing
  -Enhanced the testing framework with CI/CD Pipeline for automated testing

Resource Server v4.5.0

New Features

• End to end data encryption extension to APIs to ensure data integrity and trust
• Multi-attribute support for search queries to support on-the-fly filtration using APIs
• Introduced new auditing and summary APIs to enhance the User Interface

Enhancement

• Improved and enhanced auditing service workflow for better QOS

Testing

• Improved Unit, Integration, Performance, and Security testing
• Enhanced the testing framework with CICD Pipeline for automated testing

Resource Server v4.0.0

New Features

• Asynchronous API: Introduced an asynchronous API service which can be used by consumers to search and download data which is beyond the predefined limits of IUDX search APIs
  • /async/entities: Used to search the data which is beyond the /entities search limits
  • /async/status: Used to understand the status of the request using the searchID
• Resource Usage Limiting: This feature allows providers to define access limits for a given consumer. The resource usage is limited based on the number of API calls made on a resource.
• Enhanced HTTP Response for large queries: To inform users about queries exceeding retrieval limits (> 50k records) via search APIs, the resource server can now respond with a new URN based status code for HTTP 400 status.

Test Reports

• Unit tests available here
• Coverage available here
• Performance tests available here
• Security tests available here
• Integration tests available here

Resource Server v3.5.0

New Features

• Audit / Metering APIs for consumers, providers and delegates
  • /consumer/audit: This API shall allow IUDX consumer to understand their resource usage
  • /provider/audit: This API shall allow IUDX Providers to understand their resource usage
• Admin API for resource management
  • Admin APIs for updating item-id and unique attribute of a resource and updating token invalidation
• Broadcast Integration with RabbitMQ
• Resource Server integration with RabbitMQ:
  • to broadcast invalid token information provided by Authorization Server
  • to broadcast Unique Attributes Information published by IUDX Admin
• Invalidation of streaming subscription
• With release v3.5.0, a consumer should use refresh subscription API (at least once in 12 hours) to keep their subscription live. The streaming will be cancelled automatically by the resource server if the subscription is not refreshed

Updates, bugs, and performance fixes

• Enhanced Validation

Test Reports

• Unit tests available here
• Coverage available here
• Performance tests available here
• Security tests available here
• Integration tests available here

What's Changed

• Redis docker tuning by @abhi4578 in #126
• patch/cleanup/refactoring by @kailash in #127
• Updated postman collection by @swaminathanvasanth in #128
• Updated postman collection by @swaminathanvasanth in #129
• Adding small patch to docker push and dockerfiles by @abhi4578 in #130
• subscription validation by @kailash in #131
• Remove HTTP_PATH label by @abhi4578 in #133
• configurable broker params by @kailash in #134
• latest API changes by @kailash in #135
• latest API changes by @kailash in #136
• bug fix by @kailash in #137
• Metering coverage by @gopal-mahajan in #152
• immudb connection issue by @kailash in #155
• Adding contact info open api docs by @abhi4578 in #156
• catalogue URN changes by @kailash in #157
• Add jwtIgnoreExpiry option in config by @abhi4578 in #158
• Refactoring/coverage by @kailash in #153
• issue #160, #161 and #162 by @kailash in #162
• Feature/1.1.2 token invalidation by @kailash in #164
• compilation bug/ workflow by @kailash in #168
• Audit Api Created. Closes #1.2 by @gopal-mahajan in #167
• Feature/1.1.1 unique attribs by @kailash in #170
• Bump log4j-core from 2.14.1 to 2.15.0 by @dependabot in #174
• Provider Api. by @gopal-mahajan in #173
• Bump log4j-core from 2.15.0 to 2.16.0 by @dependabot in #176
• token invalidation flow changes by @kailash in #177
• Bump log4j-core from 2.16.0 to 2.17.0 by @dependabot in #178
• refresh sub flow by @kailash in #175
• Added pagination fix for POST query by @swaminathanvasanth in #179
• Bug/1.17 ingestion & sub API by @kailash in #181
• pagination only through request parameters by @kailash in #183
• Bump log4j-core from 2.17.0 to 2.17.1 by @dependabot in #185
• Reset password response by @pranavrd in #186
• Feature/cache verticle by @kailash in #187
• Integration with Jenkins CI pipeline by @karun-singh in #188
• Postman collection by @swaminathanvasanth in #189
• Bump postgresql from 42.2.22 to 42.2.25 by @dependabot in #190
• Jenkins CI improvements by @karun-singh in #191
• Updated git badge references by @karun-singh in #192
• revoke endpoint typo by @kailash in #193
• API typo in auth handler by @kailash in #194
• Bump postgresql from 42.2.25 to 42.3.3 by @dependabot in #196
• open resource id check for sub & ingestion by @kailash in #198
• script for subscriptions by @gopal-mahajan in #199
• Async APIs by @pranavrd in #197
• flyway for RS by @kailash in #201
• api docs improvements and additions by @kailash in #206
• Jenkins CI update: Added cleanup of unit test reports after stage completion by @karun-singh in #205
• Fix: path for async by @pranavrd in #207
• Added CLI command to deploy selected services/modules in production by @pranavrd in #208
• non-root image for resource server + change docker-compose file for temp storage + make apiserver port configurable by @hackcoderr in #204
• Fix: logging dependencies for slf4j by @pranavrd in #209
• vhost/access changes changes by @kailash in #210
• Adding Setup.md by @pranavrd in #211
• Fix: databroker configs by @pranavrd in #212

New Contributors

• @dependabot made their first contribution in #174
• @pranavrd made their first contribution in #186
• @karun-singh made their first contribution in #188
• @hackcoderr made their first contribution in #204

Full Changelog: v3.0.0...v3.5.0

Resource Server v3.0.0

New features

• New APIs
  • Complex temporal and spatial search: /temporal/entityPostQuery
  • Data Ingestion: /ingestion
  • Reset User Credentials: /management/user/resetPassword
• Paginated access to search APIs
  • Pagination based on optional query parameters (offset, limit)
• Metering and Auditing Service
  • Integrated with immutable database (ImmuDB)
• Updated Redis client flow for Latest Ingestion Pipeline
• URN based response formats
• Integration with JWT based tokens provided by IUDX Auth Server
• All APIs will not require a token to access data

Updates, bugs, and performance fixes

• Library, Security Updates, and migration from v3.9.1 to 4.0.3
• Enhanced Validation and cache performance
• Fully complied with IS 18003 (Part 2):2021 Doc No: LITD 28 (17249) | Unified Data Exchange Part 2: API specifications

Test Reports

• Stress and Endurance Test Report is available here
• Load and Performance Test Report is available here
• Unit Test Report is available here
• Integration Test Report is available here
• Penetration Test Report is available here

Resource Server v2.5.0

Features

• Resource Group Latest Data Access APIs
• Secure data access operations on a group level
• REDIS based latest data cache implementation
• Enhanced Unit and Integration testing
• Enhanced Input parameter validations

Alpha release

Resource Server Alpha Release.
This release is to test completeness before the actual release.

Some of the highlights of the alpha release are -

• Provides data access from available resources using standard APIs, streaming subscriptions (AMQP) and/or callbacks
• Search and count APIs for searching through available data: Support for Spatial (Circle, Polygon, Bbox, Linestring), Temporal (Before, during, After) and Attribute searches
• Adaptor registration endpoints and streaming endpoints for data ingestion
• Integration with the authorization server (token introspection) to serve private data as per the access control policies set by the provider
• Secure data access over TLS
• Scalable, service mesh architecture based implementation using open source components: Vert.X API framework, Elasticsearch/Logstash for database and RabbitMQ for data broker.
• Hazelcast and Zookeeper based cluster management and service discovery

Attached -

• Newman Integration Test Report - integration-test-report.html
• Surefire Unit Test Report - unit-test-report.html