Skip to content

Commit

Permalink
master
Browse files Browse the repository at this point in the history
  • Loading branch information
@ncovercash
ncovercash committed Oct 30, 2024
1 parent 29e9791 commit 695c49f
Show file tree
Hide file tree
Showing 174 changed files with 55,650 additions and 16,415 deletions.
Binary file added .DS_Store
View file
Binary file not shown.
1 change: 1 addition & 0 deletions NEWS.md
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
# 2.0.x
- [MODFQMMGR-523](https://folio-org.atlassian.net/browse/MODFQMMGR-523) Upgrade `holdings-storage` to 8.0

## 2.0.0
- [MODFQMMGR-87](https://folio-org.atlassian.net/browse/MODFQMMGR-87) Move FQM-specific code from lib-fqm-query-processor
Expand Down
3 changes: 3 additions & 0 deletions PULL_REQUEST_TEMPLATE.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,6 @@ If you are adding entity type(s), have you:
- [ ] Added views to liquibase, as applicable?
- [ ] Added required interfaces to the module descriptor?
- [ ] Checked that querying fields works correctly and all SQL is valid?

If you are changing/removing entity type(s), have you:
- [ ] Added migration code for any changes?
60 changes: 44 additions & 16 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,22 +37,31 @@ By utilizing the combination of source views, computed views, and other relevant
mvn clean install
```
## Environment Variables
| Name | Default Value | Description |
|---------------------------------------------------|---------------|---------------------------------------|
| DB_HOST | localhost | Postgres hostname |
| DB_PORT | 5432 | Postgres port |
| DB_HOST_READER | localhost | Postgres hostname |
| DB_PORT_READER | 5432 | Postgres port |
| DB_USERNAME | postgres | Postgres username |
| DB_PASSWORD | postgres | Postgres password |
| DB_DATABASE | postgres | Postgres database name |
| MAX_QUERY_SIZE | 1250000 | max result count per query |
| mod-fqm-manager.permissions-cache-timeout-seconds | 60 | Cache duration for user permissions |
| server.port | 8081 | Server port |
| QUERY_RETENTION_DURATION | 3h | Older queries get deleted |
| task.execution.pool.core-size | 9 | Core number of concurrent async tasks |
| task.execution.pool.max-size | 10 | Max number of concurrent async tasks |
| task.execution.pool.queue-capacity | 1000 | Size of the task queue |
| Name | Default Value | Description |
|---------------------------------------------------|---------------|---------------------------------------------------|
| DB_HOST | localhost | Postgres hostname |
| DB_PORT | 5432 | Postgres port |
| DB_HOST_READER | localhost | Postgres hostname |
| DB_PORT_READER | 5432 | Postgres port |
| DB_USERNAME | postgres | Postgres username |
| DB_PASSWORD | postgres | Postgres password |
| DB_DATABASE | postgres | Postgres database name |
| IS_EUREKA | false | Specifies if environment is configured for EUREKA |
| MAX_QUERY_SIZE | 1250000 | max result count per query |
| mod-fqm-manager.permissions-cache-timeout-seconds | 60 | Cache duration for user permissions |
| mod-fqm-manager.entity-type-cache-timeout-seconds | 3600 | Cache duration for entity type definitions |
| server.port | 8081 | Server port |
| QUERY_RETENTION_DURATION | 3h | Older queries get deleted |
| task.execution.pool.core-size | 9 | Core number of concurrent async tasks |
| task.execution.pool.max-size | 10 | Max number of concurrent async tasks |
| task.execution.pool.queue-capacity | 1000 | Size of the task queue |

> **Note regarding `mod-fqm-manager.entity-type-cache-timeout-seconds`:** The default value defined in the project's
> module descriptor is `300`, as this value is more appropriate for development environments while still being reasonable
> for production. In general, production environments will see more benefit with higher values, since entity type
> definitions are very static and normally only change with module upgrades. `86400` (1 day) or `604800` (1 week) would
> be entirely reasonable for production. In the event that an entity type was updated in the database and this must be
> reflected quickly when the cache timeout is very high, simply restarting the module should be sufficient.
### Resource requirements

Expand All @@ -63,6 +72,15 @@ to mod-fqm-manager to ensure optimal performance and query handling. For larger
used, allocate at least 2 gigabytes. In extreme cases with extremely large responses and concurrent usage, 5 gigabytes of
heap space should be sufficient to maintain performance and stability.

### Local development only environment variables

> [!WARNING]
>
> These environment variables are intended for local development only and should not be used in production environments!
| Name | Default Value | Description |
|------------------------------------|---------------|-------------------------------|
| mod-fqm-manager.bypass-permissions | false | Disable all permission checks |

## Installing the module
Follow the guide of Deploying Modules sections of the [Okapi Guide](https://github.com/folio-org/okapi/blob/master/doc/guide.md#example-1-deploying-and-using-a-simple-module) and Reference, which describe the process in detail.
Expand Down Expand Up @@ -102,6 +120,13 @@ Then we need to enable the module for the tenant:
http://localhost:9130/_/proxy/tenants/<tenant_name>/modules
```

> **Note regarding ECS environments:** In ECS environments, FQM needs the central tenant ID when it is enabled, so FQM
> requests this data from mod-users. However, in new ECS environments, modules are enabled for a tenant before the tenant
> is added to a consortium, so mod-users is not able to provide this data at enable-time. As a result, in this case, it is
> necessary to provide the central tenant ID as a parameter, named `centralTenantId`, in the `POST` request to the tenant
> management API when enabling mod-fqm-manager. After the tenant has been added to the consortium, this parameter is no
> longer required when enabling mod-fqm-manager (e.g., when updating the module).

## Querying FQM - Step-by-step guide

Expand Down Expand Up @@ -438,6 +463,9 @@ Fields prefixed with an underscore are reserved for use by FQM.
#### _deleted
The field name "_deleted" can be used to indicate that the record corresponding to a particular content ID has been deleted.

#### _version
The name "_version" can be used to indicate the version number of the query.

## Additional information

### Issue tracker
Expand Down
146 changes: 115 additions & 31 deletions descriptors/ModuleDescriptor-template.json
View file
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@
{
"methods": ["POST"],
"pathPattern": "/_/tenant",
"permissionsRequired": []
"permissionsRequired": [],
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get","permissions.users.item.get"]
},
{
"methods": ["GET", "DELETE"],
Expand All @@ -21,77 +22,90 @@
},
{
"id": "entity-types",
"version": "1.0",
"version": "2.0",
"handlers": [
{
"methods": ["GET"],
"pathPattern": "/entity-types/{entity-type-id}",
"permissionsRequired": ["fqm.entityTypes.item.get"],
"modulePermissions": ["perms.users.get"]
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get", "permissions.users.item.get"]
},
{
"methods": ["GET"],
"pathPattern": "/entity-types",
"permissionsRequired": ["fqm.entityTypes.collection.get"],
"modulePermissions": ["perms.users.get"]
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get", "permissions.users.item.get"]
},
{
"methods": ["GET"],
"pathPattern": "/entity-types/{entity-type-id}/columns/{column-name}/values",
"permissionsRequired": ["fqm.entityTypes.item.columnValues.get"],
"modulePermissions": ["perms.users.get"]
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get", "permissions.users.item.get","search.facets.collection.get", "configuration.entries.collection.get"]
},
{
"methods": ["POST"],
"pathPattern": "/entity-types/materialized-views/refresh",
"permissionsRequired": ["fqm.materializedViews.post"]
"permissionsRequired": ["fqm.materializedViews.post"],
"modulePermissions": ["configuration.entries.collection.get", "finance.exchange-rate.item.get"]
}
]
},
{
"id": "fqm-query",
"version": "1.0",
"version": "2.0",
"handlers": [
{
"methods": ["GET"],
"pathPattern": "/query",
"permissionsRequired": ["fqm.query.sync.get"],
"modulePermissions": ["perms.users.get"]
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get", "permissions.users.item.get"]
},
{
"methods": ["GET"],
"pathPattern": "/query/{query-id}",
"permissionsRequired": ["fqm.query.async.results.get"],
"modulePermissions": ["perms.users.get"]
"permissionsRequired": ["fqm.query.async.results.query.get"],
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get","permissions.users.item.get"]
},
{
"methods": ["GET"],
"pathPattern": "/query/{query-id}/sortedIds",
"permissionsRequired": ["fqm.query.async.results.get"],
"modulePermissions": ["perms.users.get"]
"permissionsRequired": ["fqm.query.async.results.sortedids.get"],
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get","permissions.users.item.get"]
},
{
"methods": ["GET"],
"pathPattern": "/fqm/version",
"permissionsRequired": ["fqm.version.get"],
"modulePermissions": ["perms.users.get", "permissions.users.item.get"]
},
{
"methods": ["POST"],
"pathPattern": "/fqm/migrate",
"permissionsRequired": ["fqm.migrate.post"],
"modulePermissions": ["perms.users.get", "permissions.users.item.get"]
},
{
"methods": ["POST"],
"pathPattern": "/query/contents",
"permissionsRequired": ["fqm.query.async.results.get"],
"modulePermissions": ["perms.users.get"]
"permissionsRequired": ["fqm.query.async.results.post"],
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get","permissions.users.item.get"]
},
{
"methods": ["POST"],
"pathPattern": "/query",
"permissionsRequired": ["fqm.query.async.post"],
"modulePermissions": ["perms.users.get"]
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get", "permissions.users.item.get"]
},
{
"methods": ["POST"],
"pathPattern": "/query/purge",
"permissionsRequired": ["fqm.query.purge"]
"permissionsRequired": ["fqm.query.purge.post"]
},
{
"methods": ["DELETE"],
"pathPattern": "/query/{query-id}",
"permissionsRequired": ["fqm.query.async.delete"],
"modulePermissions": ["perms.users.get"]
"modulePermissions": ["perms.users.get", "user-tenants.collection.get", "consortia.user-tenants.collection.get", "permissions.users.item.get"]
}
]
},
Expand All @@ -106,7 +120,7 @@
"unit": "hour",
"delay": "1",
"modulePermissions": [
"fqm.query.purge"
"fqm.query.purge.post"
]
},
{
Expand All @@ -123,7 +137,8 @@
],
"permissionSets" : [
{
"permissionName": "fqm.query.purge",
"permissionName": "fqm.query.purge.post",
"replaces": ["fqm.query.purge"],
"displayName": "FQM - purge old queries",
"description": "Purge old queries",
"visible": true
Expand Down Expand Up @@ -156,6 +171,29 @@
"permissionName": "fqm.query.async.results.get",
"displayName": "FQM - Retrieve results of a submitted query",
"description": "Retrieve results of a submitted query",
"visible": true,
"subPermissions": [
"fqm.query.async.results.post",
"fqm.query.async.results.sortedids.get",
"fqm.query.async.results.query.get"
]
},
{
"permissionName": "fqm.query.async.results.sortedids.get",
"displayName": "FQM - Retrieve results of a submitted query with sorted Ids ",
"description": "Retrieve results of a submitted query with sorted Ids",
"visible": true
},
{
"permissionName": "fqm.query.async.results.query.get",
"displayName": "FQM - Retrieve results of a submitted query",
"description": "Retrieve results of a submitted query",
"visible": true
},
{
"permissionName": "fqm.query.async.results.post",
"displayName": "FQM - Retrieve results of a submitted query",
"description": "Retrieve results of a submitted query",
"visible": true
},
{
Expand All @@ -170,15 +208,23 @@
"description": "Run a query synchronously and get results",
"visible": true
},
{
"permissionName": "fqm.version.get",
"displayName": "FQM - Get version of fqm",
"description": "Get the version of fqm",
"visible": true
},
{
"permissionName": "fqm.migrate.post",
"displayName": "FQM - Submit FQM Migration",
"description": "Submit FQM Migration and get results",
"visible": true
},
{
"permissionName": "fqm.materializedViews.post",
"displayName": "FQM - Refresh materialized views",
"description": "Refresh FQM materialized views",
"visible": true,
"subPermissions": [
"configuration.entries.collection.get",
"finance.exchange-rate.item.get"
]
"visible": true
},
{
"permissionName": "fqm.query.all",
Expand All @@ -191,7 +237,9 @@
"fqm.query.async.post",
"fqm.query.async.results.get",
"fqm.query.async.delete",
"fqm.query.sync.get"
"fqm.query.sync.get",
"fqm.version.get",
"fqm.migrate.post"
],
"visible": true
}
Expand All @@ -203,11 +251,19 @@
},
{
"id": "holdings-storage",
"version": "6.0 7.0"
"version": "6.0 7.0 8.0"
},
{
"id": "instance-storage",
"version": "10.0"
"version": "10.0 11.0"
},
{
"id": "instance-date-types",
"version": "1.0"
},
{
"id": "instance-types",
"version": "2.0"
},
{
"id": "loan-types",
Expand All @@ -217,10 +273,6 @@
"id": "material-types",
"version": "2.2"
},
{
"id": "shelf-locations",
"version": "1.1"
},
{
"id": "location-units",
"version": "2.0"
Expand All @@ -245,6 +297,10 @@
"id": "users",
"version": "16.0"
},
{
"id": "user-tenants",
"version": "1.0"
},
{
"id": "loan-policy-storage",
"version": "2.3"
Expand Down Expand Up @@ -316,6 +372,28 @@
{
"id": "organizations.organizations",
"version": "1.2"
},
{
"id": "voucher-storage.vouchers",
"version": "1.2"
},
{
"id": "voucher-storage.voucher-lines",
"version": "2.1"
},
{
"id": "voucher-storage.voucher-number",
"version": "1.1"
},
{
"id": "search",
"version": "1.3"
}
],
"optional": [
{
"id": "consortia",
"version": "1.0"
}
],
"launchDescriptor": {
Expand Down Expand Up @@ -366,6 +444,12 @@
},
{ "name": "MAX_QUERY_SIZE",
"value": "1250000"
},
{ "name": "mod.fqm-manager.entity-type-cache-timeout-seconds",
"value": "300"
},
{ "name": "IS_EUREKA",
"value": "false"
}
]
}
Expand Down
Loading

0 comments on commit 695c49f

Please sign in to comment.