Skip to content

Commit

Permalink
Release qdes-1.1.0 (#31)
Browse files Browse the repository at this point in the history 
* [DDCI-110] Added "upsert" functionality for vocabulary_service_term records.

* [DDCI-107] Added access restrictions for vocab service functions.

* [DDCI-111] Added background process command.

* [DDCI-109] - vocab service edit delete

* [DDCI-109] - change wording

* Fix refresh script.

* Fix for context issue when fetching VocPrez vocab via click.

* Tidy up vocab service table display.

* [DDCI-125] - added CSV reader for vocab

* [DDCI-125] - removed unused vocab type

* Update `datae_modified` when updating vocab service term.

* Added `remote_csv` vocab service to automated refresh.

* Added link to terms to vocab service refresh message.

* Added info to vocab service terms detail page.

* Added CSV example.

* Create nature-of-relationship

* Create temporal-resolution-ranges.csv

* [DDCI-124] - added vocab auto complete

* Add files via upload

* [DDCI-41] PoC work.

* [DDCI-124] Refactoring `vocabulary_service_term_search`
- Adjusted search query to be case insensitive
- Refactored code for simplicity and readability

* [DDCI-213] - accepted duplicate terms

* [DDCI-41]
- Added new blueprint endpoint for 'secure_vocabulary_search'
- Added new action 'secure_vocabulary_search' to search for secure vocabs
- Updated config for 'point-of-contact'
- Added config values
- Added file upload validation
- Removed unused code

* - Added new secure vocab 'the-party'
- 'get_secure_vocabulary_record_label' will not return the query as the label if it cannot find the secure vocab

* Removed debug logs

* Delete key.key

* Delete test.csv

* [DDCI-213] Added migrations
- Added migration for `allow_duplicate_terms` column on `vocabulary_service` table.

* [DDCI-213] Renamed migration file.

* Added new column 'definition' to table 'vocabulary_service_term'

* Added definition to vocabulary service terms

* Added definition to action 'vocabulary_service_term_search'

* Reutrn default value of None if definition key does not exist for csv vocabs

* Fixed definition key lookup

* Added 'definition' to new property 'title' for 'scheming_vocabulary_service_choices'

* Tidy up vocabulary service list table.

* [DDCI-245] - added context to an action

* [DDCI-193] - added never option to refresh freq

* [DDCI-192] - added new cols for vocab

* [DDCI-192] - modified vocpress logic

* [DDCI-192] - added detection for broader uri changes

* [DDCI-169] - validation of invalid uri for vocab (#17)

* [DDCI-170] - added email notification for invalid vocab uri (#18)

* [DDCI-170] - added email notification for invalid vocab uri

* [DDCI-170] - updated email wording

* Refactored code to use toolkit.config

Co-authored-by: Mark Calvert <[email protected]>

* [DDCI-170] - fix refresh issue (#19)

* [DDCI-170] - updated link to use ckan helper (#20)

* Updated email templates to use vocab name instead of internal identifier

* Update email template text for 'vocab_service_invalid_urls'

* Added `application/vnd.ms-excel` to allowed mimetypes

* [DDCI-390] - updated the success message

* fix auth issue when looking up point of contact

* [DDCI-244] - added vocab support for render the tree

* Pinned `cryptography` requirement to 3.3.x

On Mon 8th Feb, 2021 - a new version of the `cryptography` package was released that caused deployment/build issues due to:
```
error: Can not find Rust compiler
```
Since cryptography is a future requirement, the package has been pinned to the last stable version.

* Added new action 'get_vocabulary_service_term'
Updated model methods 'get_by_label_or_uri' & 'get_by_uri' to be case insensitive

* [DDCI-74] - added alternative display for search

* [DDCI-482] Fixed `get_remote_csv_vocabulary_terms` action call.

* [DDCI-634] Change secure CSV title

* [DDCI-696] Fix for secure vocab search
- Allow sysadmins without any organisation roles to access `secure_vocabulary_search` get action
- Replaced use of `c.user` and `c.userobj` with `toolkit.g` equivalents

* [DDCI-660] - added octet when upload csv on windows

* [DDCI-685] - updated email wording

* Added job worker title for Vocabulary service update

* Updated 'vocabulary_service_term_table' column 'definition' to be nullable

* Add Email to the search fields (#29)

* Search on uri and term for vocab services (#15)

Added search for 'Email' in secure vocabs

Co-authored-by: Mark Calvert <[email protected]>

* [DDCI-260] Replace VocPrez with SPARQL+JSON (#30)

* Replace VocPrez with SPARQL+JSON

* Remove CSIRO service

* Change vocprez mentions to SPARQL_JSON

* Remove breakpoint

* Added check for environment variable 'DISABLE_CRON_JOB_EMAILS' to not send email notification

Co-authored-by: Nathan Perry <[email protected]>
Co-authored-by: awang setyawan <[email protected]>
Co-authored-by: tasjim <[email protected]>
Co-authored-by: Mark Calvert <[email protected]>
Co-authored-by: Konstantin Sivakov <[email protected]>
  • Loading branch information
@MarkCalvert @salsa-nathan
@awset @tasjim @tino097
6 people authored Jul 5, 2021
1 parent d744378 commit 7081e48
Show file tree
Hide file tree
Showing 33 changed files with 2,820 additions and 182 deletions.
22 changes: 22 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,25 @@ This will create the following two tables:
vocabulary_service_term

*(NOT to be confused with the CKAN core `vocabulary` table)*

## Background tasks

A cron job needs to be set up to refresh the vocabulary sevices periodically based on each `vocabulary_service.update_frequency` setting.

ckan -c path/to/ckan.ini vocabulary-services-refresh

## Secure Vocabularies

A secure vocabulary is an ecnrypted CSV that is not stored in the `vocabulary_service` or `vocabulary_service_term`
database tables.

To enable secure vocabularies, enable the `secure_vocabularies` plugin in CKAN `.ini` file, e.g.

ckan.plugins = ... secure_vocabularies ...

You must also specify a path to a secure vocabulary configuration file, i.e.

ckan.vocabulary_services.configuration_file = /etc/ckan/default/secure_vocabularies.json

... to be continued ...

161 changes: 119 additions & 42 deletions ckanext/vocabulary_services/blueprint.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,11 @@
import logging

from ckan.common import request
from ckanext.vocabulary_services import helpers
from ckanext.invalid_uris.helpers import valid_uri
from flask import Blueprint
from ckan.views.api import _finish_ok
from pprint import pformat

clean_dict = logic.clean_dict
get_action = toolkit.get_action
Expand All @@ -18,88 +22,161 @@


def index():
# @TODO: Restrict to sysadmins

helpers.check_access({})

try:
data = {}
errors = {}
is_update = False
vocab_service_id = request.args.get('id')
if request.method == 'POST':
data = clean_dict(dict_fns.unflatten(tuplize_dict(parse_params(
request.form))))
try:
get_action('vocabulary_service_create')({}, data)
if vocab_service_id:
get_action('vocabulary_service_edit')({}, data)
h.flash_success('Vocabulary service %s updated.' % data['title'])

return h.redirect_to('vocabulary_services.index')
else:
get_action('vocabulary_service_create')({}, data)
h.flash_success('Vocabulary service %s added.' % data['title'])

# Reset the form data.
data = {}
except toolkit.ValidationError as e:
log.warn(e)
errors = e.error_dict
log.debug(errors)

if vocab_service_id:
is_update = True
h.flash_error('Error updating vocabulary service %s.' % data['title'])
else:
h.flash_error('Error adding vocabulary service %s.' % data['title'])

elif request.method == 'GET' and vocab_service_id:
is_update = True
data = get_action('get_vocabulary_service')({}, vocab_service_id)

services = get_action('get_vocabulary_services')({}, {})

return toolkit.render('vocabulary/index.html',
extra_vars={
'data': data,
'errors': errors,
'services': services
'services': services,
'is_update': is_update
})
except Exception as e:
log.error(e)
toolkit.abort(503, str(e))


def refresh(id):
# @TODO: Restrict to sysadmins
service = get_action('get_vocabulary_service')({}, id)

# @TODO: Implement the fetching of the terms from the actual vocabulary endpoint
data = [
{'label': 'Blah', 'uri': 'https://www.google.com/blah'},
{'label': 'Wah', 'uri': 'https://www.google.com/wah'},
{'label': 'Haha', 'uri': 'https://www.google.com/haha'},
]
helpers.check_access({})

if service:
if service.type == 'csiro':
log.debug('>>> Attempting to fetch vocabulary from CSIRO service...')
if get_action('get_csiro_vocabulary_terms')({}, service):
log.debug('>>> Finished fetching vocabulary from CSIRO service.')
else:
log.error('>>> ERROR Attempting to fetch vocabulary from CSIRO service')
elif service.type == 'vocprez':
log.debug('>>> Attempting to fetch vocabulary from VocPrez service...')
if get_action('get_vocprez_vocabulary_terms')({}, service):
log.debug('>>> Finished fetching vocabulary from VocPrez service.')
else:
log.error('>>> ERROR Attempting to fetch vocabulary from VocPrez service')
else:
for d in data:
data_dict = {
'vocabulary_service_id': service.id,
'label': d['label'],
'uri': d['uri'],
}
# @TODO: Implement as an UPSERT
get_action('vocabulary_service_term_create')({}, data_dict)
service = get_action('get_vocabulary_service')({}, id)

# @TODO: Do we need to check for terms removed from the vocabulary?
# And do we need to check if the term is in user before deleting?
if service:

h.flash_success('Terms in vocabulary refreshed')
data_dict = {
'id': service.id,
'uri': service.uri,
}

# Validate uri.
valid_uri_resp = valid_uri(data_dict.get('uri'))
if not valid_uri_resp.get('valid'):
h.flash_error('Uri is not valid.')
return h.redirect_to('vocabulary_services.index')

action = None

if service.type == 'sparql_json':
action = 'get_sparql_json_vocabulary_terms'
elif service.type == 'remote_csv':
action = 'get_remote_csv_vocabulary_terms'

if action:
if get_action(action)({}, data_dict):
get_action('update_vocabulary_service_last_processed')({}, service.id)
h.flash_success(
'Terms in vocabulary refreshed. <a href="{}">View terms</a>'.format(h.url_for('vocabulary_services.terms', id=service.id)),
allow_html=True
)
else:
h.flash_error('Vocabulary service type %s not currently implemented.' % service.type)

return h.redirect_to('vocabulary_services.index')


def terms(id):
# @TODO: Restrict to sysadmins
terms = get_action('get_vocabulary_service_terms')({}, id)

helpers.check_access({})

return toolkit.render('vocabulary/terms.html',
extra_vars={
'terms': terms,
'vocabulary_service': get_action('get_vocabulary_service')({}, id),
'terms': get_action('get_vocabulary_service_terms')({}, id),
})

def delete(id):
"""
Delete vocabulary service.
"""
if request.method == 'POST':
try:
get_action('vocabulary_service_delete')({}, id)
h.flash_success('Vocabulary service deleted.')
except Exception as e:
log.error(e)
h.flash_error('Error deleting vocabulary service.')
else:
h.flash_error('Can not delete vocabulary service.')

vocabulary_services.add_url_rule(u'/vocabulary-services',
methods=[u'GET', u'POST'], view_func=index)
return h.redirect_to('vocabulary_services.index')

vocabulary_services.add_url_rule(u'/vocabulary-service/refresh/<id>', view_func=refresh)
def vocabulary_service_term_autocomplete(term_name):
q = request.args.get('incomplete', '')
limit = request.args.get('limit', 5)
search_dict = {'term_name': term_name, 'q': q, 'limit': limit}

if not q:
return _finish_ok({})

result = get_action('vocabulary_service_term_search')({}, search_dict)
if not result:
return _finish_ok({})

result_set = {'ResultSet': {u'Result': result}}

return _finish_ok(result_set)


def vocabulary_service_term_json(term_name):
result = get_action('get_vocabulary_service_terms')({}, term_name)
if not result:
return _finish_ok({})

res = []
for term in result:
dict = term.__dict__
dict.pop('_sa_instance_state')
dict.pop('date_created')
dict.pop('date_modified')
dict['title'] = dict.get('label')
dict['key'] = dict.get('uri')
res.append(dict)

return _finish_ok(helpers.render_hierarchical(res))


vocabulary_services.add_url_rule(u'/vocabulary-services', methods=[u'GET', u'POST'], view_func=index)
vocabulary_services.add_url_rule(u'/vocabulary-service/refresh/<id>', view_func=refresh)
vocabulary_services.add_url_rule(u'/vocabulary-service/terms/<id>', view_func=terms)
vocabulary_services.add_url_rule(u'/vocabulary-service/delete/<id>', methods=[u'POST'], view_func=delete)
vocabulary_services.add_url_rule(u'/vocabulary-service/term-autocomplete/<term_name>', view_func=vocabulary_service_term_autocomplete)
vocabulary_services.add_url_rule(u'/vocabulary-service/term-json/<term_name>', view_func=vocabulary_service_term_json)
Loading

0 comments on commit 7081e48

Please sign in to comment.