360Giving Registry

This version of the 360Giving registry is an iteration based on the
original version https://github.com/ThreeSixtyGiving/registry-classic

Which contains the work by the following authors:
   Bibiana Cristofol Amat <[email protected]>
   David Kane <[email protected]>
   David Kane <[email protected]>
   Michael Wood <[email protected]>
   bibiana <[email protected]>
   Bibiana Cristòfol <[email protected]>
   Rob Redpath <[email protected]>
   Ben Webb <[email protected]>
   Rob Redpath <[email protected]>
   Ben Webb <[email protected]>
   Myles Lewando <[email protected]>
   Bibiana Cristofol Amat <[email protected]>
   Bibiana Cristòfol <[email protected]>
   David Kane <[email protected]>
   Rikki Guy <[email protected]>
   James (ODSC) <[email protected]>
   James <[email protected]>
   Katherine Duerden <[email protected]>
   kindly <[email protected]>
   morchickit <[email protected]>

.github/workflows/lint.yml

@@ -0,0 +1,24 @@
+name: Lint
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Setup python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+        architecture: x64
+    - uses: actions/cache@v1
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}-${{ hashFiles('**/requirements_dev.txt') }}
+
+    - run: pip install -r requirements.txt
+    - run: pip install -r requirements_dev.txt
+
+    - run: flake8
+    # To local srcs, we must specifically exclude src
+    - run: black --check --exclude "/(\.eggs|\.git|_cache|src)/" ./

.github/workflows/test.yml

@@ -0,0 +1,36 @@
+name: Test
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - name: Setup python
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.8
+        architecture: x64
+    - uses: actions/cache@v1
+      with:
+        path: ~/.cache/pip
+        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}-${{ hashFiles('**/requirements_dev.txt') }}
+
+    - run: pip install -r requirements.txt
+    - run: pip install -r requirements_dev.txt
+
+    - run: ./registry/manage.py makemigrations --dry-run --check
+    - run: ./registry/manage.py migrate
+
+    - env:
+        SALESFORCE_USERNAME: ${{ secrets.SALESFORCE_USERNAME }}
+        SALESFORCE_PASSWORD: ${{ secrets.SALESFORCE_PASSWORD }}
+        SALESFORCE_SECURITY_TOKEN: ${{ secrets.SALESFORCE_SECURITY_TOKEN }}
+        # GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # coveralls
+
+      run: ./registry/manage.py test -v 3 tests
+#- run: coverage run --source='./registry' ./registry/manage.py test -v 3 tests
+
+#    - env:
+ #       GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+  #    run: coveralls --service=github

.gitignore

@@ -0,0 +1,15 @@
+.DS_Store
+.idea/
+.pytest_cache/
+.ve/
+.vscode/
+*.sqlite3
+get-pip.py
+
+chromedriver/
+__pycache__/
+registry/__pycache__/
+tests/__pycache__/
+tests/samples/__pycache__/
+
+chromedriver_mac64.zip

.gitmodules

@@ -0,0 +1,3 @@
+[submodule "360-ds"]
+	path = 360-ds
+	url = https://github.com/ThreeSixtyGiving/360-ds.git

LICENCE.md

@@ -0,0 +1,28 @@
+360Giving Registry software
+===========================
+
+Note: This license doesn't apply to the 360Giving logo The 360Giving logo is copyright [360Giving](http://threesixtygiving.org) and may not be used without express permission.
+
+The look-and-feel of the registry is that of 360Giving, and care should be
+taken when running any public instance not to falsely give the impression,
+intentionally or otherwise, that the instance is operated by or related to
+360Giving.
+
+The 360Giving Registry is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License as published by
+the Free Software Foundation, either version 3 of the License, or
+(at your option) any later version.
+
+360Giving Registry software is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Affero General Public License
+for more details.
+
+You should have received a copy of the GNU Affero General Public License along
+with the 360Giving Registry software.  If not, see
+<http://www.gnu.org/licenses/>.
+
+Copyright 2024 360Giving.
+
+
+

README.md

@@ -0,0 +1,51 @@
+# 360Giving Registry
+
+![360Giving registry logo](registry/static/images/360-logos/360giving-registry.svg)
+
+The [360Giving Registry](http://data.threesixtygiving.org) lists files that use the [360Giving Standard](http://standard.threesixtygiving.org). The list of files is maintained by 360Giving, and this repo contains the source code for the website that obtains the listing from 360Giving (via the Salesforce API) and presents it as a website. The live website is updated once a day from this repo, unless manually triggered.
+
+To download all the files listed in the registy, the 360Giving [datagetter](https://github.com/ThreeSixtyGiving/datagetter) tool is available.
+
+## Install dependencies
+
+```
+$ python3 -m venv .ve
+$ source .ve/bin/activate
+$ pip install -r requirements_dev.txt
+```
+
+## Run
+
+Development server
+```
+$ ./registry/manage.py runserver
+```
+
+## Tests
+```
+$ ./registry/manage.py test tests
+```
+
+## Update requirements
+
+```
+$ pip install pip-tools
+$ pip-compile requirements.in
+$ pip-compile requirements_dev.in
+```
+
+
+## Styles - 360-ds
+
+The [360Giving design system](https://github.com/ThreeSixtyGiving/360-ds) is used for styling this project.
+
+To compile the theme, check out the submodule then:
+
+```
+cd 360-ds
+npm install
+npm run compile-sass
+cp ./build/360-ds/css/main.css ../registry/ui/static/css/
+```
+
+For more information on the design system see the [360-ds project](https://github.com/ThreeSixtyGiving/360-ds)

registry/api/apps.py

@@ -0,0 +1,5 @@
+from django.apps import AppConfig
+
+
+class ApiConfig(AppConfig):
+    name = "api"

registry/api/urls.py

@@ -0,0 +1,9 @@
+from django.urls import path
+
+import api.views
+
+app_name = "api"
+
+urlpatterns = [
+    path("", api.views.DataView.as_view(), name="data"),
+]

registry/api/views.py

@@ -0,0 +1,14 @@
+from django.http.response import JsonResponse
+from django.views import View
+
+from salesforce import salesforce
+
+
+class DataView(View):
+    def get(self, *args, **kwargs):
+        return JsonResponse(salesforce.get_salesforce_data(), safe=False)
+
+
+class PublishersView(View):
+    def get(self, *args, **kwargs):
+        return JsonResponse(salesforce.get_salesforce_publishers(), safe=False)

registry/manage.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python
+"""Django's command-line utility for administrative tasks."""
+import os
+import sys
+
+
+def main():
+    """Run administrative tasks."""
+    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings.settings")
+    try:
+        from django.core.management import execute_from_command_line
+    except ImportError as exc:
+        raise ImportError(
+            "Couldn't import Django. Are you sure it's installed and "
+            "available on your PYTHONPATH environment variable? Did you "
+            "forget to activate a virtual environment?"
+        ) from exc
+    execute_from_command_line(sys.argv)
+
+
+if __name__ == "__main__":
+    main()

registry/salesforce/salesforce.py

@@ -0,0 +1,117 @@
+import os
+
+from simple_salesforce import Salesforce
+
+
+def get_salesforce_access():
+    return Salesforce(
+        username=os.environ["SALESFORCE_USERNAME"],
+        password=os.environ["SALESFORCE_PASSWORD"],
+        security_token=os.environ["SALESFORCE_SECURITY_TOKEN"],
+        domain=os.environ.get("SALESFORCE_DOMAIN"),
+    )
+
+
+def clean_object(obj):
+    new_obj = {}
+
+    obj.pop("attributes", None)
+    for key, value in obj.items():
+        new_key = key.replace("__c", "").replace("__r", "").replace("_", "")
+        new_key = new_key[:1].lower() + new_key[1:] if new_key else ""
+        new_obj[new_key] = value
+    return new_obj
+
+
+def get_output_dataset(dataset):
+    license = clean_object(dataset["license"])
+    publisher = clean_object(dataset["account"])
+
+    return {
+        "title": dataset["name"],
+        "description": (
+            dataset["description"] if dataset["description"] is not None else ""
+        ),
+        "identifier": dataset["id"],
+        "license": license["uRL"],
+        "license_name": license["name"],
+        "issued": dataset["dateFirstPublished"],
+        "modified": dataset["lastModifiedDate"],
+        "publisher": {
+            "name": publisher["name"],
+            "website": publisher["website"] if publisher["website"] else "",
+            "logo": publisher["logo"] if publisher["logo"] else "",
+            "prefix": publisher["prefix"],
+            "last_published": publisher["lastpublisheddate"],
+            "org_id": publisher.get("orgIdentifier", ""),
+        },
+        "distribution": [
+            {
+                "downloadURL": dataset["downloadURL"],
+                "accessURL": dataset["accessURL"],
+                "title": dataset["name"],
+            }
+        ],
+    }
+
+
+def clean_output(api_output):
+    datasets = []
+    for dataset in api_output["records"]:
+        if dataset["Approved__c"]:
+            try:
+                clean_dataset = clean_object(dataset)
+                output_dataset = get_output_dataset(clean_dataset)
+                datasets.append(output_dataset)
+            # We have bad data
+            except Exception as e:
+                print(e)
+                pass
+
+    if len(dataset) == 0:
+        raise Exception("No data could be cleaned for output")
+
+    return datasets
+
+
+def get_salesforce_data():
+    salesforce = get_salesforce_access()
+    sf_query = (
+        "SELECT Id, Name, License__r.Name, License__r.URL__c, Access_URL__c, Description__c, Download_URL__c,"
+        "Account__r.Id, Account__r.Logo__c, Account__r.Name, Account__r.Website, Account__r.prefix__c, "
+        "Date_First_Published__c, LastModifiedDate, Approved__c, Account__r.Last_published_date__c, Account__r.Org_Identifier__c from Dataset__c ORDER BY Account__r.Name"
+    )  # noqa: E126
+
+    return clean_output(salesforce.query_all(sf_query))
+
+
+def get_salesforce_publishers():
+    salesforce = get_salesforce_access()
+    sf_query = (
+        "SELECT Id, Logo__c, Name, Website, prefix__c, "
+        "Org_Identifier__c, Authorised_Domain__c, Self_registration_enabled__c from Account WHERE prefix__c != null ORDER BY Name "
+    )  # noqa: E126
+
+    sf_data = salesforce.query_all(sf_query)
+    publishers = {}
+
+    for publisher in sf_data["records"]:
+        publisher = clean_object(clean_object(publisher))
+        publishers[publisher["prefix"]] = {
+            "id": publisher["id"],
+            "name": publisher["name"],
+            "website": publisher.get("website", ""),
+            "logo": publisher.get("logo", ""),
+            "prefix": publisher["prefix"],
+            "org_id": publisher.get("orgIdentifier", ""),
+            "self_publish": {
+                "enabled": publisher.get("selfregistrationenabled", False),
+                "authorised_domains": (
+                    [publisher["authorisedDomain"]]
+                    if publisher.get("authorisedDomain")
+                    else []
+                ),
+            },
+        }
+
+    return publishers