docs: AP-5783 add QuickSight user docs (#476)

* docs: add QuickSight user docs

* feat: add quicksight faqs

style: move external links to bottom of pages for easier markdown parsing and link reuse

* feat: add (first pass) best-practices and troubleshooting guides

style: renamed markdown files to reflect file order/hierarchy

* docs: add section for adding user to quicksight

docs: add review suggestions

docs: formatting / linting

* docs: add publishing and sharing info

source/data/curated-databases/data-documentation/index.html.md.erb

@@ -1,6 +1,6 @@
 ---
 title: Database Documentation
-weight: 110
+weight: 7
 last_reviewed_on: 2023-03-10
 review_in: 1 year
 show_expiry: true

source/documentation/tools/index.md

@@ -15,6 +15,10 @@ Allows you to browse the databases that are available on the Analytical Platform
 ### [GitHub](https://github.com/)
 Online hosting platform for git. Git is a distributed version control system that allows you to track changes in files, while GitHub hosts the Analytical Platform's code.
 
+## Data Visualisation
+### [QuickSight](quicksight)
+Business Intelligence (BI/MI) tool for visualising / dashboarding data.
+
 ## Data Ingest
 
 ### [Data Extractor](https://github.com/ministryofjustice/data-engineering-data-extractor)

source/documentation/tools/quicksight/01-index.md

@@ -0,0 +1,50 @@
+# Using QuickSight to visualise your data
+
+⚠️ This service is in beta ⚠️
+
+AWS QuickSight is a cloud-powered business intelligence (BI) tool that makes it easy to create and share interactive dashboards and visualisations from data available to the Analytical Platform.
+
+We’re still early in our implementation and we'd love to get some of you using QuickSight to [get your feedback] to help guide best practices for data visualization in the Ministry of Justice and make sure we can continue to improve the user experience.
+
+## Overview
+
+### What data is in QuickSight?
+
+If you're using QuickSight on the Analytical Platform, you can only use data published through [Create a Derived Table](/tools/create-a-derived-table).
+
+We currently have each of the [domains/business areas](/tools/create-a-derived-table/project-structure/#domains) within '[mojap-derived-tables]' on QuickSight.
+
+## [Getting Started](/tools/quicksight/getting-started.html)
+
+- [Prerequisites](/tools/quicksight/getting-started.html#prerequisites-to-using-quicksight-in-analytical-platform)
+- [Accessing QuickSight](/tools/quicksight/getting-started.html#accessing-quicksight)
+
+## [Working with QuickSight](/tools/quicksight/working-with-quicksight.html)
+
+- [Finding datasets](/tools/quicksight/working-with-quicksight.html#finding-datasets)
+- [Publishing dashboards](/tools/quicksight/working-with-quicksight.html#publishing-dashboards)
+- [Sharing dashboards](/tools/quicksight/working-with-quicksight.html#sharing-dashboards)
+
+## [FAQs](/tools/quicksight/faqs.html)
+
+## [Best Practices](/tools/quicksight/best-practices.html)
+
+## [Common Issues and Troubleshooting](/tools/quicksight/troubleshooting.html)
+
+## Resources
+
+To learn more about QuickSight, take a look at [the AWS documentation]. Some of the basics about working with QuickSight in this user guide, but you can also sign up and work through [QuickSight's own examples].
+
+AWS also have further resources:
+
+- [AWS resources]
+- [AWS example analysis] (_dataset requires file upload_)
+
+<!-- External links -->
+
+[get your feedback]: https://controlpanel.services.analytical-platform.service.justice.gov.uk/feedback/
+[mojap-derived-tables]: https://github.com/moj-analytical-services/create-a-derived-table/tree/main/mojap_derived_tables/models
+[the AWS documentation]: https://docs.aws.amazon.com/quicksight/latest/user/welcome.html
+[QuickSight's own examples]: https://docs.aws.amazon.com/quicksight/latest/user/quickstart-createanalysis.html
+[AWS resources]: https://AWS.amazon.com/quicksight/resources/
+[AWS example analysis]: https://docs.aws.amazon.com/quicksight/latest/user/example-analysis.html

source/documentation/tools/quicksight/02-getting-started.md

@@ -0,0 +1,53 @@
+# Getting Started
+
+## Prerequisites to using QuickSight in Analytical Platform
+
+- Permission to read [data available in QuickSight](/tools/quicksight/#what-data-is-in-quicksight)
+- QuickSight enabled for your user within Analytical Platform Control Panel
+- You have a QuickSight user linked to your `alpha_user_`
+
+### Permissions
+
+You'll need permissions to databases, tables, or domains for datasets within [the `mojap-derived-table` domains] produced by [Create a Derived Table](/tools/create-a-derived-table).
+
+These permissions are defined in [the data-engineering-database-access GitHub repository].
+
+### Ensuring you have a QuickSight user
+
+> **Note:** As you read through the instructions you will notice that you are being asked to authenticate and set up user groups using Azure Entra ID and your `@justice.gov.uk` identity. This is a deliberate choice. Our long-term strategic goal as a platform is to transition away from Github for access management and utilise MOJ’s central directory.
+
+#### 1. Create a User Group in Entra ID
+
+- Go to the [Azure Portal] and sign in with your `justice.gov.uk` credentials.
+- In the top search bar, search for **"Entra ID"** and click on **Microsoft Entra ID**.
+  - if this is the first time using the Azure Portal there are a few pages you can skip to get to this point.
+- Navigate to **Manage** \> **Groups** and click **\+ New group**.
+- Set **Group type** to **Security**.
+- Enter a **Group name** beginning with `azure-aws-sso-qs-` and a unique identifier (e.g., `azure-aws-sso-qs-yourteam`).
+- Choose **Membership type** as **Assigned**.
+- Click on **No members selected**, then search for and select all users who will participate in the MVP. Ensure all users have a `justice.gov.uk` email address.
+- To add yourself as an admin, click on the **Owners** section, search for your user account, and add it as an owner. This will give you administrative rights to manage group membership and settings.
+- Click **Create** to finalise the group setup, and notify the AP team.
+
+#### 2. Notify Us of the Group Name and Begin Synchronisation
+
+- Provide the group name to the AP team by posting in **[\#analytical-platform-quicksight-mvp]** on Slack. The AP team will synchronise the group and update QuickSight roles.
+
+### Enable QuickSight in Analytical Platform Control Panel
+
+[Raise a support request] to enable QuickSight for your user
+
+## Accessing QuickSight
+
+- Complete [prerequisites](#prerequisites-to-using-quicksight-in-analytical-platform)
+- Navigate to [QuickSight UI within Control Panel]
+
+<!-- External links -->
+
+[the `mojap-derived-table` domains]: https://github.com/moj-analytical-services/create-a-derived-table/tree/main/mojap_derived_tables/models
+[the data-engineering-database-access GitHub repository]: https://github.com/moj-analytical-services/data-engineering-database-access/?tab=readme-ov-file#access-to-curated-databases
+[Raise a support request]: https://github.com/ministryofjustice/data-platform-support/issues/new/choose
+[Azure portal]: https://portal.azure.com/
+[\#analytical-platform-quicksight-mvp]: https://moj.enterprise.slack.com/archives/C07LAJB86TZ
+[Analytical Platform]: https://analytical-platform.service.justice.gov.uk/
+[QuickSight UI within Control Panel]: https://controlpanel.services.analytical-platform.service.justice.gov.uk/quicksight/

source/documentation/tools/quicksight/03-working-with-quicksight.md

@@ -0,0 +1,59 @@
+# Working with QuickSight
+
+## Adding a dataset
+
+Datasets are available via the `Athena` data source in QuickSight:
+
+> Datasets > New Dataset (top right) > Athena > New Athena data source.
+
+When adding a new dataset via Athena you'll be prompted for a `Data source name`.
+This is just a descriptive name for the new data source tile for **you** in **your** QuickSight interface.
+
+Here's [the advice from AWS] about naming your Data source:
+
+> _This name displays on the Amazon QuickSight list of existing data sources, which is at the bottom of the 'Create a Dataset' view.
+> Use a name that makes it easy to distinguish your data sources from other similar data sources._
+
+Unless specified elsewhere for your data, use the default `Athena workgroup`.
+
+Data will be in the default Catalog (`AwsDataCatalog`), and you should only see Databases that your user has access to via the Database Access Repo.
+
+Adding a dataset in this way enables you to add a single table at a time.
+
+You'll be asked if you want to 'Import to [SPICE] for quicker analytics'.
+This imports the data into memory, rather than querying the data where it lives in S3. Whether to use SPICE or directly query the data is a judgement call, and will likely come down to:
+
+- how complex the analysis is
+- the size, format and structure of the underlying data
+- how likely users are to drill down into the dashboards
+- user tolerance for load times while making changes to dashboards
+
+## Publishing Dashboards
+
+- To create a dashboard, you'll need to publish it from an analysis. 
+[The AWS QuickSight documentation] has details around the publishing flow.
+
+## Sharing Dashboards
+
+>**⚠️ Important ⚠️**
+> 
+>Users who have access to the dashboard can also see the data used in the associated analysis.
+
+[Published dashboards](#publishing-dashboards) can be shared with anyone who already has a QuickSight user. 
+Dashoards can be shared as-is, or they can be shared on a schedule via email, as an attachment or a link to the dashboard.
+
+From [the AWS documentation on sharing dashboards]:
+
+"By default, dashboards in Amazon QuickSight aren't shared with anyone and are only accessible to the owner.
+However, after you publish a dashboard, you can share it with other users or groups in your QuickSight account. 
+You can also choose to share the dashboard with everyone in your QuickSight account and make the dashboard visible on the QuickSight homepage for all users in your account. 
+Additionally, you can copy a link to the dashboard to share with others who have access to it."
+
+<!-- External links -->
+
+[the advice from AWS]: https://docs.aws.amazon.com/quicksight/latest/user/create-a-data-source.html
+[SPICE]: https://docs.aws.amazon.com/quicksight/latest/user/managing-spice-capacity.html
+[AWS resources]: https://aws.amazon.com/quicksight/resources/
+[AWS example analysis]: https://docs.aws.amazon.com/quicksight/latest/user/example-analysis.html
+[The AWS QuickSight documentation]: https://docs.aws.amazon.com/quicksight/latest/user/creating-a-dashboard.html
+[the AWS documentation on sharing dashboards]: https://docs.aws.amazon.com/quicksight/latest/user/sharing-a-dashboard.html

source/documentation/tools/quicksight/best-practices.md

@@ -0,0 +1,36 @@
+# Best Practices
+
+<!-- relevant once data expands beyond CaDeT
+## Data Preparation
+- Clean and preprocess your data before importing it into QuickSight.
+- Use meaningful names for datasets and fields.
+-->
+
+## Dashboard Design
+
+- Keep dashboards simple and focused.
+- Use appropriate visualization types for your data.
+- Limit the number of visuals on a single dashboard to improve performance.
+
+## Performance Optimization
+
+- see [this issue in the troubleshooting guide](/tools/quicksight/troubleshooting.html#issue-slow-dashboard-performance).
+- Optimize your queries to reduce load times.
+- Load datasets into SPICE.
+
+## Collaboration
+
+- Share dashboards with relevant stakeholders.
+- Use descriptive names and annotations for visuals.
+
+## Security
+
+- Ensure proper permissions are set for datasets and dashboards.
+- Regularly review and update access controls.
+
+## Maintenance
+
+- Regularly update datasets and dashboards to reflect changes in data.
+- Monitor dashboard performance and make necessary adjustments.
+
+<!-- External links -->

source/documentation/tools/quicksight/faqs.md

@@ -0,0 +1,51 @@
+# Frequently Asked Questions
+
+## How do I know if I should be able to see a particular table/database/domain in QuickSight?
+
+Databases and Tables should be available via the `Athena` data source in QuickSight provided that:
+
+- your Analytical Platform `alpha_user` has access to the data via [the data-engineering-database-access GitHub repository]
+- the data is within one of [the `mojap-derived-table` domains] in Create a Derived Table
+
+## Can I share the dashboards that I create?
+
+See [Sharing dashboards](/tools/quicksight/working-with-quicksight.html#sharing-dashboards) for information on sharing published dashboards.
+
+Be aware that [users who have access to the dashboard can also see the data used in the associated analysis].
+
+## I’m already a user of another version of QuickSight on the AP, what should I do if:
+
+### I can do all of my work on the new QuickSight (all of my tables are create-a-derived-table tables)
+
+It is possible to export dashboards from other instances of QuickSight using [Asset Bundle export].
+If your previous dashboard is sufficiently complex that recreation in this QuickSight would be a significant issue, [raise a support request] and the Analytical Platform team may be able to help you migrate the dashboard.
+
+### I can’t do all of my work on the new QuickSight (some / all of my tables are outside create-a-derived-table)
+
+We will keep you updated as new data becomes available to QuickSight via the Control Panel.
+
+## When should I use SPICE?
+
+Importing a dataset into [SPICE] imports the data into memory, rather than querying the data where it lives in S3. Whether to use SPICE or directly query the data is a judgement call, and will likely come down to:
+
+- how complex the analysis is
+- the size, format and structure of the underlying data
+- how likely users are to drill down into the dashboards
+- user tolerance for load times while making changes to dashboards
+
+## What should I do if I encounter an error while using QuickSight?
+
+Check [the Troubleshooting guide](/tools/quicksight/troubleshooting), and [raise a support request] if you're unable to resolve your issue there.
+
+## How can I optimize the performance of my QuickSight dashboards?
+
+- see [this issue in the troubleshooting guide](/tools/quicksight/troubleshooting.html#issue-slow-dashboard-performance)
+
+<!-- External links -->
+
+[the `mojap-derived-table` domains]: https://github.com/moj-analytical-services/create-a-derived-table/tree/main/mojap_derived_tables/models
+[the data-engineering-database-access GitHub repository]: https://github.com/moj-analytical-services/data-engineering-database-access/?tab=readme-ov-file#access-to-curated-databases
+[users who have access to the dashboard can also see the data used in the associated analysis]: https://docs.aws.amazon.com/quicksight/latest/user/creating-a-dashboard.html
+[Asset Bundle export]: https://docs.aws.amazon.com/quicksight/latest/developerguide/assetbundle-export.html
+[SPICE]: https://docs.aws.amazon.com/quicksight/latest/user/managing-spice-capacity.html
+[raise a support request]: https://github.com/ministryofjustice/data-platform-support/issues/new/choose

source/documentation/tools/quicksight/troubleshooting.md

@@ -0,0 +1,29 @@
+# Common Issues and Troubleshooting
+
+### Issue: Unable to see a dataset in QuickSight
+
+- Ensure you have the necessary permissions to access the dataset.
+- Verify that the dataset is within the `mojap-derived-table` domains.
+- Check if the dataset is available via the `Athena` data source.
+
+See [this question in FAQs](/tools/quicksight/faqs.html#how-do-i-know-if-i-should-be-able-to-see-a-particular-tabledatabasedomain-in-quicksight) for more information
+
+### Issue: Slow dashboard performance
+
+- Optimize your queries
+  - Remove unnecessary data from queries
+  - Optimise joins and data structure
+- Check out [the QuickSight optimisation tips page] for specific tips
+- Load data into SPICE
+
+### Issue: Error while importing data
+
+- Check the data format and structure
+  - Can you read and query the data via the Athena console (in the Ireland region within the data account) or via `boto3`/`pydbtools`?
+- Ensure the data source is correctly configured in QuickSight (see [Finding datasets](/tools/quicksight/03-working-with-quicksight.html#finding-datasets))
+- [Raise a support request] providing details of the problem and what you've tried.
+
+<!-- External links -->
+
+[the QuickSight optimisation tips page]: https://aws.amazon.com/blogs/big-data/tips-and-tricks-for-high-performant-dashboards-in-amazon-quicksight/
+[Raise a support request]: https://github.com/ministryofjustice/data-platform-support/issues/new/choose

source/tools/control-panel.html.md.erb

@@ -1,6 +1,6 @@
 ---
 title: Control Panel
-weight: 10
+weight: 5
 last_reviewed_on: 2022-07-06
 review_in: 1 year
 show_expiry: true

source/tools/data-processing/index.html.md.erb

@@ -1,6 +1,6 @@
 ---
 title: Tools for Data Processing
-weight: 5
+weight: 8
 last_reviewed_on: 2024-07-11
 review_in: 1 year
 show_expiry: true

source/tools/quicksight/best-practices.html.md.erb

@@ -0,0 +1,11 @@
+---
+title: Best Practices
+weight: 30
+tags: quicksight
+last_reviewed_on: 2024-12-11
+review_in: 2 months
+owner_slack: "#analytical-platform-support"
+owner_slack_workspace: "mojdt"
+---
+
+<%= partial 'documentation/tools/quicksight/best-practices' %>

source/tools/quicksight/faqs.html.md.erb

@@ -0,0 +1,11 @@
+---
+title: FAQs
+weight: 25
+tags: quicksight
+last_reviewed_on: 2024-12-11
+review_in: 2 months
+owner_slack: "#analytical-platform-support"
+owner_slack_workspace: "mojdt"
+---
+
+<%= partial 'documentation/tools/quicksight/faqs' %>

source/tools/quicksight/getting-started.html.md.erb

@@ -0,0 +1,11 @@
+---
+title: Getting Started
+weight: 15
+tags: quicksight
+last_reviewed_on: 2024-12-11
+review_in: 2 months
+owner_slack: "#analytical-platform-support"
+owner_slack_workspace: "mojdt"
+---
+
+<%= partial 'documentation/tools/quicksight/02-getting-started' %>

source/tools/quicksight/index.html.md.erb

@@ -0,0 +1,11 @@
+---
+title: QuickSight
+weight: 6
+tags: quicksight
+last_reviewed_on: 2024-12-11
+review_in: 2 months
+owner_slack: "#analytical-platform-support"
+owner_slack_workspace: "mojdt"
+---
+
+<%= partial 'documentation/tools/quicksight/01-index' %>

source/tools/quicksight/troubleshooting.html.md.erb

@@ -0,0 +1,11 @@
+---
+title: Common Issues and Troubleshooting
+weight: 35
+tags: quicksight
+last_reviewed_on: 2024-12-11
+review_in: 2 months
+owner_slack: "#analytical-platform-support"
+owner_slack_workspace: "mojdt"
+---
+
+<%= partial 'documentation/tools/quicksight/troubleshooting' %>

source/tools/quicksight/working-with-quicksight.html.md.erb

@@ -0,0 +1,11 @@
+---
+title: Working with QuickSight
+weight: 20
+tags: quicksight
+last_reviewed_on: 2024-12-11
+review_in: 2 months
+owner_slack: "#analytical-platform-support"
+owner_slack_workspace: "mojdt"
+---
+
+<%= partial 'documentation/tools/quicksight/03-working-with-quicksight' %>