Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add data lifecycly policy #216

Merged
merged 1 commit into from
Apr 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions docs/concepts/data_lifecycle_policy.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
.. _data-lifecycle-policy:

Data Lifecycle Policy
*********************

What it is
##########

Aspects is a data pipeline that captures, transforms, and aggregates tracking logs from the Open edX platform into xAPI statements and stores them in a ClickHouse database.
However, the data is not stored indefinitely by default. The data is keep for 1 year by default, but this can be adjusted by the site operator via the setting `ASPECTS_DATA_TTL_EXPRESSION` in the tutor plugin.

The setting value is a ClickHouse expression that defines the time-to-live policy (TTL) for the data. The expression is evaluated for each row in the table and should return a date. Rows with a date in the past are deleted. You can read more about the TTL policy in the ClickHouse documentation: https://clickhouse.tech/docs/en/engines/table-engines/mergetree-family/mergetree/#ttl

The data is partioned by month this way the TLL policy is applied per partition. Make sure to set the TTL policy to a date that is compatible with the partitioning policy. e.g. `ASPECTS_DATA_TTL_EXPRESSION: toDateTime(emission_time) + INTERVAL 2 MONTH` or `ASPECTS_DATA_TTL_EXPRESSION: toDateTime(emission_time) + INTERVAL 2 YEAR`.
1 change: 1 addition & 0 deletions docs/concepts/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ Concepts
xAPI <xapi_concepts>
Tracking Logs <tracking_logs>
Clickhouse <clickhouse>
Data Lifecycle Policy <data_lifecycle_policy>
dbt <dbt>
Ralph <ralph>
Vector <vector>
Expand Down