Add workflow to calculate schema changes

[amishas157]

PR Checklist

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with the jira ticket associated with the PR.

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated the README with the added features, breaking changes, new instructions on how to use the repository.

What

This PR creates a github workflow which:

• runs once a day
• queries for changes in marts schemas
• If any diff, create a branch and push the changes
• Makes a pull request to update changelog file. Sample PR (Merge update-data-schema-changelog-11582894903 into master #526)

Since the results from script will be deterministic, no new PRs will be created if no schema has changed

Why

To track data schema changes.

I will follow up with 2 PRs:

• Add support to list models added/deleted
• Make CI job to do diff for public datasets schema (from /schemas dir) and create changelog/source_data.md

Question

Does the format for changelog look alright? Happy to discuss other ideas

[chowbao]

Hmm is it possible to make this append instead of overwrite the entire changelog each run?

There might be a time we delete the elementary.alerts_schema_changes table like the times we have deleted the whole elementary dataset because it was broken or elementary was doing something unexpected.

If it's too hard to make this append instead of overwrite we should remember to keep a copy of elementary.alerts_schema_changes otherwise we'd lose changelog history

[amishas157]

hrm this process is pretty dependent on elementary.alerts_schema_changes. Even if we were to delete this dataset ever, that in itself will affect changelog computing part. So ideally we should not delete that dataset.

However as far as append is concerned, we could

• parse tracked date if any
• add date filter to get data between tracked date and today
• update the tracked date as max(date)

Structure of .md file:

Date: 2024-10-10

Table 1

Date: 2024-09-01

Table 2

[amishas157]

But also, I think it's quite a bit of text formatting. We can leave it as is and can revisit later if needed. In the first place, I don't think we are going to see a lot of schema changes in general.

[chowbao]

Yup sounds good