feat: Adds helper functions for migrations

[luizotavio32]

SUMMARY

This PR aims to implement helper functions that will provide standard interfaces for the most used methods found in the migrations for the past 2 years, which are: create and drop tables, add and drop columns, create and drop indexes. Another objective to make logging standard among migrations by providing more human readable logs and displaying the same type of information.

These new functions will apply some validations before executing such as:

• When adding/dropping a column, does this column already exist on the table?
• Does the index to be created is already present on the table?

These types of validations will make the migrations more resilient. The added helper functions and their logs are:

def create_table(table_name: str, *columns: SchemaItem) -> None:

INFO [alembic] Creating table temp_table…
INFO [alembic] Table temp_table created

def drop_table(table_name: str) -> None:

INFO  [alembic] Dropping table temp_table...
INFO  [alembic] Table temp_table dropped

def add_columns(table_name: str, *columns: Column) -> None:

INFO [alembic] Adding column new_id on table temp_table
INFO [alembic] Column id already present on table temp_table Skipping…

def drop_columns(table_name: str, *columns: str) -> None:

INFO  [alembic] Column address is not present on table temp_table Skipping...
INFO  [alembic] Dropping column new_id from table temp_table

def create_index(table_name: str, index_name: str, *columns: str) -> None:

INFO [alembic] Creating index my_index on table temp_table
INFO [alembic] Table temp_table already has index my_index Skipping…

def drop_index(table_name: str, index_name: str) -> None:

INFO  [alembic] Dropping index my_index from table temp_table
INFO  [alembic] Table temp_table doesn't have index other_index Skipping...

There’s also a new method for executing database operations in batch. This function will receive a Callable, the count and batch_size as the parameters. When executing, the Callable will receive the offset and limit for the current processed batch.

def batch_operation(callable: Callable[[int, int], None], count: int, batch_size: int) -> None:

INFO [alembic] Progress: 0/100,000 (0.00%)
INFO [alembic] Progress: 25,000/100,000 (25.00%)
INFO [alembic] Progress: 50,000/100,000 (50.00%)
INFO [alembic] Progress: 75.000/100,000 (75.00%)
INFO [alembic] Progress: 100,000/100,000 (100%)
INFO [alembic] End: callable_example batch operation successfully executed

TESTING INSTRUCTIONS

Execute the migrations using superset db upgrade and superset db downgrade` commands and make sure everything works as expected.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[github-actions]

Congrats on making your first PR and thank you for contributing to Superset! 🎉 ❤️

We hope to see you in our Slack community too! Not signed up? Use our Slack App to self-register.

[codecov]

Codecov Report

Attention: Patch coverage is 23.37662% with 59 lines in your changes missing coverage. Please review.

Project coverage is 83.73%. Comparing base (76d897e) to head (f225fb7).
Report is 1170 commits behind head on master.

Files with missing lines	Patch %	Lines
superset/migrations/shared/utils.py	23.37%	59 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #31303       +/-   ##
===========================================
+ Coverage   60.48%   83.73%   +23.24%     
===========================================
  Files        1931      537     -1394     
  Lines       76236    39061    -37175     
  Branches     8568        0     -8568     
===========================================
- Hits        46114    32709    -13405     
+ Misses      28017     6352    -21665     
+ Partials     2105        0     -2105     

Flag	Coverage Δ
hive	48.74% <23.37%> (-0.42%)	⬇️
javascript	?
mysql	76.44% <23.37%> (?)
postgres	76.53% <23.37%> (?)
presto	53.26% <23.37%> (-0.54%)	⬇️
python	83.73% <23.37%> (+20.25%)	⬆️
sqlite	75.99% <23.37%> (?)
unit	60.87% <23.37%> (+3.24%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mistercrunch]

Not directly related but I've been considering squashing old migration into a big large initial one, keeping say 2 years worth or migrations or since say 2.x

[michael-s-molina]

Thank you for the valuable PR @luizotavio32. I left some comments.