Add Column for Column-Level Visibility in Data Quality Framework Result Table

[sudeep7978]

Add Column for Column-Level Visibility in Data Quality Framework Result Table

Details

1. Purpose of the Enhancement

• To improve data observability by explicitly linking results to individual columns.
• To provide better traceability and aid in diagnosing data quality issues at a granular level.
• To support detailed reporting and actionable insights for data quality assessments.

2. Key Changes to be Made

• Schema Update:
  • Modified the schema of the result table to include a new column, e.g., affected_column_name.
  • Ensured backward compatibility by verifying the update does not affect existing workflows.
• Code Updates:
  • Adjusted the data validation logic to capture and populate the affected_column_name field for each rule check.
  • Schema Evolution with AutoMerge:

Enabled Delta Lake's spark.databricks.delta.schema.autoMerge.enabled configuration to allow schema evolution during write operations.
Modified the data quality framework to include the affected_column_name field dynamically if not already present.

[sudeep7978]

PR associated with this enhancement.https://github.com/Nike-Inc/spark-expectations/pull/125