feat: improve schema generator

[saikumarrs]

Description of the change

Improvements:

• Formatted the JSON diff output.
• Allowed updating all the files (via -all option).
• Added checks for the directory.
• Added a warning for missing fields in db-config.json but found in ui-config.json.
• Fixed the generation of the useNativeSDK field always.
• Fixed the issue where the schema is generated for a field even though it is not present in the db-config.json file.
• Added support for dynamicCustomForm.
• Instead of generating a fresh schema file using the update option, the existing file is updated in-place. This will result in fewer overall changes, making it easier to review the diff.
• Merged existing anyOf conditions with generated ones.
• Generated allOf and anyOf schema components for the new UI config.
• Handled preRequisites for the new UI config.

Checklists

Development

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• The code changed/added as part of this pull request has been covered with tests
• All tests related to the changed code pass in development
• I have executed schemaGenerator tests and updated schema if needed

Code review

• This pull request has a descriptive title and information useful to a reviewer. There may be a screenshot or screencast attached
• Changes have been reviewed by at least one other engineer
• Issue from task tracker has a link to this pull request

[linear]

SDK-998 Make legacy consent configuration fields source specific

Description

As the modern consent configuration field in the destination, consentManagement, is source-type specific, i.e., users can configure different consents for different sources (JS, Android, etc.), we have to make the legacy fields also source-type specific.

Otherwise, we cannot make the configuration APIs backward compatible, where we have to backfill modern consent data into the legacy fields.

Reference:

https://github.com/rudderlabs/rudder-config-backend/pull/3894/files#r1408949602

Research Spec

• Criteo Audience
• FB Custom Audience
• Google Adwords Remarketing List
• LaunchDarkly Segment
• Marketo Static List
• Snapchat Custom Audience
• Tiktok Audience
• Lotame
• Yahoo DSP

---

• Ortto
• Rockerbox
• Sprig
• Klaviyo (multiselect)
• Kafka
• Iterable
• HS
• GA4
• FullStory
• Firehose
• Deltalake*
• Clickhouse*
• Azure Synapse
• Azure Blob
• Adobe Analytics

---

• Shopify
• AMP
• Cloud

Scope

• Update all destination definitions
• Add an API in the config-backend scripts service to migrate the database
• Deploy and test in the dev environment

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (009c8da) 100.00% compared to head (a61c1cd) 100.00%.

Additional details and impacted files

@@            Coverage Diff            @@
##           develop     #1128   +/-   ##
=========================================
  Coverage   100.00%   100.00%           
=========================================
  Files            2         2           
  Lines           53        53           
  Branches         7         7           
=========================================
  Hits            53        53           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[devops-github-rudderstack]

This PR is considered to be stale. It has been open for 20 days with no further activity thus it is going to be closed in 7 days. To avoid such a case please consider removing the stale label manually or add a comment to the PR.