Merge branch 'develop' into feat.sftp

CHANGELOG.md

@@ -2,6 +2,42 @@
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
+### [1.71.2](https://github.com/rudderlabs/rudder-config-schema/compare/v1.71.1...v1.71.2) (2024-04-24)
+
+
+### Bug Fixes
+
+* iterable schema ([#1332](https://github.com/rudderlabs/rudder-config-schema/issues/1332)) ([73060ab](https://github.com/rudderlabs/rudder-config-schema/commit/73060ab151274fadd7701ccfd30f1af9d93cc5b3))
+
+### [1.71.1](https://github.com/rudderlabs/rudder-config-schema/compare/v1.71.0...v1.71.1) (2024-04-22)
+
+
+### Bug Fixes
+
+* onboard yandex metrica offline events ([#1326](https://github.com/rudderlabs/rudder-config-schema/issues/1326)) ([090ea76](https://github.com/rudderlabs/rudder-config-schema/commit/090ea76402480f1df747d5696fe4277341f6dcd6))
+
+## [1.71.0](https://github.com/rudderlabs/rudder-config-schema/compare/v1.70.1...v1.71.0) (2024-04-15)
+
+
+### Features
+
+* add connectionMode for all sourceTypes in dest def for all destinations ([#1271](https://github.com/rudderlabs/rudder-config-schema/issues/1271)) ([d2420e3](https://github.com/rudderlabs/rudder-config-schema/commit/d2420e354873fdce8476d7792e9c84c5a2c0934e))
+* make regex mandatory for required textInputs ([#1289](https://github.com/rudderlabs/rudder-config-schema/issues/1289)) ([ce6eaa1](https://github.com/rudderlabs/rudder-config-schema/commit/ce6eaa182b34be31871aabd1f040d1318ec0b343))
+
+### [1.70.1](https://github.com/rudderlabs/rudder-config-schema/compare/v1.70.0...v1.70.1) (2024-04-08)
+
+
+### Bug Fixes
+
+* revert fb marketing connector ([242e7c1](https://github.com/rudderlabs/rudder-config-schema/commit/242e7c161904917d8685452ef72959b78ed37e4f))
+
+## [1.70.0](https://github.com/rudderlabs/rudder-config-schema/compare/v1.69.0...v1.70.0) (2024-04-05)
+
+
+### Features
+
+* extend gcm support for additional destinations ([#1307](https://github.com/rudderlabs/rudder-config-schema/issues/1307)) ([d819a43](https://github.com/rudderlabs/rudder-config-schema/commit/d819a43b2287c33cb30fd26a90da9abb2e6c5ac0))
+
 ## [1.69.0](https://github.com/rudderlabs/rudder-config-schema/compare/v1.68.1...v1.69.0) (2024-04-02)
 
 

CONTRIBUTING.md

@@ -1,4 +1,4 @@
-# Contributing to RudderStack
+# Contributing to Rudder-Integrations-Config
 
 Thanks for taking the time and for your help in improving this project!
 
@@ -13,22 +13,51 @@ Thanks for taking the time and for your help in improving this project!
 
 To contribute to this project, we need you to sign the [**Contributor License Agreement (“CLA”)**][CLA] for the first commit you make. By agreeing to the [**CLA**][CLA], we can add you to list of approved contributors and review the changes proposed by you.
 
-## How you can contribute to RudderStack
+## How you can contribute a destination to this project
 
-If you come across any issues or bugs, or have any suggestions for improvement, you can navigate to the specific file in the [**repo**](https://github.com/rudderlabs/rudder-config-schema), make the change, and raise a PR.
+To contribute a destination, you need to provide all the required data for all the fields you want as the settings to configure the destination.
 
-You can also contribute to any open-source RudderStack project. View our [**GitHub page**](https://github.com/rudderlabs) to see all the different projects.
+## How you can provide your destination connection setting details
+
+You can checkout the sample input file [**here**](/test/configData/inputData.json):
+
+For the above input data, the UI will look like as shown below:
+
+<img width="1303" alt="connectionSettings" src="https://github.com/rudderlabs/rudder-integrations-config/assets/63387036/ee28b3fe-5a81-4289-9f6a-54cf5abe0f4c">
+<img width="1292" alt="configurationSettings" src="https://github.com/rudderlabs/rudder-integrations-config/assets/63387036/ce765f0f-b680-46ca-b46d-ed58e39a7667">
+
+In the input file, you need to provide the destination name you want to display in the UI in the displayName field.
+
+Each json object inside the formFields represents a field you want to add as a connection/configuration settings to the destination.
 
-## Committing
+Description of Keys inside each JSON object:
 
-We prefer squash or rebase commits so that all changes from a branch are committed to master as a single commit. All pull requests are squashed when merged, but rebasing prior to merge gives you better control over the commit message.
+| Property      | Description                                                                            |
+| ------------- | -------------------------------------------------------------------------------------- |
+| type          | This is the input field type you want to add                                           |
+| label         | This is the display name for your field                                                |
+| configKey     | This is the key to which the value for this field will be stored in the db             |
+| required      | You can provide true as a value to make this field as required or false                |
+| placeholder   | This is the the placeholder for this field                                             |
+| secret        | You can provide true as a value to make this field as secret or false                  |
+| preRequisites | You can add the configKey and value if you there is a preRequiste field for this field |
+
+The input field types that you can add are:
+
+| type         | Description                                                               |
+| ------------ | ------------------------------------------------------------------------- |
+| textInput    | This is a simple field for adding text inputs                             |
+| checkbox     | This is used for boolean values (true or false)                           |
+| singleSelect | This is used for choosing one of the option from a set of defined options |
+| multiSelect  | This is used for choosing multiple options from a set of defined options  |
+| tagInput     | This can be used to add multiple text inputs for a field                  |
+
+You can also contribute to any open-source RudderStack project. View our [**GitHub page**](https://github.com/rudderlabs) to see all the different projects.
 
 ## Getting help
 
 For any questions, concerns, or queries, you can start by asking a question in our [**Slack**](https://rudderstack.com/join-rudderstack-slack-community/) community.
 
-### We look forward to your feedback on improving this project!
-
 
 <!----variables---->
 

package.json

@@ -1,6 +1,6 @@
 {
   "name": "rudder-config-schema",
-  "version": "1.69.0",
+  "version": "1.71.2",
   "description": "",
   "main": "src/index.ts",
   "private": true,

scripts/schemaGenerator.py

@@ -126,6 +126,26 @@ def is_dest_field_dependent_on_source(field, dbConfig, schema_field_name):
     return False
 
 
+def is_key_present_in_dest_config(dbConfig, key):
+    """Checks if the given key is present in destConfig across all source types.
+
+    Args:
+        dbConfig (object): Destination configuration in db-config.json.
+        key (string): key to be searched in destConfig tree.
+
+    Returns:
+        boolean: True if the key is present in destConfig else, False.
+    """
+    if not dbConfig:
+        return False
+
+    if "destConfig" in dbConfig:
+        for configSection in dbConfig["destConfig"]:
+            if key in dbConfig["destConfig"][configSection]:
+                return True
+    return False
+
+
 def is_field_present_in_default_config(field, dbConfig, schema_field_name):
     """Checks if the given field is present in defaultConfig list present in dbConfig.
 
@@ -740,6 +760,7 @@ def generate_schema_for_allOf(uiConfig, dbConfig, schema_field_name):
     - For each unique preRequisiteField, the properties are found by matching the current preRequisiteField.
     - preRequisiteField becomes if block and corresponding properties become then block.
 
+
     Args:
         uiConfig (object): file content of ui-config.json.
         dbConfig (object): Configurations of db-config.json.
@@ -971,17 +992,14 @@ def generate_connection_mode(dbConfig):
     return connectionObj
 
 
-def generate_schema_properties(
-    uiConfig, dbConfig, schemaObject, properties, name, selector
-):
+def generate_schema_properties(uiConfig, dbConfig, schemaObject, properties, selector):
     """Generates corresponding schema properties by iterating over each of the ui-config fields.
 
     Args:
         uiConfig (object): file content of ui-config.json.
         dbConfig (object): Configurations of db-config.json.
         schemaObject (object): schema being generated
         properties (object): properties of schema
-        name (string): name of the source or destination.
         selector (string): either 'source' or 'destination'
     """
     if is_old_format(uiConfig):
@@ -992,9 +1010,16 @@ def generate_schema_properties(
                     continue
                 generateFunction = uiTypetoSchemaFn.get(field["type"], None)
                 if generateFunction:
-                    properties[field["value"]] = generateFunction(
-                        field, dbConfig, "value"
-                    )
+                    # Generate schema for the field if it is defined in the destination config
+                    if is_key_present_in_dest_config(dbConfig, field["value"]):
+                        properties[field["value"]] = generateFunction(
+                            field, dbConfig, "value"
+                        )
+                    else:
+                        warnings.warn(
+                            f'The field {field["value"]} is defined in ui-config.json but not in db-config.json\n',
+                            UserWarning,
+                        )
                 if field.get(
                     "required", False
                 ) == True and is_field_present_in_default_config(
@@ -1012,9 +1037,18 @@ def generate_schema_properties(
                         for field in group.get("fields", []):
                             generateFunction = uiTypetoSchemaFn.get(field["type"], None)
                             if generateFunction:
-                                properties[field["configKey"]] = generateFunction(
-                                    field, dbConfig, "configKey"
-                                )
+                                # Generate schema for the field if it is defined in the destination config
+                                if is_key_present_in_dest_config(
+                                    dbConfig, field["configKey"]
+                                ):
+                                    properties[field["configKey"]] = generateFunction(
+                                        field, dbConfig, "configKey"
+                                    )
+                                else:
+                                    warnings.warn(
+                                        f'The field {field["configKey"]} is defined in ui-config.json but not in db-config.json\n',
+                                        UserWarning,
+                                    )
                             if (
                                 template.get("title", "") == "Initial setup"
                                 and is_field_present_in_default_config(
@@ -1027,9 +1061,17 @@ def generate_schema_properties(
             for field in sdkTemplate.get("fields", []):
                 generateFunction = uiTypetoSchemaFn.get(field["type"], None)
                 if generateFunction:
-                    properties[field["configKey"]] = generateFunction(
-                        field, dbConfig, "configKey"
-                    )
+                    # Generate schema for the field if it is defined in the destination config
+                    if is_key_present_in_dest_config(dbConfig, field["configKey"]):
+                        properties[field["configKey"]] = generateFunction(
+                            field, dbConfig, "configKey"
+                        )
+                    else:
+                        warnings.warn(
+                            f'The field {field["configKey"]} is defined in ui-config.json but not in db-config.json\n',
+                            UserWarning,
+                        )
+
                 if field.get(
                     "required", False
                 ) == True and is_field_present_in_default_config(
@@ -1051,9 +1093,12 @@ def generate_schema_properties(
                     schemaObject["required"].append(field["configKey"])
 
             # default properties in new ui-config based schemas.
-            schemaObject["properties"]["useNativeSDK"] = generate_schema_for_checkbox(
-                {"type": "checkbox", "value": "useNativeSDK"}, dbConfig, "value"
-            )
+            if is_key_present_in_dest_config(dbConfig, "useNativeSDK"):
+                schemaObject["properties"]["useNativeSDK"] = (
+                    generate_schema_for_checkbox(
+                        {"type": "checkbox", "value": "useNativeSDK"}, dbConfig, "value"
+                    )
+                )
             schemaObject["properties"]["connectionMode"] = generate_connection_mode(
                 dbConfig
             )
@@ -1099,6 +1144,7 @@ def generate_schema(uiConfig, dbConfig, name, selector):
     schemaObject["type"] = "object"
     schemaObject["properties"] = {}
     allOfSchemaObj = {}
+    print(f"Generating schema for {name} {selector}")
     if is_old_format(uiConfig):
         allOfSchemaObj = generate_schema_for_allOf(uiConfig, dbConfig, "value")
     if allOfSchemaObj:
@@ -1110,8 +1156,9 @@ def generate_schema(uiConfig, dbConfig, name, selector):
                 schemaObject["anyOf"] = allOfSchemaObj
         else:
             schemaObject["allOf"] = allOfSchemaObj
+
     generate_schema_properties(
-        uiConfig, dbConfig, schemaObject, schemaObject["properties"], name, selector
+        uiConfig, dbConfig, schemaObject, schemaObject["properties"], selector
     )
     newSchema["configSchema"] = schemaObject
 
@@ -1154,7 +1201,10 @@ def generate_warnings_for_each_type(uiConfig, dbConfig, schema, curUiType):
                             )
                     if (
                         curUiType == "textInput"
-                        and field["value"] in schema["required"]
+                        and (
+                            schema.get("required", False) == True
+                            and field["value"] in schema["required"]
+                        )
                         and "regex" not in field
                     ):
                         warnings.warn(
@@ -1204,7 +1254,10 @@ def generate_warnings_for_each_type(uiConfig, dbConfig, schema, curUiType):
                                     )
                             if (
                                 curUiType == "textInput"
-                                and field["configKey"] in schema["required"]
+                                and (
+                                    schema.get("required", False) == True
+                                    and field["configKey"] in schema["required"]
+                                )
                                 and "regex" not in field
                             ):
                                 warnings.warn(
@@ -1423,11 +1476,9 @@ def get_schema_diff(name, selector, shouldUpdateSchema=False):
         selector (string): either 'source' or 'destination'.
         shouldUpdateSchema (boolean): if it should update the existing schema with generated one
     """
-
     file_selectors = ["db-config.json", "ui-config.json", "schema.json"]
     directory = f"./{CONFIG_DIR}/{selector}s/{name}"
     if not os.path.isdir(directory):
-        print(f"No {selector}s directory found for {name}")
         return
 
     if name not in EXCLUDED_DEST: