e2e groupby additional tests

cdapio · Dec 9, 2024 · 002e0cf · 002e0cf
1 parent da1c3a8
commit 002e0cf
Show file tree

Hide file tree

Showing 11 changed files with 448 additions and 1 deletion.
diff --git a/core-plugins/src/e2e-test/features/groupby/GroupByMacros.feature b/core-plugins/src/e2e-test/features/groupby/GroupByMacros.feature
@@ -57,3 +57,153 @@ Feature:GroupBy - Verification of GroupBy pipeline with File as source and File
     Then Close the pipeline logs
     Then Validate OUT record count of groupby is equal to IN record count of sink
     Then Validate output file generated by file sink plugin "fileSinkTargetBucket" is equal to expected output file "groupByMacroOutputFile"
+
+  @GROUP_BY_TEST @FILE_SINK_TEST
+  Scenario: To verify pipeline is failed when fields and aggregates set as macro arguments with invalid value
+    Given Open Datafusion Project to configure pipeline
+    When Select plugin: "File" from the plugins list as: "Source"
+    When Expand Plugin group in the LHS plugins list: "Analytics"
+    When Select plugin: "Group By" from the plugins list as: "Analytics"
+    Then Connect plugins: "File" and "Group By" to establish connection
+    When Expand Plugin group in the LHS plugins list: "Sink"
+    When Select plugin: "File" from the plugins list as: "Sink"
+    Then Connect plugins: "Group By" and "File2" to establish connection
+    Then Navigate to the properties page of plugin: "File"
+    Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
+    Then Enter input plugin property: "path" with value: "groupByTest"
+    Then Select dropdown plugin property: "format" with option value: "csv"
+    Then Click plugin property: "skipHeader"
+    Then Click on the Get Schema button
+    Then Verify the Output Schema matches the Expected Schema: "groupByCsvDataTypeFileSchema"
+    Then Validate "File" plugin properties
+    Then Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "Group By"
+    Then Click on the Macro button of Property: "groupByFields" and set the value to: "groupByField"
+    Then Click on the Macro button of Property: "aggregates" and set the value to: "groupByAggregatesField"
+    Then Click on the Get Schema button
+    Then Click on the Validate button
+    Then Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "File2"
+    Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
+    Then Enter input plugin property: "path" with value: "fileSinkTargetBucket"
+    Then Replace input plugin property: "pathSuffix" with value: "yyyy-MM-dd-HH-mm-ss"
+    Then Select dropdown plugin property: "format" with option value: "tsv"
+    Then Validate "File2" plugin properties
+    Then Close the Plugin Properties page
+    Then Save the pipeline
+    Then Deploy the pipeline
+    Then Run the Pipeline in Runtime
+    Then Enter runtime argument value "invalidGroupByFields" for key "groupByField"
+    Then Enter runtime argument value "invalidGroupByAggregatesField" for key "groupByAggregatesField"
+    Then Run the Pipeline in Runtime with runtime arguments
+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Failed"
+    Then Close the pipeline logs
+    Then Open Pipeline logs and verify Log entries having below listed Level and Message:
+      | Level | Message                                |
+      | ERROR | errorLogsMessageGroupByInvalidFields   |
+
+  @GROUP_BY_TEST @FILE_SINK_TEST
+  Scenario: To verify pipeline is failed when number of partitions set as macro arguments with invalid value
+    Given Open Datafusion Project to configure pipeline
+    When Select plugin: "File" from the plugins list as: "Source"
+    When Expand Plugin group in the LHS plugins list: "Analytics"
+    When Select plugin: "Group By" from the plugins list as: "Analytics"
+    Then Connect plugins: "File" and "Group By" to establish connection
+    When Expand Plugin group in the LHS plugins list: "Sink"
+    When Select plugin: "File" from the plugins list as: "Sink"
+    Then Connect plugins: "Group By" and "File2" to establish connection
+    Then Navigate to the properties page of plugin: "File"
+    Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
+    Then Enter input plugin property: "path" with value: "groupByTest"
+    Then Select dropdown plugin property: "format" with option value: "csv"
+    Then Click plugin property: "skipHeader"
+    Then Click on the Get Schema button
+    Then Verify the Output Schema matches the Expected Schema: "groupByCsvDataTypeFileSchema"
+    Then Validate "File" plugin properties
+    Then Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "Group By"
+    Then Select dropdown plugin property: "groupByFields" with option value: "groupByValidFirstField"
+    Then Press ESC key to close the unique fields dropdown
+    Then Select dropdown plugin property: "groupByFields" with option value: "groupByValidSecondField"
+    Then Press ESC key to close the unique fields dropdown
+    Then Enter GroupBy plugin Fields to be Aggregate "groupByGcsAggregateFields"
+    Then Click on the Macro button of Property: "numberOfPartitions" and set the value to: "groupByInvalidPartitions"
+    Then Click on the Get Schema button
+    Then Click on the Validate button
+    Then Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "File2"
+    Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
+    Then Enter input plugin property: "path" with value: "fileSinkTargetBucket"
+    Then Replace input plugin property: "pathSuffix" with value: "yyyy-MM-dd-HH-mm-ss"
+    Then Select dropdown plugin property: "format" with option value: "tsv"
+    Then Validate "File2" plugin properties
+    Then Close the Plugin Properties page
+    Then Save the pipeline
+    Then Deploy the pipeline
+    Then Run the Pipeline in Runtime
+    Then Enter runtime argument value "groupByInvalidPartitionValue" for key "groupByInvalidPartitions"
+    Then Run the Pipeline in Runtime with runtime arguments
+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Failed"
+    Then Close the pipeline logs
+    Then Open Pipeline logs and verify Log entries having below listed Level and Message:
+      | Level | Message                                            |
+      | ERROR | errorLogsMessageGroupByInvalidNumberOfPartitions   |
+
+  @GROUP_BY_TEST @FILE_SINK_TEST
+  Scenario: To verify complete flow of data extract and transfer from File source to File sink with GroupBy plugin using count(Asterisk) aggregate using macro-enabled
+    Given Open Datafusion Project to configure pipeline
+    When Select plugin: "File" from the plugins list as: "Source"
+    When Expand Plugin group in the LHS plugins list: "Analytics"
+    When Select plugin: "Group By" from the plugins list as: "Analytics"
+    Then Connect plugins: "File" and "Group By" to establish connection
+    When Expand Plugin group in the LHS plugins list: "Sink"
+    When Select plugin: "File" from the plugins list as: "Sink"
+    Then Connect plugins: "Group By" and "File2" to establish connection
+    Then Navigate to the properties page of plugin: "File"
+    Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
+    Then Enter input plugin property: "path" with value: "groupByTest"
+    Then Select dropdown plugin property: "format" with option value: "csv"
+    Then Click plugin property: "skipHeader"
+    Then Click on the Get Schema button
+    Then Verify the Output Schema matches the Expected Schema: "groupByCsvDataTypeFileSchema"
+    Then Validate "File" plugin properties
+    Then Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "Group By"
+    Then Select dropdown plugin property: "groupByFields" with option value: "groupByValidFirstField"
+    Then Press ESC key to close the unique fields dropdown
+    Then Select dropdown plugin property: "groupByFields" with option value: "groupByValidSecondField"
+    Then Press ESC key to close the unique fields dropdown
+    Then Click on the Macro button of Property: "aggregates" and set the value to: "groupByAggregate"
+    Then Click on the Get Schema button
+    Then Click on the Validate button
+    Then Close the Plugin Properties page
+    Then Navigate to the properties page of plugin: "File2"
+    Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
+    Then Enter input plugin property: "path" with value: "fileSinkTargetBucket"
+    Then Replace input plugin property: "pathSuffix" with value: "yyyy-MM-dd-HH-mm-ss"
+    Then Select dropdown plugin property: "format" with option value: "csv"
+    Then Validate "File2" plugin properties
+    Then Close the Plugin Properties page
+    Then Save the pipeline
+    Then Preview and run the pipeline
+    Then Enter runtime argument value "groupByMacroCountAggregate" for key "groupByAggregate"
+    Then Run the preview of pipeline with runtime arguments
+    Then Wait till pipeline preview is in running state
+    Then Open and capture pipeline preview logs
+    Then Verify the preview run status of pipeline in the logs is "succeeded"
+    Then Close the pipeline logs
+    Then Close the preview
+    Then Deploy the pipeline
+    Then Run the Pipeline in Runtime
+    Then Enter runtime argument value "groupByMacroCountAggregate" for key "groupByAggregate"
+    Then Run the Pipeline in Runtime with runtime arguments
+    Then Wait till pipeline is in running state
+    Then Open and capture logs
+    Then Verify the pipeline status is "Succeeded"
+    Then Close the pipeline logs
+    Then Validate OUT record count of groupby is equal to IN record count of sink
+    Then Validate output file generated by file sink plugin "fileSinkTargetBucket" is equal to expected output file "groupByTest7OutputFile"