Skip to content

Commit

Permalink
e2e groupby additional tests
Browse files Browse the repository at this point in the history
  • Loading branch information
itsmekumari committed Dec 6, 2024
1 parent 7793ef2 commit a7aa203
Show file tree
Hide file tree
Showing 11 changed files with 448 additions and 1 deletion.
150 changes: 150 additions & 0 deletions core-plugins/src/e2e-test/features/groupby/GroupByMacros.feature
Original file line number Diff line number Diff line change
Expand Up @@ -57,3 +57,153 @@ Feature:GroupBy - Verification of GroupBy pipeline with File as source and File
Then Close the pipeline logs
Then Validate OUT record count of groupby is equal to IN record count of sink
Then Validate output file generated by file sink plugin "fileSinkTargetBucket" is equal to expected output file "groupByMacroOutputFile"

@GROUP_BY_TEST @FILE_SINK_TEST
Scenario: To verify pipeline is failed when fields and aggregates set as macro arguments with invalid value
Given Open Datafusion Project to configure pipeline
When Select plugin: "File" from the plugins list as: "Source"
When Expand Plugin group in the LHS plugins list: "Analytics"
When Select plugin: "Group By" from the plugins list as: "Analytics"
Then Connect plugins: "File" and "Group By" to establish connection
When Expand Plugin group in the LHS plugins list: "Sink"
When Select plugin: "File" from the plugins list as: "Sink"
Then Connect plugins: "Group By" and "File2" to establish connection
Then Navigate to the properties page of plugin: "File"
Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
Then Enter input plugin property: "path" with value: "groupByTest"
Then Select dropdown plugin property: "format" with option value: "csv"
Then Click plugin property: "skipHeader"
Then Click on the Get Schema button
Then Verify the Output Schema matches the Expected Schema: "groupByCsvDataTypeFileSchema"
Then Validate "File" plugin properties
Then Close the Plugin Properties page
Then Navigate to the properties page of plugin: "Group By"
Then Click on the Macro button of Property: "groupByFields" and set the value to: "groupByField"
Then Click on the Macro button of Property: "aggregates" and set the value to: "groupByAggregatesField"
Then Click on the Get Schema button
Then Click on the Validate button
Then Close the Plugin Properties page
Then Navigate to the properties page of plugin: "File2"
Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
Then Enter input plugin property: "path" with value: "fileSinkTargetBucket"
Then Replace input plugin property: "pathSuffix" with value: "yyyy-MM-dd-HH-mm-ss"
Then Select dropdown plugin property: "format" with option value: "tsv"
Then Validate "File2" plugin properties
Then Close the Plugin Properties page
Then Save the pipeline
Then Deploy the pipeline
Then Run the Pipeline in Runtime
Then Enter runtime argument value "invalidGroupByFields" for key "groupByField"
Then Enter runtime argument value "invalidGroupByAggregatesField" for key "groupByAggregatesField"
Then Run the Pipeline in Runtime with runtime arguments
Then Wait till pipeline is in running state
Then Open and capture logs
Then Verify the pipeline status is "Failed"
Then Close the pipeline logs
Then Open Pipeline logs and verify Log entries having below listed Level and Message:
| Level | Message |
| ERROR | errorLogsMessageGroupByInvalidFields |

@GROUP_BY_TEST @FILE_SINK_TEST
Scenario: To verify pipeline is failed when number of partitions set as macro arguments with invalid value
Given Open Datafusion Project to configure pipeline
When Select plugin: "File" from the plugins list as: "Source"
When Expand Plugin group in the LHS plugins list: "Analytics"
When Select plugin: "Group By" from the plugins list as: "Analytics"
Then Connect plugins: "File" and "Group By" to establish connection
When Expand Plugin group in the LHS plugins list: "Sink"
When Select plugin: "File" from the plugins list as: "Sink"
Then Connect plugins: "Group By" and "File2" to establish connection
Then Navigate to the properties page of plugin: "File"
Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
Then Enter input plugin property: "path" with value: "groupByTest"
Then Select dropdown plugin property: "format" with option value: "csv"
Then Click plugin property: "skipHeader"
Then Click on the Get Schema button
Then Verify the Output Schema matches the Expected Schema: "groupByCsvDataTypeFileSchema"
Then Validate "File" plugin properties
Then Close the Plugin Properties page
Then Navigate to the properties page of plugin: "Group By"
Then Select dropdown plugin property: "groupByFields" with option value: "groupByValidFirstField"
Then Press ESC key to close the unique fields dropdown
Then Select dropdown plugin property: "groupByFields" with option value: "groupByValidSecondField"
Then Press ESC key to close the unique fields dropdown
Then Enter GroupBy plugin Fields to be Aggregate "groupByGcsAggregateFields"
Then Click on the Macro button of Property: "numberOfPartitions" and set the value to: "groupByInvalidPartitions"
Then Click on the Get Schema button
Then Click on the Validate button
Then Close the Plugin Properties page
Then Navigate to the properties page of plugin: "File2"
Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
Then Enter input plugin property: "path" with value: "fileSinkTargetBucket"
Then Replace input plugin property: "pathSuffix" with value: "yyyy-MM-dd-HH-mm-ss"
Then Select dropdown plugin property: "format" with option value: "tsv"
Then Validate "File2" plugin properties
Then Close the Plugin Properties page
Then Save the pipeline
Then Deploy the pipeline
Then Run the Pipeline in Runtime
Then Enter runtime argument value "groupByInvalidPartitionValue" for key "groupByInvalidPartitions"
Then Run the Pipeline in Runtime with runtime arguments
Then Wait till pipeline is in running state
Then Open and capture logs
Then Verify the pipeline status is "Failed"
Then Close the pipeline logs
Then Open Pipeline logs and verify Log entries having below listed Level and Message:
| Level | Message |
| ERROR | errorLogsMessageGroupByInvalidNumberOfPartitions |

@GROUP_BY_TEST @FILE_SINK_TEST
Scenario: To verify complete flow of data extract and transfer from File source to File sink with GroupBy plugin using count(Asterisk) aggregate using macro-enabled
Given Open Datafusion Project to configure pipeline
When Select plugin: "File" from the plugins list as: "Source"
When Expand Plugin group in the LHS plugins list: "Analytics"
When Select plugin: "Group By" from the plugins list as: "Analytics"
Then Connect plugins: "File" and "Group By" to establish connection
When Expand Plugin group in the LHS plugins list: "Sink"
When Select plugin: "File" from the plugins list as: "Sink"
Then Connect plugins: "Group By" and "File2" to establish connection
Then Navigate to the properties page of plugin: "File"
Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
Then Enter input plugin property: "path" with value: "groupByTest"
Then Select dropdown plugin property: "format" with option value: "csv"
Then Click plugin property: "skipHeader"
Then Click on the Get Schema button
Then Verify the Output Schema matches the Expected Schema: "groupByCsvDataTypeFileSchema"
Then Validate "File" plugin properties
Then Close the Plugin Properties page
Then Navigate to the properties page of plugin: "Group By"
Then Select dropdown plugin property: "groupByFields" with option value: "groupByValidFirstField"
Then Press ESC key to close the unique fields dropdown
Then Select dropdown plugin property: "groupByFields" with option value: "groupByValidSecondField"
Then Press ESC key to close the unique fields dropdown
Then Click on the Macro button of Property: "aggregates" and set the value to: "groupByAggregate"
Then Click on the Get Schema button
Then Click on the Validate button
Then Close the Plugin Properties page
Then Navigate to the properties page of plugin: "File2"
Then Enter input plugin property: "referenceName" with value: "FileReferenceName"
Then Enter input plugin property: "path" with value: "fileSinkTargetBucket"
Then Replace input plugin property: "pathSuffix" with value: "yyyy-MM-dd-HH-mm-ss"
Then Select dropdown plugin property: "format" with option value: "csv"
Then Validate "File2" plugin properties
Then Close the Plugin Properties page
Then Save the pipeline
Then Preview and run the pipeline
Then Enter runtime argument value "groupByMacroCountAggregate" for key "groupByAggregate"
Then Run the preview of pipeline with runtime arguments
Then Wait till pipeline preview is in running state
Then Open and capture pipeline preview logs
Then Verify the preview run status of pipeline in the logs is "succeeded"
Then Close the pipeline logs
Then Close the preview
Then Deploy the pipeline
Then Run the Pipeline in Runtime
Then Enter runtime argument value "groupByMacroCountAggregate" for key "groupByAggregate"
Then Run the Pipeline in Runtime with runtime arguments
Then Wait till pipeline is in running state
Then Open and capture logs
Then Verify the pipeline status is "Succeeded"
Then Close the pipeline logs
Then Validate OUT record count of groupby is equal to IN record count of sink
Then Validate output file generated by file sink plugin "fileSinkTargetBucket" is equal to expected output file "groupByTest7OutputFile"
Loading

0 comments on commit a7aa203

Please sign in to comment.