SONARPY-2496 Create rule S7193 PySpark DataFrame toPandas function should be avoided

rules/S7193/metadata.json

@@ -0,0 +1,2 @@
+{
+}

rules/S7193/python/metadata.json

@@ -0,0 +1,25 @@
+{
+  "title": "PySpark DataFrame toPandas function should be avoided",
+  "type": "CODE_SMELL",
+  "status": "ready",
+  "remediation": {
+    "func": "Constant\/Issue",
+    "constantCost": "5min"
+  },
+  "tags": [
+    "data-science",
+    "pyspark"
+  ],
+  "defaultSeverity": "Major",
+  "ruleSpecification": "RSPEC-7193",
+  "sqKey": "S7193",
+  "scope": "All",
+  "defaultQualityProfiles": ["Sonar way"],
+  "quickfix": "unknown",
+  "code": {
+    "impacts": {
+      "RELIABILITY": "MEDIUM"
+    },
+    "attribute": "EFFICIENT"
+  }
+}

rules/S7193/python/rule.adoc

@@ -0,0 +1,68 @@
+This rule raises an issue when `toPandas` is used in PySpark in a way that can lead to memory or performance bottlenecks.
+
+== Why is this an issue?
+
+The use of `toPandas` in PySpark can lead to performance and memory management issues.
+
+PySpark is designed to handle large-scale data processing in a distributed manner, leveraging the power of cluster computing to efficiently manage and process big data. Calling `toPandas` collects all data from a Spark `DataFrame` into a Pandas `DataFrame` on a single machine. This can lead to memory issues and performance bottlenecks, especially with large datasets, which is contrary to the distributed nature of Spark.
+
+For this reason, it is generally advisable to avoid using `toPandas` unless you are certain that the dataset is small enough to be handled comfortably by a single machine. Instead, consider using Spark's built-in functions and capabilities to perform data processing tasks in a distributed manner.
+
+If the conversion to Pandas is necessary, ensure that the dataset size is manageable and that the conversion is justified by specific requirements, such as integration with libraries that require Pandas DataFrames.
+
+=== Exceptions
+
+This rule will not raise issues in the following context:
+
+* When visualization is performed with other libraries such as `matplotlib`, `seaborn`, etc.
+* When the `DataFrame` is of a limited size (e.g. following a call to `limit` or an aggregation through `groupBy`)
+* In tests
+
+== How to fix it
+
+To fix this issue, consider using PySpark built-in capabilities without relying on Pandas whenever possible.
+
+=== Code examples
+
+==== Noncompliant code example
+
+[source,python,diff-id=1,diff-type=noncompliant]
+----
+# Converting a PySpark DataFrame to a Pandas DataFrame
+df = spark.read.csv("my_data.csv")
+pandas_df = df.toPandas()  # Noncompliant: May cause memory issues with large datasets
+filtered_df = df[df['id'] > 1]
+print(filtered_df)
+----
+
+==== Compliant solution
+
+[source,python,diff-id=1,diff-type=compliant]
+----
+from pyspark.sql.functions import col
+
+# Using PySpark DataFrame operations
+df = spark.read.csv("my_data.csv")
+
+# Perform operations using PySpark's distributed capabilities
+filtered_df = df.filter(col("id") > 1)
+filtered_df.show()
+----
+
+== Resources
+=== Documentation
+
+* PySpark Documentation - https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.DataFrame.toPandas.html[pyspark.sql.DataFrame.toPandas]
+
+
+ifdef::env-github,rspecator-view[]
+
+'''
+== Implementation Specification
+(visible only on this page)
+
+=== Message
+
+Consider using PySpark's built-in capabilities instead of converting this `DataFrame` to Pandas.
+
+endif::env-github,rspecator-view[]