maxEventsPerTrigger is not working

[thetruelam]

Bug Report:

• Actual behavior

I'm trying to use maxEventsPerTrigger config in pyspark but it doesn't work
this is how I set the config

`connection_string = dbutils.secrets.get(scope="edpprodseakv-dbwscope", key=config['connection_string'])
ehConf = {'eventhubs.connectionString' : sc._jvm.org.apache.spark.eventhubs.EventHubsUtils.encrypt(connection_string)}

startTime = None # current_date - 7 days
endTime = None # current_date

startingEventPosition = {
"offset": 0,
"seqNo": -1,
"enqueuedTime": startTime,
"isInclusive": True
}

endingEventPosition = {
"offset": None,
"seqNo": -1,
"enqueuedTime": endTime,
"isInclusive": True
}

ehConf["eventhubs.startingPosition"] = json.dumps(startingEventPosition)
ehConf["eventhubs.endingPosition"] = json.dumps(endingEventPosition)
ehConf["maxEventsPerTrigger"] = 200000`

This is how I call the read stream function
df = spark.readStream.format("eventhubs").options(**ehConf).load()

But the streaming jobs seem to ignore the config

• Expected behavior
  I expect the maximum of events in each microbatch to be 200000.

• Spark version
  Databricks Runtime Version: 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12)
  Spark 2.12

• spark-eventhubs artifactId and version
  com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.21