Exception More than Int.MaxValue elements

[YuelongCai]

spark-redshift/src/main/scala/io/github/spark_redshift_community/spark/redshift/RedshiftRelation.scala

Line 125 in 20e7ccb

sqlContext.sparkContext

Caused by: java.lang.IllegalArgumentException: More than Int.MaxValue elements.
at scala.collection.immutable.NumericRange$.check$1(NumericRange.scala:318)
at scala.collection.immutable.NumericRange$.count(NumericRange.scala:328)
at scala.collection.immutable.NumericRange.numRangeElements$lzycompute(NumericRange.scala:53)
at scala.collection.immutable.NumericRange.numRangeElements(NumericRange.scala:52)
at scala.collection.immutable.NumericRange.length(NumericRange.scala:55)
at org.apache.spark.rdd.ParallelCollectionRDD$.slice(ParallelCollectionRDD.scala:143)

[YuelongCai]

val countQuery = s"SELECT count(*) FROM $tableNameOrSubquery $whereClause"
If the countQuery return result more than Int.MaxValue, then use this number to construct a RDD[Row] will cause exception.
Refer to spark source code https://github.com/apache/spark/blob/a073bf38c7d8802e2ab12c54299e1541a48a394e/core/src/main/scala/org/apache/spark/rdd/ParallelCollectionRDD.scala#L143
when call nr.length which is expected to return Int, but exceed int.MaxValue.

[YuelongCai]

The possible solution is

if the total count is too big, like N
to split it into multiple rdds, each should be less than int.MaxValue and union all to generate rdd with size N

for example,
sc.parallelize(1L to 1000000000L, 200).union(sc.parallelize(1L to 1000000000L, 200)).union(sc.parallelize(1L to 1000000000L, 200)).union(sc.parallelize(1L to 1000000000L, 200)) will avoid this issue

[bsharifi]

Thank you @YuelongCai for reporting this limitation and suggestions on how to work around it. While we will consider addressing this in a future release, please feel free to submit a PR and we will be happy to review and merge it.