Code

Upgrade to modern spark (3.4)#166

Code

Merged

colindean merged 16 commits intotarget:mastertarget/data-validator:masterfrom holdenk:upgrade-to-modern-sparkholdenk/data-validator:upgrade-to-modern-sparkCopy head branch name to clipboardApr 10, 2024

+88-33

Commits

Commits on Aug 29, 2023

• Upgrade to Spark 2.4.8 and now the tests pass on JDK11 yay!

  holdenk

  committed
  b334e8a

• Most of the way to working on Spark 3, still needs some tests to be updated since the SQL pretty printer has changed in Spark 3.

  holdenk

  committed
  38f2ff4

• Use same circe version for yaml as well.

  holdenk

  committed
  7d8ab64

• Ok the tests pass now

  holdenk

  committed
  8c7ced0

Commits on Sep 2, 2023

• A bit of work towards cross-building

  holdenk

  committed
  92e137b

Commits on Sep 5, 2023

• Fix JDK11 funtimes.

  holdenk

  committed
  d00d2ef

Commits on Sep 8, 2023

• In Spark 3 we need to shade shapeless/cats (or users need to specify user classpath first) and also pick a better Scala version.

  holdenk

  committed
  782767e

Commits on Sep 19, 2023

• Only set these params for modern JDK.

  holdenk

  committed
  0e5a2ab

Commits on Jan 30, 2024

• Merge branch 'master' into upgrade-to-modern-spark

  colindean

  authored
  b6f6e7c

Commits on Apr 10, 2024

• Use Spark 3.5.1 in CI

  colindean

  authored
  7f78124

• Adds nudge to set MODERN_JAVA for JDK11+

  colindean

  authored
  028ea77

• Fixes CI step name to match Spark version therein

  colindean

  authored
  459ac54

• Use latest Scala 2.12

  colindean

  authored
  81d9d9d

• Use latest circe 0.14.6

  colindean

  authored
  f363f38

• Adds a note about new MODERN_JAVA options

  colindean

  authored
  aacdf7f

• Use circe-yaml 0.15.1

  Show description for faa81ab
  colindean

  authored
  faa81ab