Error: NoSuchMethodError: cats.kernel.Eq$.catsKernelInstancesForString()Lcats/kernel/Order;

[ericwelshdev]

Getting the following error while

Caused by: java.lang.NoSuchMethodError: cats.kernel.Eq$.catsKernelInstancesForString()Lcats/kernel/Order;
	at io.lemonlabs.uri.Url$.<init>(Uri.scala:597)
	at io.lemonlabs.uri.Url$.<clinit>(Uri.scala)
	at com.iqvia.ecom.util.uriParser$.parse(uriParser.scala:14

To Reproduce
Simply

import io.lemonlabs.uri.Url

val uriObj = Url.parse("https://www.amazon.de/Kopfband-Verl%C3%A4ngerung-Nasenpolster-Atemschutzmaske-Gesichtsmasken-Staubschutzmaske/dp/B08QLPSRWP")

Was looking at the URL scala class and the reference to implicit val orderUrl: Order[Url] = Order.by(_.toString()) is there as does
val Order = cats.kernel.Order in cats however not understanding why its complaining
the

Full Stack

Driver stacktrace:
	at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2765)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2712)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2706)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2706)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1255)
	at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1255)
	at scala.Option.foreach(Option.scala:407)
	at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1255)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2973)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2914)
	at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2902)
	at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:1028)
	at org.apache.spark.SparkContext.runJobInternal(SparkContext.scala:2446)
	at org.apache.spark.sql.execution.collect.Collector.runSparkJobs(Collector.scala:289)
	at org.apache.spark.sql.execution.collect.Collector.collect(Collector.scala:299)
	at org.apache.spark.sql.execution.collect.Collector$.collect(Collector.scala:82)
	at org.apache.spark.sql.execution.collect.Collector$.collect(Collector.scala:88)
	at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:445)
	at org.apache.spark.sql.Dataset.$anonfun$isEmpty$1(Dataset.scala:620)
	at org.apache.spark.sql.Dataset.$anonfun$isEmpty$1$adapted(Dataset.scala:619)
	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3802)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$5(SQLExecution.scala:126)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:267)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withCustomExecutionEnv$1(SQLExecution.scala:104)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:852)
	at org.apache.spark.sql.execution.SQLExecution$.withCustomExecutionEnv(SQLExecution.scala:77)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:217)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3800)
	at org.apache.spark.sql.Dataset.isEmpty(Dataset.scala:619)
	at com.iqvia.ecom.clickStreamParseTest$.main(clickStreamParseTest.scala:65)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read$$iw$$iw$$iw$$iw$$iw$$iw.<init>(command--1:1)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read$$iw$$iw$$iw$$iw$$iw.<init>(command--1:43)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read$$iw$$iw$$iw$$iw.<init>(command--1:45)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read$$iw$$iw$$iw.<init>(command--1:47)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read$$iw$$iw.<init>(command--1:49)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read$$iw.<init>(command--1:51)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read.<init>(command--1:53)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read$.<init>(command--1:57)
	at $linef0b12aa4a7214879be8d81015b474a8225.$read$.<clinit>(command--1)
	at $linef0b12aa4a7214879be8d81015b474a8225.$eval$.$print$lzycompute(<notebook>:7)
	at $linef0b12aa4a7214879be8d81015b474a8225.$eval$.$print(<notebook>:6)
	at $linef0b12aa4a7214879be8d81015b474a8225.$eval.$print(<notebook>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at scala.tools.nsc.interpreter.IMain$ReadEvalPrint.call(IMain.scala:745)
	at scala.tools.nsc.interpreter.IMain$Request.loadAndRun(IMain.scala:1021)
	at scala.tools.nsc.interpreter.IMain.$anonfun$interpret$1(IMain.scala:574)
	at scala.reflect.internal.util.ScalaClassLoader.asContext(ScalaClassLoader.scala:41)
	at scala.reflect.internal.util.ScalaClassLoader.asContext$(ScalaClassLoader.scala:37)
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:41)
	at scala.tools.nsc.interpreter.IMain.loadAndRunReq$1(IMain.scala:573)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:600)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:570)
	at com.databricks.backend.daemon.driver.DriverILoop.execute(DriverILoop.scala:219)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.$anonfun$repl$1(ScalaDriverLocal.scala:235)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.backend.daemon.driver.DriverLocal$TrapExitInternal$.trapExit(DriverLocal.scala:903)
	at com.databricks.backend.daemon.driver.DriverLocal$TrapExit$.apply(DriverLocal.scala:856)
	at com.databricks.backend.daemon.driver.ScalaDriverLocal.repl(ScalaDriverLocal.scala:235)
	at com.databricks.backend.daemon.driver.DriverLocal.$anonfun$execute$13(DriverLocal.scala:544)
	at com.databricks.logging.UsageLogging.$anonfun$withAttributionContext$1(UsageLogging.scala:240)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
	at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:235)
	at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:232)
	at com.databricks.backend.daemon.driver.DriverLocal.withAttributionContext(DriverLocal.scala:53)
	at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:279)
	at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:271)
	at com.databricks.backend.daemon.driver.DriverLocal.withAttributionTags(DriverLocal.scala:53)
	at com.databricks.backend.daemon.driver.DriverLocal.execute(DriverLocal.scala:521)
	at com.databricks.backend.daemon.driver.DriverWrapper.$anonfun$tryExecutingCommand$1(DriverWrapper.scala:689)
	at scala.util.Try$.apply(Try.scala:213)
	at com.databricks.backend.daemon.driver.DriverWrapper.tryExecutingCommand(DriverWrapper.scala:681)
	at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:522)
	at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:634)
	at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:427)
	at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:370)
	at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:221)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError: cats.kernel.Eq$.catsKernelInstancesForString()Lcats/kernel/Order;
	at io.lemonlabs.uri.Url$.<init>(Uri.scala:597)
	at io.lemonlabs.uri.Url$.<clinit>(Uri.scala)
	at com.iqvia.ecom.util.uriParser$.parse(uriParser.scala:14)
	at com.iqvia.ecom.clickStreamParseTest$.$anonfun$parseData$1(clickStreamParseTest.scala:113)
	at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:484)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:490)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:458)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:757)
	at org.apache.spark.sql.execution.collect.UnsafeRowBatchUtils$.encodeUnsafeRows(UnsafeRowBatchUtils.scala:80)
	at org.apache.spark.sql.execution.collect.Collector.$anonfun$processFunc$1(Collector.scala:178)
	at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
	at org.apache.spark.scheduler.Task.doRunTask(Task.scala:150)
	at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:119)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.scheduler.Task.run(Task.scala:91)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:803)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1643)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:806)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:662)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

Any thoughts as to what's going on ? I recall seeing some talk about moving to cats verses fastparse. ( Didnt knowof that was related )

Thanks!
-E

[theon]

I have seen a similar issue before. Previously it was caused by spark depending on cats milestone, which was binary incompatible with other versions of cats. This was causing a similar NoSuchMethodError when using Spark with many libraries (scala-uri included). Details here: typelevel/cats#3628

I made a demo project using scala-uri with spark which would workaround the above issue by shading cats: https://github.com/lemonlabsuk/scala-uri-spark-demo

It looks like the spark issue has been resolved. If it has been resolved, the shading workaround wouldn't be needed anymore, but I'm not sure exactly which version of spark has the fix. Which version of spark are you using? I could do some testing to see if that issue still exists.

[ericwelshdev]

Hi thank you for the quick response!

I'm currently running on
def scalaVersion = '2.12'
def sparkVersion = '3.1.1'

Funny enough , like you mentioned it looks like this was fixed in 3.0.1 and I'm running 3.1.1 so it shouldn't have been an issue

I built a fat jar including the 'cats-core_2.12', version: '2.0.0' and in addition just for the heck of it tried cats-core_2.12 too unfortunately that didn't seem to correct the issue either - in the process of attempting to try the shading cats work around
you provided however I'm using grade so I'm attempting find a way to implement the same shading as used in sbt.

Meanwhile even attempting to

dependencies {



    def scalaVersion = '2.12'
    def sparkVersion = '3.1.1'
    shadow 'junit:junit:3.8.2'


    //compileOnly "com.typesafe.scala-logging:scala-logging_$scalaVersion:3.9.4"
    compileOnly "org.scala-lang:scala-library:$scalaVersion.14"
    compileOnly "org.apache.spark:spark-sql_$scalaVersion:$sparkVersion"
    compileOnly "org.apache.spark:spark-hive_$scalaVersion:$sparkVersion"


// https://mvnrepository.com/artifact/org.scala-lang.modules/scala-parser-combinators
// implementation group: 'org.scala-lang.modules', name: 'scala-parser-combinators_3', version: '2.0.0'

    implementation  'io.lemonlabs:scala-uri_2.12:3.5.0'
//    implementation  ('org.typelevel:cats-core_2.12') {
//        version {
//            strictly '2.0.0'
//        }
//    }

//    implementation  ('org.typelevel:cats-core_2.12:2.0.0') {
//       force= true
//    }


//    compile('org.typelevel:cats-core_2.12') {
//        version {
//            strictly '2.0.0'
//        }
//    }

    shadow('org.typelevel:cats-core_2.12:2.0.0')


        testImplementation 'org.junit.jupiter:junit-jupiter-api:5.7.0'
        testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.7.0'
    }


task relocateShadowJar(type: ConfigureShadowRelocation) {
    target = tasks.shadowJar
}

tasks.shadowJar.dependsOn tasks.relocateShadowJar


tasks.register('uberJar', Jar) {
    archiveClassifier = 'uber'

    from sourceSets.main.output

    dependsOn configurations.runtimeClasspath
    from {
        configurations.runtimeClasspath.findAll { it.name.endsWith('jar') }.collect { zipTree(it) }
    }
}

Unfortunately neither worked

[theon]

Sorry for the delayed reply. I have upgraded https://github.com/lemonlabsuk/scala-uri-spark-demo to spark 3.1.1 and can reproduce the problem by removing the shading.

21/09/13 20:52:38 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, macbook-pro, 64817, None)
Exception in thread "main" java.lang.NoSuchMethodError: 'cats.kernel.Order cats.kernel.Eq$.catsKernelInstancesForString()'
	at io.lemonlabs.uri.Url$.<init>(Uri.scala:521)
	at io.lemonlabs.uri.Url$.<clinit>(Uri.scala)
	at example.UriDemo$.parseSearchRequest(UriDemo.scala:12)
21/09/13 20:52:38 INFO SparkContext: Invoking stop() from shutdown hook
21/09/13 20:52:38 INFO SparkUI: Stopped Spark web UI at http://macbook-pro:4040

With the shading in place, the spark job runs correctly and I can see www.example.com printed:

21/09/13 20:54:56 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, macbook-pro, 64860, None)
www.example.com
21/09/13 20:54:57 INFO SparkUI: Stopped Spark web UI at http://macbook-pro:4040

So it seems to still be the Spark bug described in typelevel/cats#3628 and shading should solve the issue. I'm not sure why the shading isn't working for you gradle build, but happy to help if you can push a self contained project demoing the issue to github

[cequencer]

I stumbled on your excellent library and thank you for writing and maintaining it!

Not trying to reopen this issue, but simply adding another observation at this point in time.

Using Spark 3.2.1, Scala 2.12 and the additional Maven coordinates:

• io.lemonlabs:scala-uri_2.12:4.0.1
• org.typelevel:cats-kernel_2.12:2.7.0
• org.typelevel:cats-core_2.12:2.7.0

I am still getting the same error message:
NoSuchMethodError: cats.kernel.Eq$.catsKernelInstancesForString()Lcats/kernel/Order;

Referencing the following pull request:

• Add new shading rules to make GT work with Spark 3.x locationtech/geotrellis#3397

Spark depends on the old cats.kernel which requires us to shade it to avoid typelevel/cats#3628.

I am guessing at this point here are the options (non-exhausive, open for ideation):

1. Spark 3.x updates the cats.kernel per referenced above
   a. No clue how much work this ask is and hoping someone who is more knowledgable can stage that request
2. The users of this library will have to shade the newer cats as part of the spark-jar to use this library as a work around.
3. Fall back to using java.net.{URL,URI}
4. Possibly try this method with your own risk
5. Another idea / option is brewing