[BUG]can't download error :org.apache.commons#commons-math3;3.2!commons-math3.jar

[guxiaobo]

SynapseML version

1.0.2

System information

• Spark Version 3.4.2
• Spark Platform : apache offical version

Describe the problem

pyspark shipped with spark 3.4.2 can't start with synapseml packages

Code to reproduce issue

./bin/pyspark --packages com.microsoft.azure:synapseml_2.12:1.0.2

Other info / logs

(base) linda@LindadeMacBook-Air spark-3.4.2 % ./bin/pyspark --packages com.microsoft.azure:synapseml_2.12:1.0.2
Python 3.8.8 (default, Apr 13 2021, 12:59:45)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
24/02/21 22:43:47 WARN Utils: Your hostname, LindadeMacBook-Air.local resolves to a loopback address: 127.0.0.1; using 192.168.1.4 instead (on interface en0)
24/02/21 22:43:47 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
:: loading settings :: url = jar:file:/Users/linda/Downloads/spark-3.4.2/jars/ivy-2.5.1.jar!/org/apache/ivy/core/settings/ivysettings.xml
Ivy Default Cache set to: /Users/linda/.ivy2/cache
The jars for the packages stored in: /Users/linda/.ivy2/jars
com.microsoft.azure#synapseml_2.12 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent-2a3be309-852e-4a0b-95ce-34e2681962b1;1.0
confs: [default]
found com.microsoft.azure#synapseml_2.12;1.0.2 in central
found com.microsoft.azure#synapseml-core_2.12;1.0.2 in central
found org.apache.spark#spark-avro_2.12;3.4.1 in central
found org.tukaani#xz;1.9 in local-m2-cache
found commons-lang#commons-lang;2.6 in local-m2-cache
found org.scalactic#scalactic_2.12;3.2.14 in central
found org.scala-lang#scala-reflect;2.12.15 in central
found io.spray#spray-json_2.12;1.3.5 in central
found com.jcraft#jsch;0.1.54 in central
found org.apache.httpcomponents.client5#httpclient5;5.1.3 in central
found org.apache.httpcomponents.core5#httpcore5;5.1.3 in central
found org.apache.httpcomponents.core5#httpcore5-h2;5.1.3 in central
found org.slf4j#slf4j-api;1.7.25 in local-m2-cache
found commons-codec#commons-codec;1.15 in central
found org.apache.httpcomponents#httpmime;4.5.13 in central
found org.apache.httpcomponents#httpclient;4.5.13 in local-m2-cache
found org.apache.httpcomponents#httpcore;4.4.13 in local-m2-cache
found commons-logging#commons-logging;1.2 in local-m2-cache
found com.linkedin.isolation-forest#isolation-forest_3.4.1_2.12;3.0.3 in central
found com.chuusai#shapeless_2.12;2.3.10 in central
found org.testng#testng;6.8.8 in central
found org.beanshell#bsh;2.0b4 in local-m2-cache
found com.beust#jcommander;1.27 in central
found org.scalanlp#breeze_2.12;2.1.0 in central
found org.scalanlp#breeze-macros_2.12;2.1.0 in central
found org.typelevel#spire_2.12;0.17.0 in central
found org.typelevel#spire-macros_2.12;0.17.0 in central
found org.typelevel#algebra_2.12;2.0.1 in central
found org.typelevel#cats-kernel_2.12;2.1.1 in central
found org.typelevel#spire-platform_2.12;0.17.0 in central
found org.typelevel#spire-util_2.12;0.17.0 in central
found dev.ludovic.netlib#blas;3.0.1 in central
found net.sourceforge.f2j#arpack_combined_all;0.1 in central
found dev.ludovic.netlib#lapack;3.0.1 in central
found dev.ludovic.netlib#arpack;3.0.1 in central
found net.sf.opencsv#opencsv;2.3 in central
found com.github.wendykierp#JTransforms;3.1 in central
found pl.edu.icm#JLargeArrays;1.5 in central
found org.apache.commons#commons-math3;3.2 in local-m2-cache
found org.scala-lang.modules#scala-collection-compat_2.12;2.7.0 in central
found com.microsoft.azure#synapseml-deep-learning_2.12;1.0.2 in central
found com.microsoft.azure#synapseml-opencv_2.12;1.0.2 in central
found org.openpnp#opencv;3.2.0-1 in central
found com.microsoft.azure#onnx-protobuf_2.12;0.9.3 in central
found com.microsoft.onnxruntime#onnxruntime_gpu;1.8.1 in central
found com.microsoft.azure#synapseml-cognitive_2.12;1.0.2 in central
found com.microsoft.cognitiveservices.speech#client-sdk;1.24.1 in central
found com.microsoft.azure#synapseml-vw_2.12;1.0.2 in central
found com.github.vowpalwabbit#vw-jni;9.3.0 in central
found com.microsoft.azure#synapseml-lightgbm_2.12;1.0.2 in central
found com.microsoft.ml.lightgbm#lightgbmlib;3.3.510 in central
:: resolution report :: resolve 1937ms :: artifacts dl 71ms
:: modules in use:
com.beust#jcommander;1.27 from central in [default]
com.chuusai#shapeless_2.12;2.3.10 from central in [default]
com.github.vowpalwabbit#vw-jni;9.3.0 from central in [default]
com.github.wendykierp#JTransforms;3.1 from central in [default]
com.jcraft#jsch;0.1.54 from central in [default]
com.linkedin.isolation-forest#isolation-forest_3.4.1_2.12;3.0.3 from central in [default]
com.microsoft.azure#onnx-protobuf_2.12;0.9.3 from central in [default]
com.microsoft.azure#synapseml-cognitive_2.12;1.0.2 from central in [default]
com.microsoft.azure#synapseml-core_2.12;1.0.2 from central in [default]
com.microsoft.azure#synapseml-deep-learning_2.12;1.0.2 from central in [default]
com.microsoft.azure#synapseml-lightgbm_2.12;1.0.2 from central in [default]
com.microsoft.azure#synapseml-opencv_2.12;1.0.2 from central in [default]
com.microsoft.azure#synapseml-vw_2.12;1.0.2 from central in [default]
com.microsoft.azure#synapseml_2.12;1.0.2 from central in [default]
com.microsoft.cognitiveservices.speech#client-sdk;1.24.1 from central in [default]
com.microsoft.ml.lightgbm#lightgbmlib;3.3.510 from central in [default]
com.microsoft.onnxruntime#onnxruntime_gpu;1.8.1 from central in [default]
commons-codec#commons-codec;1.15 from central in [default]
commons-lang#commons-lang;2.6 from local-m2-cache in [default]
commons-logging#commons-logging;1.2 from local-m2-cache in [default]
dev.ludovic.netlib#arpack;3.0.1 from central in [default]
dev.ludovic.netlib#blas;3.0.1 from central in [default]
dev.ludovic.netlib#lapack;3.0.1 from central in [default]
io.spray#spray-json_2.12;1.3.5 from central in [default]
net.sf.opencsv#opencsv;2.3 from central in [default]
net.sourceforge.f2j#arpack_combined_all;0.1 from central in [default]
org.apache.commons#commons-math3;3.2 from local-m2-cache in [default]
org.apache.httpcomponents#httpclient;4.5.13 from local-m2-cache in [default]
org.apache.httpcomponents#httpcore;4.4.13 from local-m2-cache in [default]
org.apache.httpcomponents#httpmime;4.5.13 from central in [default]
org.apache.httpcomponents.client5#httpclient5;5.1.3 from central in [default]
org.apache.httpcomponents.core5#httpcore5;5.1.3 from central in [default]
org.apache.httpcomponents.core5#httpcore5-h2;5.1.3 from central in [default]
org.apache.spark#spark-avro_2.12;3.4.1 from central in [default]
org.beanshell#bsh;2.0b4 from local-m2-cache in [default]
org.openpnp#opencv;3.2.0-1 from central in [default]
org.scala-lang#scala-reflect;2.12.15 from central in [default]
org.scala-lang.modules#scala-collection-compat_2.12;2.7.0 from central in [default]
org.scalactic#scalactic_2.12;3.2.14 from central in [default]
org.scalanlp#breeze-macros_2.12;2.1.0 from central in [default]
org.scalanlp#breeze_2.12;2.1.0 from central in [default]
org.slf4j#slf4j-api;1.7.25 from local-m2-cache in [default]
org.testng#testng;6.8.8 from central in [default]
org.tukaani#xz;1.9 from local-m2-cache in [default]
org.typelevel#algebra_2.12;2.0.1 from central in [default]
org.typelevel#cats-kernel_2.12;2.1.1 from central in [default]
org.typelevel#spire-macros_2.12;0.17.0 from central in [default]
org.typelevel#spire-platform_2.12;0.17.0 from central in [default]
org.typelevel#spire-util_2.12;0.17.0 from central in [default]
org.typelevel#spire_2.12;0.17.0 from central in [default]
pl.edu.icm#JLargeArrays;1.5 from central in [default]
:: evicted modules:
commons-codec#commons-codec;1.11 by [commons-codec#commons-codec;1.15] in [default]
org.scala-lang.modules#scala-collection-compat_2.12;2.2.0 by [org.scala-lang.modules#scala-collection-compat_2.12;2.7.0] in [default]
org.apache.commons#commons-math3;3.5 by [org.apache.commons#commons-math3;3.2] in [default]
org.slf4j#slf4j-api;1.7.5 by [org.slf4j#slf4j-api;1.7.25] in [default]
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 55 | 0 | 0 | 4 || 51 | 0 |
---------------------------------------------------------------------

:: problems summary ::
:::: WARNINGS
[NOT FOUND ] org.apache.commons#commons-math3;3.2!commons-math3.jar (2ms)

==== local-m2-cache: tried

  file:/Users/linda/.m2/repository/org/apache/commons/commons-math3/3.2/commons-math3-3.2.jar

	::::::::::::::::::::::::::::::::::::::::::::::

	::              FAILED DOWNLOADS            ::

	:: ^ see resolution messages for details  ^ ::

	::::::::::::::::::::::::::::::::::::::::::::::

	:: org.apache.commons#commons-math3;3.2!commons-math3.jar

	::::::::::::::::::::::::::::::::::::::::::::::

:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
Exception in thread "main" java.lang.RuntimeException: [download failed: org.apache.commons#commons-math3;3.2!commons-math3.jar]
at org.apache.spark.deploy.SparkSubmitUtils$.resolveMavenCoordinates(SparkSubmit.scala:1528)
at org.apache.spark.util.DependencyUtils$.resolveMavenDependencies(DependencyUtils.scala:185)
at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:332)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:192)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:215)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1111)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Traceback (most recent call last):
File "/Users/linda/Downloads/spark-3.4.2/python/pyspark/shell.py", line 66, in
SparkContext._ensure_initialized()
File "/Users/linda/Downloads/spark-3.4.2/python/pyspark/context.py", line 432, in _ensure_initialized
SparkContext._gateway = gateway or launch_gateway(conf)
File "/Users/linda/Downloads/spark-3.4.2/python/pyspark/java_gateway.py", line 106, in launch_gateway
raise RuntimeError("Java gateway process exited before sending its port number")
RuntimeError: Java gateway process exited before sending its port number

quit()

What component(s) does this bug affect?

• area/cognitive: Cognitive project
• area/core: Core project
• area/deep-learning: DeepLearning project
• area/lightgbm: Lightgbm project
• area/opencv: Opencv project
• area/vw: VW project
• area/website: Website
• area/build: Project build system
• area/notebooks: Samples under notebooks folder
• area/docker: Docker usage
• area/models: models related issue

What language(s) does this bug affect?

• language/scala: Scala source code
• language/python: Pyspark APIs
• language/r: R APIs
• language/csharp: .NET APIs
• language/new: Proposals for new client languages

What integration(s) does this bug affect?

• integrations/synapse: Azure Synapse integrations
• integrations/azureml: Azure ML integrations
• integrations/databricks: Databricks integrations