what exactly is the right path of the Spark-HDFS's Config Url?

[2qif49lt]

I have intalled Spark and hdfs on my minidcos .
In Spark service's config, there is a configuration item - "hdfs config url".

Base URL that serves HDFS config files (hdfs-site.xml, core-site.xml). If not set, DC/OS Spark will use its default configuration and read from DC/OS HDFS.

i tried many way, but they are not working.

1. /service/hdfs/v1/endpoints/ ， error log: file not exist.
2. http://my-minidcos-ui-site/service/hdfs/v1/endpoints/ ，i am able to open that two files in browser. error log: Failed to fetch 'http://my-minidcos-ui-site/service/hdfs/v1/endpoints//hdfs-site.xml': Error downloading resource, received HTTP return code 401

any help?
TIA!

[snzhizhi]

Hey there,

This is probably not a topic for minidcos but to DC/OS Data Services, I will still try to help out.

Did you install HDFS as well?
Did you make sure all services (Spark and HDFS) are deployed successfully?

A 401 Unauthorized return code indicates that you're not authorized to access this page. In Open DC/OS Authentication == Authorization. Whenever you access a resource through Admin Router (/service/ endpoint in this case) in your browser you will need a DC/OS Authentication token.
Did you log in to DC/OS (by going to http://my-minidcos-ui-site first) before calling these endpoints in the Browser?

[2qif49lt]

thanks.

i work it out by copy this thing "http://api.hdfs.marathon.l4lb.thisdcos.directory/v1/endpoints" that looks have nothing to do with my installation.

Did you install HDFS as well?
Did you make sure all services (Spark and HDFS) are deployed successfully?

i think i didnt install successfully. and i try intall HDFS with nothing. my 8core i7,32G memory still looks powerless.

after twenty minuts, there have no datanode. = =

[timaa2k]

Yes, your machine is most likely not powerful enough for the default configuration of HDFS + Spark on minidcos.
You will need to reduce the resource usage of both packages by supplying custom options when installing them.
Turn down the CPU, Mem and Disk to 0.1, 0.1 and maybe 1024MB and see if it deploys successfully.
I might be that even the minimum configuration uses too much resources for minidcos.
In that case please open an issue on the https://github.com/mesosphere/universe repo.

[rajcspsg]

I have had somewhat similar problem. I'm able to successfully deploy the HDFS when I have created the cluster with 10 docker nodes(3 datanodes, 3 journal nodes, 2 zk failover nodes and 2 namenodes and 1 hdfs node). Then all the process are working fine. This is just to deploy hdfs service from the catalog. I'm not able to install spark and I'm facing some issues. My mac laptop have 32 GB RAM and 256 GB Hard disk.

Please see the stackoverflow questions below -

https://stackoverflow.com/questions/55506952/how-to-get-url-for-core-site-xml-and-hdfs-site-xml-in-mesos-dc-os-hdfs-service
https://stackoverflow.com/questions/54819145/mounting-cgroup-to-rootfs-var-lib-docker-overlay2-at-sys-no-subsyste

[RogerWeihrauch]

Hi all
I made the experience that:
1st) HDFS only installs cleanly and successfully w/ more than 64GB since default options for each HDFS part needs a minimal amount of resources (pls see also default definitions while ^editing^ HDFS options before deployment).
2nd) pls see also: https://docs.mesosphere.com/services/spark/2.8.0-2.4.0/hdfs/ ->
http://api.hdfs.marathon.l4lb.thisdcos.directory/v1/endpoints as url for SPARK to access HDFS
3rd) I also have some problems w/ accessing HDFS from SPARK/to get SPARK up and running successfully together with HDFS/Gett access to SPARK`s webpage.
Donno why; still investigating on this.

Regars,
Roger