Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve application start API #7

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

kmaehashi
Copy link
Member

This patch improves the Jubatus-on-YARN API to start the Jubatus cluster.

@rimms
Copy link
Member

rimms commented Oct 21, 2015

I faced compile error:

[info] Set current project to jubatus-on-yarn-client (in build file:/path/to/jubatus-on-yarn/jubatusonyarn/)
[info] Compiling 7 Scala sources to /path/to/jubatus-on-yarn/jubatusonyarn/jubatus-on-yarn-client/target/scala-2.10/classes...
[error] /path/to/jubatus-on-yarn/jubatusonyarn/jubatus-on-yarn-client/src/main/scala/us/jubat/yarn/client/Model.scala:31: overloaded method value start with alternatives:
[error]   (aLearningMachineName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigFile: org.apache.hadoop.fs.Path,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aBasePath: org.apache.hadoop.fs.Path,aApplicationName: String)scala.concurrent.Future[us.jubat.yarn.client.JubatusYarnApplication] <and>
[error]   (aLearningMachineName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigFile: org.apache.hadoop.fs.Path,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aApplicationName: String)scala.concurrent.Future[us.jubat.yarn.client.JubatusYarnApplication] <and>
[error]   (aLearningMachineName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigString: String,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aBasePath: org.apache.hadoop.fs.Path,aApplicationName: String)scala.concurrent.Future[us.jubat.yarn.client.JubatusYarnApplication] <and>
[error]   (aLearningMachineName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigString: String,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aApplicationName: String)scala.concurrent.Future[us.jubat.yarn.client.JubatusYarnApplication]
[error]  cannot be applied to (String, us.jubat.yarn.common.LearningMachineType.Classifier.type, List[us.jubat.yarn.common.Location], org.apache.hadoop.fs.Path, us.jubat.yarn.client.Resource, Int)
[error]   val tApplicationFuture = JubatusYarnApplication.start(
[error]                                                   ^
[error] /path/to/jubatus-on-yarn/jubatusonyarn/jubatus-on-yarn-client/src/main/scala/us/jubat/yarn/client/YarnClientServlet.scala:59: overloaded method value startJubatusApplication with alternatives:
[error]   (aName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigFile: org.apache.hadoop.fs.Path,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aBasePath: org.apache.hadoop.fs.Path,aApplicationName: String)us.jubat.yarn.client.ApplicationMasterProxy <and>
[error]   (aName: String,aLearningMachineType: us.jubat.yarn.common.LearningMachineType,aZookeepers: List[us.jubat.yarn.common.Location],aConfigString: String,aResource: us.jubat.yarn.client.Resource,aNodeCount: Int,aBasePath: org.apache.hadoop.fs.Path,aApplicationName: String)us.jubat.yarn.client.ApplicationMasterProxy
[error]  cannot be applied to (String, us.jubat.yarn.common.LearningMachineType, List[us.jubat.yarn.common.Location], String, us.jubat.yarn.client.Resource, Int, org.apache.hadoop.fs.Path)
[error]       mController.get.startJubatusApplication(tSettings.learningMachineInstanceName, tSettings.learningMachineType, tListZooKeeper, tSettings.configJson, tSettings.resources, tSettings.nodeCount, new Path(tSettings.basePath))
[error]                       ^
[error] two errors found
[error] (jubatus-on-yarn-client/compile:compile) Compilation failed

@kmaehashi kmaehashi force-pushed the add-yarn-application-name branch from e58715a to 5e8221d Compare November 18, 2015 11:58
@kmaehashi kmaehashi changed the title add interface to specify YARN application name Improve application start API Nov 18, 2015
@kmaehashi kmaehashi force-pushed the add-yarn-application-name branch from 5e8221d to 508105a Compare November 18, 2015 12:51
@kmaehashi kmaehashi force-pushed the add-yarn-application-name branch from 508105a to 74604f3 Compare November 18, 2015 12:59
@kmaehashi
Copy link
Member Author

I revised the contents of this PR.

  • YARN application name for JoY can now be specified externally.
  • The old (four overloaded) start APIs had too many arguments. I added new start API that accepts cluster configurations via one object, and deprecated old ones.
  • Previously, list of ZooKeepers are given as List[Location], however this was inappropriate, as ZooKeeper location string is a kind of DSL parsed by ZooKeeper client library. I fixed this to directly pass String instead.
  • Cluster launching options are now well-validated.
  • Removed /start HTTP API in YarnClientServlet, which was unused.

@kmaehashi kmaehashi force-pushed the add-yarn-application-name branch 4 times, most recently from 260aae7 to 88c2cb6 Compare November 19, 2015 01:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants