tweaks to the SparkLR example #872
Conversation
…tions. Also add an example of Kryo usage, and an optional "points" parameter.
|
Thank you for your pull request. An admin will review this request soon. |
|
Jenkins, this is ok to test |
|
Thank you for submitting this pull request. All automated tests for this request have passed. Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/764/ |
| kryo.setRegistrationRequired(true) | ||
|
|
||
| kryo.register(classOf[scala.collection.mutable.WrappedArray.ofRef[_]]) | ||
| kryo.register(classOf[java.lang.Class[_]]) |
There was a problem hiding this comment.
Hey Mike, why do you need to register WrappedArray and Class? Doesn't seem like they'll occur in our data here.
There was a problem hiding this comment.
I came up with that list by adding setRegistrationRequired, then fixing each exception that was thrown. So Kryo said they were used; I haven't considered why.
There was a problem hiding this comment.
I see. Did Kryo actually make a performance difference for you at all? We are not caching data in serialized form here, so it would only be used to send back task results, but that's just one Vector per partition. I think the WrappedArrays are because we somehow send that back within an array.
Replace fold() with aggregate() in SparkLR, to avoid Vector instatiations.
|
Thank you for submitting this pull request. All automated tests for this request have passed. Refer to this link for build results: http://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/809/ |
| (1 / (1 + exp(-p.y * (w dot p.x))) - 1) * p.y * p.x | ||
| }.reduce(_ + _) | ||
| ((1 / (1 + exp(-p.y * (w dot p.x))) - 1) * p.y, p.x) | ||
| }.aggregate(zero)((sum,v) => sum saxpy (v._1,v._2), _ += _) |
There was a problem hiding this comment.
Change the second argument to sum.saxpy(v._1, v._2); we don't use infix notation for methods unless they're operators.
|
Sorry for the long delay, just made a couple more comments. I'm happy to merge this otherwise. You can either update it here or send a new PR against the apache/incubator-spark repo (the latter would be slightly easier). |
This sets the max line length to 100 as a PEP8 exception. Author: Reynold Xin <[email protected]> Closes mesos#872 from rxin/pep8 and squashes the following commits: 2f26029 [Reynold Xin] Added PEP8 style configuration file.
I've been using
spark.examples.SparkLRfor performance testing. Just passing along some enhancements to the class.