Repository navigation
Conversation
|
Test build #105171 has finished for PR 24541 at commit
|
There was a problem hiding this comment.
@aokolnychyi this test is failing because of type erasure. To be fixed.
There was a problem hiding this comment.
@dbtsai, friendly reminder, looks like we are waiting for a fix here and next test still.
|
Test build #105173 has finished for PR 24541 at commit
|
|
@dbtsai why is this necessary? Can't you just copy ClosureCleaner's code? That class has been copied by almost every Scala project that deals with closure shipping. |
abellina
left a comment
There was a problem hiding this comment.
I have yet to look at schemaFor, and also changes in org.javassist, will do shortly.
|
|
||
| import org.apache.spark.{SparkEnv, SparkException} | ||
| import org.apache.spark.internal.Logging | ||
| import org.apache.spark.util.Utils.getSerializedLambda |
There was a problem hiding this comment.
since we are already in org.apacke.spark.util, could we just call Utils.getSerializedLambda from the clean? We seem to refer to other static functions in Utils using the classname.
| } | ||
|
|
||
| test("getSerializedLambda") { | ||
| // TODO: adds more tests |
There was a problem hiding this comment.
are you adding more tests in this PR? If so, maybe remove and add a jira to add more tests?
There was a problem hiding this comment.
@dbtsai, friendly reminder, looks like we are waiting for a fix here and next test still.
|
Overall it looks good to me, once the failing tests are fixed and the TODO are addressed like @abellina called out. |
|
cc @rednaxelafx |
|
@abellina gentle ping |
|
I think you meant you mean @dbtsai |
What changes were proposed in this pull request?
This PR is the part of the essential changes in Spark to make [SPARK-14083] Basic bytecode analyzer to speed up Datasets work as an external module without copying and pasting.
getSerializedLambdapublic so bytecode analysis can use the same implementation.def schemaFor[T](clazz: Class[T]): Schemais implemented in this PR.org.javassistwhich Spark implicitly depends on from Jersey is now an explicit dependency and pined to latest version.This PR is based on #24515 with more tests; it's a collaboration work with @aokolnychyi
How was this patch tested?
More tests are added.