Skip to content

[SPARK-27646] [SQL] [WIP] Required refactoring for bytecode analysis - #24541

Closed
dbtsai wants to merge 1 commit into
apache:masterfrom
dbtsai:deps
Closed

dbtsai wants to merge 1 commit into
apache:masterfrom
dbtsai:deps

Conversation

@dbtsai

@dbtsai dbtsai commented May 6, 2019 •

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

This PR is the part of the essential changes in Spark to make [SPARK-14083] Basic bytecode analyzer to speed up Datasets work as an external module without copying and pasting.

  1. Makes getSerializedLambda public so bytecode analysis can use the same implementation.
  2. def schemaFor[T](clazz: Class[T]): Schema is implemented in this PR.
  3. org.javassist which Spark implicitly depends on from Jersey is now an explicit dependency and pined to latest version.

This PR is based on #24515 with more tests; it's a collaboration work with @aokolnychyi

How was this patch tested?

More tests are added.

@SparkQA

SparkQA commented May 6, 2019

Copy link
Copy Markdown

Test build #105171 has finished for PR 24541 at commit a09ead7.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aokolnychyi this test is failing because of type erasure. To be fixed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dbtsai, friendly reminder, looks like we are waiting for a fix here and next test still.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

@SparkQA

SparkQA commented May 6, 2019

Copy link
Copy Markdown

Test build #105173 has finished for PR 24541 at commit 197bcfd.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@rxin

rxin commented May 8, 2019

Copy link
Copy Markdown
Contributor

@dbtsai why is this necessary? Can't you just copy ClosureCleaner's code? That class has been copied by almost every Scala project that deals with closure shipping.

@abellina abellina left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have yet to look at schemaFor, and also changes in org.javassist, will do shortly.


import org.apache.spark.{SparkEnv, SparkException}
import org.apache.spark.internal.Logging
import org.apache.spark.util.Utils.getSerializedLambda

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we are already in org.apacke.spark.util, could we just call Utils.getSerializedLambda from the clean? We seem to refer to other static functions in Utils using the classname.

}

test("getSerializedLambda") {
// TODO: adds more tests

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you adding more tests in this PR? If so, maybe remove and add a jira to add more tests?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dbtsai, friendly reminder, looks like we are waiting for a fix here and next test still.

@revans2

revans2 commented May 24, 2019

Copy link
Copy Markdown
Contributor

Overall it looks good to me, once the failing tests are fixed and the TODO are addressed like @abellina called out.

@gatorsmile

Copy link
Copy Markdown
Member

cc @rednaxelafx

@kiszk

kiszk commented Jun 15, 2019

Copy link
Copy Markdown
Member

@abellina gentle ping

@tgravescs

Copy link
Copy Markdown
Contributor

I think you meant you mean @dbtsai

@dbtsai dbtsai closed this Nov 11, 2019
@dbtsai
dbtsai deleted the deps branch November 11, 2019 23:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants