[SPARK-47445] Adding new 'Silent' ExplainMode - ASF JIRA

Log work

Agile Board

Rank to Top

Rank to Bottom

Attach files

Attach Screenshot

Bulk Copy Attachments

Bulk Move Attachments

Add vote

Voters

Watch issue

Watchers

Create sub-task

Convert to sub-task

Move

Link

Clone

Labels

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

Delete

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Open
Priority: Minor
Resolution: Unresolved
Affects Version/s: 4.0.0
Fix Version/s: None
Component/s: Connect, Documentation, PySpark, SQL
Labels:
- pull-request-available

Description

While investigating unit test duration we found that org.apache.spark.sql.execution.QueryExecution.explainString () takes approximately 14% time. This method generates the string representation of the execution plan. The string is often used for logging purposes. This is also called for each AQE job so it can save prod execution time too. While ~~SPARK-44485~~ does exist to help optimize the prod execution time, the main purpose of this PR is to save time during unit testing.

I've added a silent mode to ExplainMode to try and mitigate this issue.