Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
2.2.0
-
None
Description
TaskMetrics._updatedBlockStatuses isn't used anywhere internally by spark. It could be used by users though since its exposed by SparkListenerTaskEnd. We made it configurable to turn off the tracking of it since it uses a lot of memory in https://issues.apache.org/jira/browse/SPARK-20923. That config is still true for backwards compatibility. We should turn that to false in next release and deprecate that api altogether.
Attachments
Issue Links
- relates to
-
SPARK-20923 TaskMetrics._updatedBlockStatuses uses a lot of memory
- Resolved