[SPARK-45386] Correctness issue when persisting using StorageLevel.NONE - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.5.0, 4.0.0
Fix Version/s: 3.5.1
Component/s: SQL
Labels:
- correctness
- pull-request-available

Description

When using spark 3.5.0 this code

import org.apache.spark.storage.StorageLevel
spark.createDataset(Seq(1,2,3)).persist(StorageLevel.NONE).count()

incorrectly returns 0.

Attachments

Issue Links

links to

GitHub Pull Request #43188

GitHub Pull Request #43213

Activity

People

Assignee:: Emil Ejbyfeldt

Reporter:: Emil Ejbyfeldt

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 30/Sep/23 09:31

Updated:: 24/Nov/23 22:28

Resolved:: 05/Oct/23 00:41