Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30502

PeriodicRDDCheckpointer supports storageLevel

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • ML, Spark Core
    • None

    Description

      Intermediate RDDs in ML are cached with storageLevel=StorageLevel.MEMORY_AND_DISK.

      PeriodicRDDCheckpointer will store RDD with storageLevel=StorageLevel.MEMORY_ONLY, it maybe nice to set the storageLevel of checkpointer.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-39398 when doing iteration compute in graphx, checkpoint need support storagelevel

          • Major - Major loss of function.
          • In Progress
          links to

          Web Link GitHub Pull Request #27189

          Activity

            People

              podongfeng Ruifeng Zheng
              podongfeng Ruifeng Zheng
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: