Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28933

Reduce unnecessary shuffle in ALS when initializing factors

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Resolved
    • 3.0.0
    • 3.0.0
    • ML
    • None

    Description

      When Initializing factors in ALS, we should use mapPartitions instead of current map, so we can preserve existing partition of the RDD of InBlock. The RDD of InBlock is already partitioned by src block id. We don't change the partition when initializing factors.

      Attachments

        Issue Links

          Activity

            People

              viirya L. C. Hsieh
              viirya L. C. Hsieh
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: