Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-31182

PairRDD support aggregateByKeyWithinPartitions

Rank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Not A Problem
    • 3.1.0
    • None
    • ML, Spark Core
    • None

    Description

      When implementing `RobustScaler`, I was looking for a way to guarantee that the QuantileSummaries in {{aggregateByKey}} are compressed before network communication.

      Then I only found a tricky method to work around (however not applied), and there was no method for this.

      Attachments

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            podongfeng Ruifeng Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment