Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25286

Remove dangerous parmap

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.0
    • 2.4.0
    • Spark Core
    • None

    Description

      One of parmap methods accepts an execution context created outside of parmap. If the parmap method is called recursively on a thread pool limited by size, it could lead to deadlocks. See the JIRA tickets: SPARK-25240 and SPARK-25283 . To eliminate the problems in the future, need to remove parmap() with the signature:

      def parmap[I, O, Col[X] <: TraversableLike[X, Col[X]]]
      (in: Col[I])
      (f: I => O)
      (implicit
        cbf: CanBuildFrom[Col[I], Future[O], Col[Future[O]]], // For in.map
        cbf2: CanBuildFrom[Col[Future[O]], O, Col[O]], // for Future.sequence
        ec: ExecutionContext
      ): Col[O]

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-25283 A deadlock in UnionRDD

          • Major - Major loss of function.
          • Resolved
          links to

          Pull request #22292 [Github] Pull Request #22292 (MaxGekk)

          Activity

            People

              maxgekk Max Gekk
              maxgekk Max Gekk
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: