Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28776

SparkML MLWriter gets hadoop conf from spark context instead of session

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.4.3
    • Fix Version/s: 3.0.0
    • Component/s: MLlib
    • Labels:
      None

      Description

      In handleOverwrite of MLWriter, the hadoop configuration of the spark context is used where as the hadoop configuration of the spark session's session state should be used instead. https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala#L677

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                heleny Helen Yu
                Reporter:
                heleny Helen Yu
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 1h
                  1h
                  Remaining:
                  Remaining Estimate - 1h
                  1h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified