Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28776

SparkML MLWriter gets hadoop conf from spark context instead of session

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.4.3
    • 3.0.0
    • MLlib
    • None

    Description

      In handleOverwrite of MLWriter, the hadoop configuration of the spark context is used where as the hadoop configuration of the spark session's session state should be used instead. https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/util/ReadWrite.scala#L677

      Attachments

        Issue Links

          Activity

            People

              heleny Helen Yu
              heleny Helen Yu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - 1h
                  1h
                  Remaining:
                  Remaining Estimate - 1h
                  1h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified