Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25989

OneVsRestModel handle empty outputCols incorrectly

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • ML
    • None

    Description

      ml.classification.ClassificationModel will ignore empty output columns.

      However, OneVsRestModel still try to append new column even if its name is an empty string.

      
      scala> ovrModel.setPredictionCol("").transform(test).show
      +-----+--------------------+--------------------+---+
      |label| features| rawPrediction| |
      +-----+--------------------+--------------------+---+
      | 0.0|(4,[0,1,2,3],[-0....|[-0.0965652626152...|2.0|
      | 0.0|(4,[0,1,2,3],[-0....|[0.07880609384635...|2.0|
      | 0.0|(4,[0,1,2,3],[-1....|[0.01891571586984...|2.0|
      | 0.0|(4,[0,1,2,3],[0.1...|[0.72409973016524...|0.0|
      | 0.0|(4,[0,1,2,3],[0.1...|[0.48045978946729...|2.0|
      | 0.0|(4,[0,1,2,3],[0.3...|[1.05496616040758...|0.0|
      | 0.0|(4,[0,1,2,3],[0.3...|[0.79508659065535...|0.0|
      | 0.0|(4,[0,1,2,3],[0.6...|[1.47437469552081...|0.0|
      | 0.0|(4,[0,1,2,3],[0.6...|[1.23302929670223...|0.0|
      | 0.0|(4,[0,1,2,3],[0.8...|[1.79816156359706...|0.0|
      | 1.0|(4,[0,1,2,3],[-0....|[-3.1564309664080...|1.0|
      | 1.0|(4,[0,1,2,3],[-0....|[-3.2217906250571...|1.0|
      | 1.0|(4,[0,1,2,3],[-0....|[-2.9171126308553...|1.0|
      | 1.0|(4,[0,1,2,3],[-0....|[-2.8316993051998...|1.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-1.6486206847760...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-0.9252139721697...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-0.9025379528484...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-0.8518243169707...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-1.0990190524225...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-0.9973479746889...|2.0|
      +-----+--------------------+--------------------+---+
      only showing top 20 rows
      
      
      scala> ovrModel.setPredictionCol("").setRawPredictionCol("raw").transform(test).show
      +-----+--------------------+--------------------+---+
      |label| features| raw| |
      +-----+--------------------+--------------------+---+
      | 0.0|(4,[0,1,2,3],[-0....|[-0.0965652626152...|2.0|
      | 0.0|(4,[0,1,2,3],[-0....|[0.07880609384635...|2.0|
      | 0.0|(4,[0,1,2,3],[-1....|[0.01891571586984...|2.0|
      | 0.0|(4,[0,1,2,3],[0.1...|[0.72409973016524...|0.0|
      | 0.0|(4,[0,1,2,3],[0.1...|[0.48045978946729...|2.0|
      | 0.0|(4,[0,1,2,3],[0.3...|[1.05496616040758...|0.0|
      | 0.0|(4,[0,1,2,3],[0.3...|[0.79508659065535...|0.0|
      | 0.0|(4,[0,1,2,3],[0.6...|[1.47437469552081...|0.0|
      | 0.0|(4,[0,1,2,3],[0.6...|[1.23302929670223...|0.0|
      | 0.0|(4,[0,1,2,3],[0.8...|[1.79816156359706...|0.0|
      | 1.0|(4,[0,1,2,3],[-0....|[-3.1564309664080...|1.0|
      | 1.0|(4,[0,1,2,3],[-0....|[-3.2217906250571...|1.0|
      | 1.0|(4,[0,1,2,3],[-0....|[-2.9171126308553...|1.0|
      | 1.0|(4,[0,1,2,3],[-0....|[-2.8316993051998...|1.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-1.6486206847760...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-0.9252139721697...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-0.9025379528484...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-0.8518243169707...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-1.0990190524225...|2.0|
      | 2.0|(4,[0,1,2,3],[-0....|[-0.9973479746889...|2.0|
      +-----+--------------------+--------------------+---+
      only showing top 20 rows
      

      Attachments

        Activity

          People

            podongfeng Ruifeng Zheng
            podongfeng Ruifeng Zheng
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: