Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-25371

Vector Assembler with no input columns leads to opaque error

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 2.3.0, 2.3.1
    • 2.3.2, 2.4.0
    • ML, MLlib
    • None

    Description

      When `VectorAssembler ` is given an empty array as its inputColumns it throws an opaque error. In versions less than 2.3 `VectorAssembler` it simply appends a column containing empty vectors. 

       

      val inputCols = Array()
      val outputCols = Array("A")
      val vectorAssembler = new VectorAssembler()
      .setInputCols(inputCols)
      .setOutputCol(outputCols)
      
      vectorAssmbler.fit(data).transform(df)
      
      

      In versions 2.3 > this throws the exception below

      org.apache.spark.sql.AnalysisException: cannot resolve 'named_struct()' due to data type mismatch: input to function named_struct requires at least one argument;;
      

      Whereas in versions less than 2.3 it just adds a column containing an empty vector.

      I'm not certain if this is an intentional choice or an actual bug. If this is a bug, the `VectorAssembler` should be modified to append an empty vector column if it detects no inputCols.

       

      If it is a design decision it would be nice to throw a human readable exception explicitly stating inputColumns must not be empty. The current error is somewhat opaque.

      Attachments

        Activity

          People

            mgaido Marco Gaido
            Odessa Victor Alor
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: