Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-16210

DataFrame.drop(colName) fails if another column has a period in its name

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Cannot Reproduce
    • 1.6.1
    • None
    • SQL
    • Spark 1.6.1 on Databricks

    Description

      The following code fails with org.apache.spark.sql.AnalysisException: cannot resolve 'x.y' given input columns: [abc, x.y] because of the way drop() uses select() under the covers.

      val rdd = sc.makeRDD("""{"x.y": 5, "abc": 10}""" :: Nil)
      sqlContext.read.json(rdd).drop("abc")
      

      Attachments

        Activity

          People

            Unassigned Unassigned
            simeons Simeon Simeonov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: