Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.1.1
Description
The below code generates a dataframe successfully . Here make_date function is used inside a select expression
from pyspark.sql.functions import expr, make_date
df = spark.createDataFrame([(2020, 6, 26), (1000, 2, 29), (-44, 1, 1)],['Y', 'M', 'D'])
df.select("*",expr("make_date(Y,M,D) as lk")).show()
The below code fails with a message "cannot import name 'make_date' from 'pyspark.sql.functions'" . Here the make_date function is directly called on dataframe columns without select expression
from pyspark.sql.functions import make_date
df = spark.createDataFrame([(2020, 6, 26), (1000, 2, 29), (-44, 1, 1)],['Y', 'M', 'D'])
df.select(make_date(df.Y,df.M,df.D).alias("datefield")).show()
The error message generated is misleading when it says "cannot import make_date from pyspark.sql.functions"