Details
Minor Description
It would be very helpful if it were possible to specify the format for individual columns in a schema when reading csv files, rather than one format:
Bar.python
# Currently can only do something like: spark.read.option("dateFormat", "yyyyMMdd").csv(...) # Would like to be able to do something like: schema = StructType([ StructField("date1", DateType(format="MM/dd/yyyy"), True), StructField("date2", DateType(format="yyyyMMdd"), True) ] read.schema(schema).csv(...)
Thanks for any help, input!