Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Invalid
-
3.0.0
-
None
-
None
-
Databricks runtime 7.3 LTS ML
Description
Executing following code on databricks runtime 7.3 LTS ML errors out showing some rbind error whereas it is successfully executed without enabling Arrow in Spark session. Full error message attached.
```
library(dplyr)
library(SparkR)
SparkR::sparkR.session(sparkConfig = list(spark.sql.execution.arrow.sparkr.enabled = "true"))
mtcars %>%
SparkR::as.DataFrame() %>%
SparkR::gapply(x = .,
cols = c("cyl", "vs"),
func = function(key,
data)
,
schema = structType(structField("cyl", "double"),
structField("vs", "double"),
structField("mpg_mean", "double"),
structField("qsec_mean", "double"))
) %>%
display()
```