Details
-
Bug
-
Status: Resolved
-
Trivial
-
Resolution: Fixed
-
2.1.1
-
None
Description
i was looking over the dataset and noticed the code on top that does not make sense in java.
// To create Dataset<Row> using SparkSession Dataset<Row> people = spark.read().parquet("..."); Dataset<Row> department = spark.read().parquet("..."); people.filter("age".gt(30)) .join(department, people.col("deptId").equalTo(department("id"))) .groupBy(department.col("name"), "gender") .agg(avg(people.col("salary")), max(people.col("age")));
invalid parts:
- "age".gt(30)
- department("id")