Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Cannot Reproduce
-
2.3.0
-
None
-
None
Description
final_store.coalesce(5).write.mode("overwrite").insertInto("database.tablename",overwrite = True), this statement is not merging partitions. I've setÂ
.config("spark.default.parallelism", "2000") \
.config("spark.sql.shuffle.partitions", "2000") \
however repartition is working but takes 20-25 minutes to insert.