Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.4.1, 2.4.3
-
None
-
Databricks cluster:
{
{ "spark.databricks.delta.preview.enabled": "true" }
"num_workers": 4,
"cluster_name": "mabedfor-test-classfix",
"spark_version": "5.3.x-cpu-ml-scala2.11",
"spark_conf":,
{ "PYSPARK_PYTHON": "/databricks/python3/bin/python3" }
"node_type_id": "Standard_DS12_v2",
"driver_node_type_id": "Standard_DS12_v2",
"ssh_public_keys": [],
"custom_tags": {},
"spark_env_vars":,
"autotermination_minutes": 120,
"enable_elastic_disk": true,
"cluster_source": "UI",
"init_scripts": [],
"cluster_id": "0722-165622-calls746"
}Databricks cluster: { "num_workers": 4, "cluster_name": "mabedfor-test-classfix", "spark_version": "5.3.x-cpu-ml-scala2.11", "spark_conf": { "spark.databricks.delta.preview.enabled": "true" } , "node_type_id": "Standard_DS12_v2", "driver_node_type_id": "Standard_DS12_v2", "ssh_public_keys": [], "custom_tags": {}, "spark_env_vars": { "PYSPARK_PYTHON": "/databricks/python3/bin/python3" } , "autotermination_minutes": 120, "enable_elastic_disk": true, "cluster_source": "UI", "init_scripts": [], "cluster_id": "0722-165622-calls746" }
Description
Right after a CrossValidatorModel is trained, it has avgMetrics. After the model is written to disk and read later, it no longer has avgMetrics. To reproduce:
from pyspark.ml.tuning import CrossValidator, CrossValidatorModel
cv = CrossValidator(...) #fill with params
cvModel = cv.fit(trainDF) #given dataframe with training data
print(cvModel.avgMetrics) #prints a nonempty list as expected
cvModel.write().save("/tmp/model")
cvModel2 = CrossValidatorModel.read().load("/tmp/model")
print(cvModel2.avgMetrics) #BUG - prints an empty list
Attachments
Issue Links
- links to