[SPARK-26506] RegressionMetrics fails in Spark 2.4 - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Cannot Reproduce
Affects Version/s: 2.4.0
Fix Version/s: None
Component/s: MLlib
Labels:
None
Environment:

Hide

Windows using the Anaconda stack running Spark 2.4, using Java jdk 1.8.0_181. It may also affect unix when running Spark 2.4, not sure because my workplace where I use Spark in unix is still on Spark 2.2.

The bug does not appear cause an error in either 2.3 or 2.2 on either windows or unix.

Show
Windows using the Anaconda stack running Spark 2.4, using Java jdk 1.8.0_181. It may also affect unix when running Spark 2.4, not sure because my workplace where I use Spark in unix is still on Spark 2.2. The bug does not appear cause an error in either 2.3 or 2.2 on either windows or unix.

Description

RegressionMetrics fails in Spark 2.4 when running via Anaconda on a Windows machine. A java error comes back saying that "python worker failed to connect back". This makes all the evaluation metrics (https://spark.apache.org/docs/2.2.0/mllib-evaluation-metrics.html) unusable for scoring model performance.

Reverted to Spark 2.3 and did not have this issue, also tested 2.2 and did not have this issue. So it appears to be a bug specific to Spark 2.4. Likely also affects other evaluation metric types, e.g.BinaryClassificationMetrics.

Attachments

Activity

People

Assignee:: Unassigned

Reporter:: Casey Bennett

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 30/Dec/18 21:24

Updated:: 01/Mar/19 23:09

Resolved:: 01/Mar/19 23:09