Details
-
New Feature
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
3.1.0
-
None
-
None
Description
Add support for StreamingQueryListener to PySpark.
Currently the `StreamingQueryListener` in Scala is implemented as an abstract class, so we cannot use Python proxies (Py4j) to access it unless we create our own custom Scala/Java wrapper.
This would be very useful in my personal case, I am building a library that allows you to send Python errors to Sentry.io https://docs.sentry.io/platforms/python/pyspark/ and would like to hook onto onQueryTerminated to send errors.
I can take this on if you point me in which direction to go, new to the codebase so not quite sure what the process for porting Scala API -> PySpark API changes usually look like.
Attachments
Issue Links
- is fixed by
-
SPARK-38759 Add StreamingQueryListener support in PySpark
- Resolved