Add _toArrow() DataFrame method to PySpark

Over in the Apache Arrow community, we hear from a lot of users who want to return the contents of a PySpark DataFrame as a PyArrow Table. Currently the only documented way to do this is:

PySpark DataFrame --> pandas DataFrame --> PyArrow Table

This adds significant overhead compared to going direct from PySpark DataFrame to PyArrow Table. Since PySpark already goes through PyArrow to convert to pandas, would it be possible to publicly expose an experimental _toArrow() method of the Spark DataFrame class?