Major Description
Livy creates a fake spark shell initiating its own java gateway. This gateway imports some java classes for the user which have diverged from the ones in Spark 3. Some imports are missing which lead to methods like `dataframe.explain()` to be broken:
result.explain(True) An error was encountered: 'JavaPackage' object is not callable Traceback (most recent call last): File "/srv/hops/spark/python/lib/pyspark.zip/pyspark/sql/dataframe.py", line 386, in explain print(self._sc._jvm.PythonSQLUtils.explainString(self._jdf.queryExecution(), explain_mode)) TypeError: 'JavaPackage' object is not callable
Spark is importing:
https://github.com/apache/spark/blob/87bf6b0ea4ca0618c8604895d05037edce8b7cb0/python/pyspark/java_gateway.py#L153
Livy is currently importing:
https://github.com/apache/incubator-livy/blob/4d8a912699683b973eee76d4e91447d769a0cb0d/repl/src/main/resources/fake_shell.py#L581
