Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
zeppelin-server tests in ZeppelinSparkClusterTest consistently failing on pySpark part
-----------------
TODOs:
- General
- force CI to use [container-based infrastructure](https://docs.travis-ci.com/user/workers/container-based-infrastructure/#Routing-your-build-to-container-based-infrastructure) - #808
- PR 208
- remove SUDO everywhere from .travis.yaml
- to install R+dependencies follow https://docs.travis-ci.com/user/migrating-from-legacy/#How-do-I-install-APT-sources-and-packages%3F
-------------------
Example https://travis-ci.org/bzz/incubator-zeppelin/builds/119447025
- https://api.travis-ci.org/jobs/119447027/log.txt?deansi=true
- https://api.travis-ci.org/jobs/119447028/log.txt?deansi=true
Failed tests: ZeppelinSparkClusterTest.pySparkDepLoaderTest:167->getSparkVersionNumber:220 expected:<FINISHED> but was:<ERROR> ZeppelinSparkClusterTest.pySparkAutoConvertOptionTest:112->getSparkVersionNumber:220 expected:<FINISHED> but was:<ERROR> ZeppelinSparkClusterTest.basicRDDTransformationAndActionTest:81 expected:<FINISHED> but was:<ERROR> ZeppelinSparkClusterTest.pySparkTest:90->getSparkVersionNumber:220 expected:<FINISHED> but was:<ERROR> ZeppelinSparkClusterTest.zRunTest:152 expected:<FINISHED> but was:<ERROR> Tests run: 35, Failures: 5, Errors: 0, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Zeppelin: Interpreter .............................. SUCCESS [ 11.901 s] [INFO] Zeppelin: Zengine .................................. SUCCESS [ 3.691 s] [INFO] Zeppelin: Display system apis ...................... SUCCESS [ 2.158 s] [INFO] Zeppelin: Spark dependencies ....................... SUCCESS [ 30.747 s] [INFO] Zeppelin: Spark .................................... SUCCESS [ 2.912 s] [INFO] Zeppelin: Server ................................... FAILURE [01:52 min] [INFO] Zeppelin: R Interpreter ............................ SKIPPED [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE
Two things are suspicious in the logs:
a) IndexOutOfBoundsException (but may be harmless)
b) Spark Interpreter is not able to communicate to SparkMaster
A. IndexOutOfBoundsException
06:28:14,435 ERROR org.apache.zeppelin.rest.NotebookRestApi:405 - Exception in NotebookRestApi while moveParagraph java.lang.IndexOutOfBoundsException: paragraph size is 2 , index is 10 at org.apache.zeppelin.notebook.Note.moveParagraph(Note.java:274) at org.apache.zeppelin.rest.NotebookRestApi.moveParagraph(NotebookRestApi.java:399) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
B. Spark Interpreter is not able to communicate to SparkMaster
16/03/30 06:28:24 INFO SparkContext: Added JAR file:/home/travis/build/bzz/incubator-zeppelin/interpreter/spark/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar at http://10.128.0.211:47567/jars/zeppelin-spark-0.6.0-incubating-SNAPSHOT.jar with timestamp 1459319304855 16/03/30 06:28:24 INFO FairSchedulableBuilder: Created default pool default, schedulingMode: FIFO, minShare: 0, weight: 1 16/03/30 06:28:24 WARN MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set. 16/03/30 06:28:24 INFO AppClient$ClientEndpoint: Connecting to master spark://testing-gce-3dcf1a9b-ea54-400d-b356-693b37a2edd4.c.travis-ci-prod-5.internal:7071... 16/03/30 06:28:44 ERROR SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[appclient-registration-retry-thread,5,main] java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@f431bb4 rejected from java.util.concurrent.ThreadPoolExecutor@7ecfce3f[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 0] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)