Uploaded image for project: 'Zeppelin'
  1. Zeppelin
  2. ZEPPELIN-5681

Zeppelin image can't use spark on prem

    XMLWordPrintableJSON

Details

    Description

      Hello there!

      I have kinda complex task - run zeppelin in docker at on prem cluster and make it possible to use on prem builded spark as a source for spark interpreter.

      My docker-compose file looks like this:

      version: '3'
      services: 
       zeppelin: 
         container_name: zeppelin10
         restart: always
         build: 
            context: /opt/docker/zeppelin/build/.
            dockerfile: /opt/docker/zeppelin/build/Dockerfile
         image: apache/zeppelin:0.10.0
         ports: 
           - "18080:8080"
           - "18443:8443"
           - "18081:8081"
           - "44000-44653:44000-44653"
           - "44655-45000:44655-45000"
         environment: 
           ZEPPELIN_PORT: 8080
           ZEPPELIN_SSL_PORT: 443
           ZEPPELIN_MEM: "-Xms2048m -Xmx2048m -XX:MaxMetaspaceSize=1024m"
           ZEPPELIN_JAVA_OPTS: "-Dspark.executor.memory=8g -Dspark.cores.max=16"
           ZEPPELIN_LOG_DIR: '/opt/zeppelin/logs'
           PATH: '/opt/conda/envs/python_3_with_R/bin:/opt/spark/bin:$PATH'    
         volumes: 
           - s_var-log_bind:/var/log
           - notebook_bind:/opt/zeppelin/notebook
           - /opt/docker/zeppelin/root/opt/zeppelin/conf:/opt/zeppelin/conf
           - /opt/docker/zeppelin/root/opt/packages:/opt/packages
           - /opt/docker/zeppelin/root/opt/delta-core:/opt/delta-core
           - /opt/docker/zeppelin/root/etc/hadoop/conf:/etc/hadoop/conf
           - spark_bind:/opt/spark
           - python_bind:/opt/python-3.7.9
           - /home/zeppelin:/home/zeppelin
      
      volumes: 
        spark_bind: 
          driver_opts: 
            type: "none"
            o: "bind"
            device: "/opt/docker/zeppelin/root/opt/spark"
        notebook_bind: 
          driver_opts: 
            type: "none"
            o: "bind"
            device: "/opt/docker/zeppelin/root/opt/zeppelin/notebook"
        log_bind: 
          driver_opts: 
            type: "none"
            o: "bind"
            device: "/opt/docker/zeppelin/root/opt/zeppelin/logs"
        s_var-log_bind: 
          driver_opts: 
            type: "none"
            o: "bind"
            device: "/opt/docker/zeppelin/root/var/log"
        python_bind: 
          driver_opts: 
            type: "none"
            o: "bind"
            device: "/opt/docker/zeppelin/root/opt/python-3.7.9"
      

      Here is my interpreter for on prem spark mounted as volume:

      Here is zeppelin output when I'm trying to run default spark interpreter and my own:

      So it works in local mode but not in client.

      Can you help?

      Attachments

        1. image-2022-03-16-10-41-54-901.png
          7 kB
          Dmitry Kravchuk
        2. image-2022-03-16-09-19-19-093.png
          55 kB
          Dmitry Kravchuk
        3. image-2022-03-16-09-18-24-581.png
          27 kB
          Dmitry Kravchuk

        Activity

          People

            Unassigned Unassigned
            dishka_krauch Dmitry Kravchuk
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: