Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-32008

3.0.0 release build fails

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Build, Documentation
    • None

    Description

      Hi,
      I try to build the spark release 3.0.0 by myself.

      I got the following error.

                                                
      20/06/16 15:20:49 WARN PrefixSpan: Input data is not cached.
      20/06/16 15:20:50 WARN Instrumentation: [b307b568] regParam is zero, which might cause numerical instability and overfitting.
      Error: processing vignette 'sparkr-vignettes.Rmd' failed with diagnostics:
      'vignetteInfo' is not an exported object from 'namespace:tools'
      Execution halted
      

      I can reproduce this error with a small Dockerfile.

      EDIT: Issue is solved with ubuntu 20.04 as base image.

      FROM ubuntu:20.04 as builder
      
      ENV MVN_VERSION=3.6.3 \
          M2_HOME=/opt/apache-maven \
          MAVEN_HOME=/opt/apache-maven \
          MVN_HOME=/opt/apache-maven \
          MVN_SHA512=c35a1803a6e70a126e80b2b3ae33eed961f83ed74d18fcd16909b2d44d7dada3203f1ffe726c17ef8dcca2dcaa9fca676987befeadc9b9f759967a8cb77181c0 \
          MAVEN_OPTS="-Xmx3g -XX:ReservedCodeCacheSize=1g" \
          R_HOME=/usr/lib/R \
          GIT_REPO=https://github.com/apache/spark.git \
          GIT_BRANCH=v3.0.0 \
          SPARK_DISTRO_NAME=hadoop3.2 \
          SPARK_LOCAL_HOSTNAME=localhost
      
      # Preparation
      RUN /usr/bin/apt-get update && \
          # APT
          INSTALL_PKGS="openjdk-8-jdk-headless git wget python3 python3-pip python3-setuptools r-base r-base-dev pandoc pandoc-citeproc libcurl4-openssl-dev libssl-dev libxml2-dev texlive qpdf language-pack-en" && \
          DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get -y install --no-install-recommends $INSTALL_PKGS && \
          rm -rf /var/lib/apt/lists/* && \
          Rscript -e "install.packages(c('knitr', 'rmarkdown', 'devtools', 'testthat', 'e1071', 'survival'), repos='https://cloud.r-project.org/')" && \
          # Maven
          /usr/bin/wget -nv -O apache-maven.tar.gz "https://www.apache.org/dyn/mirrors/mirrors.cgi?action=download&filename=maven/maven-3/${MVN_VERSION}/binaries/apache-maven-${MVN_VERSION}-bin.tar.gz" && \
          echo "${MVN_SHA512} apache-maven.tar.gz" > apache-maven.sha512 && \
          sha512sum --strict -c apache-maven.sha512 && \
          tar -xvzf apache-maven.tar.gz -C /opt && \
          rm -v apache-maven.sha512 apache-maven.tar.gz && \
          /bin/ln -vs /opt/apache-maven-${MVN_VERSION} /opt/apache-maven && \
          /bin/ln -vs /opt/apache-maven/bin/mvn /usr/bin/mvn
      
      # Spark Distribution Build
      RUN mkdir -p /workspace && \
          cd /workspace && \
          git clone --branch ${GIT_BRANCH} ${GIT_REPO} && \
          cd /workspace/spark && \
          ./dev/make-distribution.sh --name ${SPARK_DISTRO_NAME} --pip --r --tgz -Psparkr -Phadoop-3.2 -Phive-2.3 -Phive-thriftserver -Pyarn -Pkubernetes
      

      I am very grateful to all helpers.

      Attachments

        Activity

          People

            Unassigned Unassigned
            Reamer Philipp Dallig
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: