Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3324

YARN module has nonstandard structure which cause compile error In IntelliJ

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Won't Fix
    • 1.1.0
    • None
    • YARN
    • Mac OS: 10.9.4
      IntelliJ IDEA: 13.1.4
      Scala Plugins: 0.41.2
      Maven: 3.0.5

    Description

      The YARN module has nonstandard path structure like:

      ${SPARK_HOME}
        |--yarn
           |--alpha (contains yarn api support for 0.23 and 2.0.x)
           |--stable (contains yarn api support for 2.2 and later)
           |     |--pom.xml (spark-yarn)
           |--common (Common codes not depending on specific version of Hadoop)
           |--pom.xml (yarn-parent)
      

      When we use maven to compile yarn module, maven will import 'alpha' or 'stable' module according to profile setting.
      And the submodule like 'stable' use the build propertie defined in yarn/pom.xml to import common codes to sourcePath.
      It will cause IntelliJ can't directly recognize sources in common directory as sourcePath.

      I thought we should change the yarn module to a unified maven jar project,
      and add specify different version of yarn api via maven profile setting.

      It will resolve the compile error in IntelliJ and make the yarn module more simple and clear.

      Attachments

        Activity

          People

            pwendell Patrick Wendell
            tianyi Tianyi Wang
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: