Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30701

SQL test running on Windows: hadoop chgrp warnings

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Not A Problem
    • 2.4.4
    • None
    • SQL

    Description

      Running SparkSQL local embedded unit tests on Win10, using winutils.

      Got warnings about 'hadoop chgrp'.

      See environment info.

      -chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...
-chgrp: 'TEST\Domain users' does not match expected pattern for group
Usage: hadoop fs [generic options] -chgrp [-R] GROUP PATH...

      Related info on SO: https://stackoverflow.com/questions/48605907/error-in-pyspark-when-insert-data-in-hive
      Seems like the problem is here: hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FsShellPermissions.java:210

      Attachments

        1. HadoopGroupTest.java
          1 kB
          Guram Savinov

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-16837 Spark-SQL test running on Windows: hadoop chgrp warnings

          • Major - Major loss of function.
          • Open
          is a clone of

          Bug - A problem which impairs or prevents the functions of the product. SPARK-20625 SQL test running on Windows hadoop command warning

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              gsavinov Guram Savinov
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: