Native Test Case #35 Fails in RHEL 8.4

When running the following maven command, Native Test Case #35 Fails and the tests are halted.

mvn test -Pnative,parallel-tests,yarn-ui -Dparallel-tests=true -Dtests=allNative -Drequire.bzip2=true -Drequire.fuse=true -Drequire.isal=true -Disal.prefix=/usr/local -Disal.lib=/usr/local/lib64 -Dbundle.isal=true -Drequire.openssl=true -Dopenssl.prefix=/usr -Dopenssl.include=/usr/include -Dopenssl.lib=/usr/lib64 -Dbundle.openssl=true -Drequire.pmdk=true -Dpmdk.lib=/usr/lib64 -Dbundle.pmdk=true -Drequire.snappy=true -Dsnappy.prefix=/usr -Dsnappy.include=/usr/include -Dsnappy.lib=/usr/lib64 -Dbundle.snappy=true -Drequire.valgrind=true -Dhbase.profile=2.0 -Drequire.zstd=true -Dzstd.prefix=/usr -Dzstd.include=/usr/include -Dzstd.lib=/usr/lib64 -Dbundle.zstd=true 

I added the following to assist with the debugging:

-Dnative_ctest_args="-V -VV --debug --stop-on-failure" -Dnative_cmake_args="-D_LIBHDFS_JNI_HELPER_DEBUGGING_ON_ -DLIBHDFSPP_C_API_ENABLE_DEBUG" -Droot.log.level=DEBUG

After much work tracing through Test Case #35 (test_libhdfs_threaded_hdfspp_test_shim_static) functions in hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_threaded.c, I found the problem, and it deals with both the test_libhdfs_threaded.c calls and the functions listed in hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/tests/hdfs_shim.c file.

test_libhdfs_threaded.c uses two file systems: hdfs (C) and hdfspp (C+).  It makes a call to hdfsGetPathInfo of hdfs_shim.c passing in a C+ FS which calls the libhdfspp_hdfsGetPathInfo, which is correct.  Then test_libhdfs_threaded.c wants to know if the file is encrypted, so it calls hdfsFileIsEncrypted of hdfs_shim.c (knowing it is using a C++ PathInfo structure) which calls libhdfs_hdfsFileIsEncrypted, which is not correct.

In hdfs_shim.c:

hdfsFileInfo hdfsGetPathInfo(hdfsFS fs, const char path)

{ return (hdfsFileInfo *)libhdfspp_hdfsGetPathInfo(fs->libhdfsppRep, path); }

int hdfsFileIsEncrypted(hdfsFileInfo *hdfsFileInfo)

{ return libhdfs_hdfsFileIsEncrypted ((libhdfs_hdfsFileInfo *) hdfsFileInfo); }

The crux of the problem deals with the extendedHdfsFileInfo which exists in the C structures but not the C++ structures.

In RHEL 8.4, I get a 1 returned from hdfsFileIsEncrypted (uses bitwise and) when the file is not encrypted. 

I  put in a quick fix to get passed Test Case #35, but then found that it clashed with #40.

I modified the following files by adding new functions appropriate for the FS type and FileInfo structures returned just to get all (40) of the Native Test Cases to pass to be able to test the remaining test cases:  hdfs.h, libhdfs_wrapper_defines.h, libhdfs_wrapper_undefs.h, libhdfspp_wrapper.h, hdfs_shim.c, hdfs.c, test_libhdfs_threaded.c. 

Your fix, I would think, would be to add the extendedHdfsFileInfo to the C++ code and pass in the FileInfo structure to determine which hdfsFileIsEncrypted function should be called or change test_libhdfs_threaded.c to call hdfsFileIsEncrypted using a C FileInfo structure.

I've communicated with a Hadoop Native developer who said he didn't have any problem in Ubuntu.  My guess would be that the memory is initialized by default on that system to 0.  Not sure as I haven't used Ubuntu.

Notes:

I followed the CentOS 8 procedures in the BUILDING.txt.

HDFS-9359 has a note that might be relevant to this problem.  "hdfs_shim.c is created to support testing part of libhdfs++ that have been implemented. Functions not implemented in libhdfs++ are delegated to libhdfs." 

The hdfsFileIsEncrypted does care which FS is being used as the FileInfo structures returned are different sizes.