Different behavior between rolling and non-rolling event log

When creating a rolling event log, the application directory is created with a call to FileSystem.mkdirs, with the file permission 770. The default behavior of HDFS is to set the permission of a file created with FileSystem.create or FileSystem.mkdirs to (P & ^umask), where P is the permission in the API call and umask is a system value set by fs.permissions.umask-mode and defaults to 0022. This means, with default settings, any mkdirs call can have at most 755 permissions, which causes rolling event log directories to be created with 750 permissions. This causes the history server to be unable to prune old applications if they are not run by the same user running the history server.

This is not a problem for non-rolling logs, because it uses SparkHadoopUtils.createFile for Hadoop 2 backward compatibility, and then calls FileSystem.setPermission with 770 after the file has been created. setPermission doesn't have the umask applied to it, so this works fine.

Obviously this could be fixed by changing fs.permissions.umask-mode, but I'm not sure the reason that's set in the first place or if this would hurt anything else. The main issue is there is different behavior between rolling and non-rolling event logs that might want to be updated in this repo to be consistent across each.