Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
In mapreduce, the key length used in "KeyGenerator.init()" is configured with configuration option "mapreduce.job.encrypted-intermediate-data-key-size-bits" as follows:
/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
protected void initJobCredentialsAndUGI(Configuration conf) { ... int keyLen = conf.getInt( MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS, MRJobConfig .DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS); KeyGenerator keyGen = KeyGenerator.getInstance(INTERMEDIATE_DATA_ENCRYPTION_ALGO); keyGen.init(keyLen); encryptedSpillKey = keyGen.generateKey().getEncoded(); ... }
The same usage is also in mapred as follows:
/org/apache/hadoop/mapred/LocalJobRunner.java
public Job(JobID jobid, String jobSubmitDir) throws IOException { ... int keyLen = conf.getInt( MRJobConfig.MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS, MRJobConfig .DEFAULT_MR_ENCRYPTED_INTERMEDIATE_DATA_KEY_SIZE_BITS); KeyGenerator keyGen = KeyGenerator.getInstance(INTERMEDIATE_DATA_ENCRYPTION_ALGO); keyGen.init(keyLen); ... }
Also, in hadoop-common, there is a configration option "hadoop.security.key.default.bitlength", it is used in "KeyProvider.java" to initiate KeyGenerator as follows:
/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
public static final String HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_KEY = "hadoop.security.key.default.bitlength"; /** Defalt value for HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_KEY. */ public static final int HADOOP_SECURITY_KEY_DEFAULT_BITLENGTH_DEFAULT = 128;
/org/apache/hadoop/crypto/key/KeyProvider.java
public Options(Configuration conf) { cipher = conf.get(DEFAULT_CIPHER_NAME, DEFAULT_CIPHER); bitLength = conf.getInt(DEFAULT_BITLENGTH_NAME, DEFAULT_BITLENGTH); } public KeyVersion createKey(String name, Options options) throws NoSuchAlgorithmException, IOException { byte[] material = generateKey(options.getBitLength(), options.getCipher()); return createKey(name, material, options); } protected byte[] generateKey(int size, String algorithm) throws NoSuchAlgorithmException { algorithm = getAlgorithm(algorithm); KeyGenerator keyGenerator = KeyGenerator.getInstance(algorithm); keyGenerator.init(size); byte[] key = keyGenerator.generateKey().getEncoded(); return key; } ... }
However, in other two usage of "KeyGenerator.init()" in mapreduce and hadoop-common, the key length is hard-coded as 64. Also, in the evolving history, this value is changed from "20" to "64". So, in the perspective of flexibility and security, these two hard coded value in "KeyGenerator.init()" should be configurable.
/org/apache/hadoop/mapreduce/JobSubmitter.java
class JobSubmitter { ... private static final int SHUFFLE_KEY_LENGTH = 64; ... JobStatus submitJobInternal(Job job, Cluster cluster) throws ClassNotFoundException, InterruptedException, IOException { ... keyGen = KeyGenerator.getInstance(SHUFFLE_KEYGEN_ALGORITHM); keyGen.init(SHUFFLE_KEY_LENGTH); ... } ... }
/org/apache/hadoop/security/token/SecretManager.java
public abstract class SecretManager<T extends TokenIdentifier> { ... private static final int KEY_LENGTH = 64; ... private final KeyGenerator keyGen; { try { keyGen = KeyGenerator.getInstance(DEFAULT_HMAC_ALGORITHM); keyGen.init(KEY_LENGTH); } catch (NoSuchAlgorithmException nsa) { throw new IllegalArgumentException("Can't find " + DEFAULT_HMAC_ALGORITHM + " algorithm."); } } ... }