Details
-
Improvement
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
3.1.0
-
None
-
None
Description
hadoopConf can be accessed via `SparkContext.hadoopConfiguration` from both user code and Spark internal. The configuration is mainly used to read files from hadoop-supported file system(eg. get URI/get FileSystem/add security credentials/get metastore connect url/etc.)
We shall keep a global config that users can set and use that to track the hadoop configurations, and avoid using `SparkContext.hadoopConfiguration`, maybe we shall mark it as deprecate.
Attachments
Issue Links
- links to