Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-40163

[SPARK][SQL] feat: SparkSession.confing(Map)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • 3.3.0
    • 3.4.0
    • Spark Core
    • None

    Description

      https://github.com/apache/spark/pull/37478 

      • as-is
        private fun config(builder: SparkSession.Builder): SparkSession.Builder {
                val map = YamlUtils.read(this::class.java, "spark", Extension.YAML)
        
                var b = builder
                map.keys.forEach {
                    val k = it
                    val v = map[k]
        
                    b = when (v) {
                        is Long -> b.config(k, v)
                        is String -> b.config(k, v)
                        is Double -> b.config(k, v)
                        is Boolean -> b.config(k, v)
                        else -> b
                    }
                }
        
                return b
            }
        } 
      • to-be
        private fun config(builder: SparkSession.Builder): SparkSession.Builder {
                val map = YamlUtils.read(this::class.java, "spark", Extension.YAML)
        
                return b.config(map)
            }
        } 

      Attachments

        Activity

          People

            seunggabi seunggabi
            seunggabi seunggabi
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: