Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28456

Add a public API `Encoder.makeCopy` to allow creating Encoder without touching Scala reflections

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.4.3
    • 3.0.0
    • SQL
    • None

    Description

      Because `Encoder` is not thread safe, the user cannot reuse an `Encoder` in multiple `Dataset`s. However, creating an `Encoder` for a complicated class is slow due to Scala reflections. To reduce the cost of Encoder creation, right now I usually use the private API `ExpressionEncoder.copy` as follows:

      object FooEncoder {
       private lazy val _encoder: ExpressionEncoder[Foo] = ExpressionEncoder[Foo]()
       implicit def encoder: ExpressionEncoder[Foo] = _encoder.copy()
      }
      

      This PR proposes a new method `makeCopy` in `Encoder` so that the above codes can be rewritten using public APIs.

      object FooEncoder {
       private lazy val _encoder: Encoder[Foo] = Encoders.product[Foo]()
       implicit def encoder: Encoder[Foo] = _encoder.makeCopy
      }
      

      Attachments

        Issue Links

          Activity

            People

              zsxwing Shixiong Zhu
              zsxwing Shixiong Zhu
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: