Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36360

Delete appName from StreamingSource

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 3.1.2
    • None
    • DStreams
    • None

    Description

      The StreamingSource includes the appName in its sourceName. However, the appName should not be handled by the StreamingSource. It is already handled by the MetricsSystem. See all other MetricSources, e.g. ExecutorMetricsSource.

      Why is this important? See this part from the documentation:

      Often times, users want to be able to track the metrics across apps for driver and executors, which is hard to do with application ID (i.e. spark.app.id) since it changes with every invocation of the app. For such use cases, a custom namespace can be specified for metrics reporting using spark.metrics.namespace configuration property. If, say, users wanted to set the metrics namespace to the name of the application, they can set the spark.metrics.namespace property to a value like ${spark.app.name}. This value is then expanded appropriately by Spark and is used as the root namespace of the metrics system.

      This is only possible if the MetricsSystem handles the namespace which it does. But the StreamingSource additionally adds the appName in its sourceName, thus there is no way to configure a namespace that does not include the appName.

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            mrcl Marcel Neumann
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: