Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6629

Make sure Java Apache Beam Logs properly include step names.

Details

    • New Feature
    • Status: Resolved
    • P3
    • Resolution: Duplicate
    • None
    • Not applicable
    • java-fn-execution
    • None

    Description

      I've noticed that the DataflowWorkerLoggingHandler grabs the current execution state and extracts the step name from it.

       

      However, in the Java SDK

      ExecutionState does not implement step names, I left that logic only in DataflowExecutionState and omitted it from the base class. Thus when the Java SDK uses its SimpleExecutionState, it will not receive these step names and log them properly.

      @swegner, can you confirm that this is the behaviour we want, for Java Beam SDKs to also include their step name in the logs, as used by the DatafloWorkerLoggingHandler? Java SDK code will end up using this handler as well, in portability?

      https://github.com/apache/beam/blob/52e73282223980fc0df9fbdbeddb2abb24d6600e/runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/logging/DataflowWorkerLoggingHandler.java#L148

       

      Also, it may matter for dataflow the specific type of step name being used. (initial, optomized, user name ,etc.). The full name context will definetly not be available in the beam java SDK,

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ajamato@google.com Alex Amato
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: