Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-31234

Add an option to redirect stdout/stderr for flink on kubernetes

    XMLWordPrintableJSON

Details

    Description

      Flink on Kubernetes does not support redirecting stdout/stderr to files. This is to allow users to get logs via "kubectl logs".

      But for our internal scenario, we use a kubernetes user to submit all jobs to the k8s cluster and provide a platform for users to submit jobs. Users can't access kubernetes directly. so we need to display logs/stdout in flink webui.

      Because the web ui retrieves the stdout file by filename, which has the same prefix as {taskmanager}.log (such as flink--kubernetes-taskmanager-0-my-first-flink-cluster-taskmanager-1-4.log). We can't support this with a simple custom image.

      IMO, we should add an option to redirect stdout/stderr to files. When this setting is enabled.
      1. flink-console.sh will redirect stdout/err to files.
      2. avoid logs twices in the log file and the stdout file. we could do this by using ThresholdFilter (log4j and logback) with a system property.

      Of course, this option is false by default.

      Attachments

        Issue Links

          Activity

            People

              huwh !huwh
              huwh !huwh
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: