Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28597

Spark streaming terminated when close meta data log error

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 3.1.0
    • None
    • None

    Description

      Now, HDFSMetaDataLog is the only implement of MetaDataLog. If HDFSMetaDataLog.writeBatchToFile() execute error, then total app will be terminated.

      The meta data log is very important for spark, we should retry some times when exception, and add a config to control retry times.

      For example HDFS has a bug that close error if has no enough replicas detail HDFS-11486.

       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ulysses XiDuo You
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated: