Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-28597

Spark streaming terminated when close meta data log error

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Labels:
      None

      Description

      Now, HDFSMetaDataLog is the only implement of MetaDataLog. If HDFSMetaDataLog.writeBatchToFile() execute error, then total app will be terminated.

      The meta data log is very important for spark, we should retry some times when exception, and add a config to control retry times.

      For example HDFS has a bug that close error if has no enough replicas detail HDFS-11486.

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ulysses ulysses you
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: