Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29973

Use nano time to calculate 'processedRowsPerSecond' to avoid 'NaN'/'Infinity'

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • Structured Streaming
    • None

    Description

      The "processingTimeSec" of batch may be less than 1 millis.  As "processingTimeSec" is calculated in millis, so "processingTimeSec" equals 0L. If there is no data in this batch, the "processedRowsPerSecond" equals "0/0.0d", i.e. "Double.NaN". If there are some data in this batch, the "processedRowsPerSecond" equals "N/0.0d", i.e. "Double.Infinity".

      Attachments

        Issue Links

          links to

          Web Link GitHub Pull Request #26610

          Activity

            People

              uncleGen Genmao Yu
              uncleGen Genmao Yu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: