Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-3276

Provide a API to specify MIN_REMEMBER_DURATION for files to consider as input in streaming

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.2.0
    • 1.4.0
    • DStreams
    • None

    Description

      Currently, only one API called textFileStream in StreamingContext to specify the text file dstream, which ignores the old files always. On some times, the old files is still useful.
      Need a API to let user choose whether the old files need to be ingored or not .

      The API currently in StreamingContext:
      def textFileStream(directory: String): DStream[String] =

      { fileStream[LongWritable, Text, TextInputFormat](directory).map(_._2.toString) }

      Attachments

        Issue Links

          Activity

            People

              emres Emre Sevinç
              jhu Jack Hu
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: