Description
Currently, only one API called textFileStream in StreamingContext to specify the text file dstream, which ignores the old files always. On some times, the old files is still useful.
Need a API to let user choose whether the old files need to be ingored or not .
The API currently in StreamingContext:
def textFileStream(directory: String): DStream[String] =
Attachments
Issue Links
- is duplicated by
-
SPARK-6061 File source dstream can not include the old file which timestamp is before the system time
- Resolved
- links to