Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-2216

Aperiodic Data handling in oozie

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • coordinator
    • None

    Description

      Currently Oozie scheduling works on periodic datasets. It does not have any mechanism to handle aperiodic datasets, which doesn’t follow a fixed schedule/frequency.

      Use cases
      When incoming dataset arrives with no fixed schedule.
      Need to trigger the job based all data available since last run with a possible cap on the max size to process in one run.
      Try to avoid creating so many instances when you know input instances will be very few.

      Attachments

        1. Oozie_aperiodic_data_handling.pdf
          104 kB
          Jaydeep Vishwakarma

        Issue Links

          Activity

            People

              jaydeepvishwakarma Jaydeep Vishwakarma
              jaydeepvishwakarma Jaydeep Vishwakarma
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: