Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1882

Configuration improvements for Flume Kudu Sink regexp operations producer

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.3.0
    • Fix Version/s: 1.8.0
    • Component/s: flume-sink, integration
    • Labels:
      None

      Description

      The RegexpKuduOperationsProducer currently has the following configuration options that could be improved:

      Property Name Default Required? Description
      producer.skipMissingColumn
      false
      No What to do if a column in the Kudu table has no corresponding capture group. If set to true, a warning message is logged and the operation is still attempted. If set to false, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback.
      producer.skipBadColumnValue
      false
      No What to do if a value in the pattern match cannot be coerced to the required type. If set to true, a warning message is logged and the operation is still attempted. If set to false, an exception is thrown and the sink will not process the Event, causing a Flume Channel rollback.
      producer.warnUnmatchedRows
      true
      No Whether to log a warning about payloads that do not match the pattern. If set to false, event bodies with no matches will be silently dropped.

      It would be an improvement if each of these concepts had the the following options: warn, ignore, reject

      Where warn would log a warning to the Flume log and continue processing, ignore would attempt to continue processing without issuing a warning, and reject would throw an exception.

      It may be that some fields are nullable or have defaults, potentially due to an ALTER TABLE, and we don't want to fill the Flume logs with useless warnings. Users may also want to reject any Events that don't match their regex so they can correct the configuration and restart Flume without losing those Events.

        Attachments

          Activity

            People

            • Assignee:
              fszabo Ferenc Szabo
              Reporter:
              mpercy Mike Percy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: