Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-27088

IntegrationLoadTestCommonCrawl async load improvements

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      ITLCC improvements:

      • Use an async client and work stealing executor for parallelism during loads.
      • Remove the verification read retries, these are not that effective during replication lag anyway.
      • Increase max task attempts because S3 might throttle.
      • Implement a side task that exercises Increments by extracting urls from content and updating a cf that tracks referrer counts. These are not validated at this time. It could be possible to log the increments, sum them with a reducer, and then verify the total, but this is left as a future exercise.

      Attachments

        Issue Links

          Activity

            People

              apurtell Andrew Kyle Purtell
              apurtell Andrew Kyle Purtell
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: