IntegrationTestLoadCommonCrawl

This integration test loads successful resource retrieval records from the Common Crawl (https://commoncrawl.org/) public dataset into an HBase table and writes records that can be used to later verify the presence and integrity of those records.

Run like:

./bin/hbase org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl \
  -files /path/to/hadoop-aws.jar \
  -Dfs.s3n.awsAccessKeyId=<AWS access key> \
  -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \
  /path/to/test-CC-MAIN-2021-10-warc.paths.gz \
  /path/to/tmp/warc-loader-output
 

Access to the Common Crawl dataset in S3 is made available to anyone by Amazon AWS, but Hadoop's S3N filesystem still requires valid access credentials to initialize.

The input path can either specify a directory or a file. The file may optionally be compressed with gzip. If a directory, the loader expects the directory to contain one or more WARC files from the Common Crawl dataset. If a file, the loader expects a list of Hadoop S3N URIs which point to S3 locations for one or more WARC files from the Common Crawl dataset, one URI per line. Lines should be terminated with the UNIX line terminator.

Included in hbase-it/src/test/resources/CC-MAIN-2021-10-warc.paths.gz is a list of all WARC files comprising the Q1 2021 crawl archive. There are 64,000 WARC files in this data set, each containing ~1GB of gzipped data. The WARC files contain several record types, such as metadata, request, and response, but we only load the response record types. If the HBase table schema does not specify compression (by default) there is roughly a 10x expansion. Loading the complete CC-MAIN-2021-10 crawl archive results in a table approximately 640 TB in size.

The hadoop-aws jar will be needed at runtime to instantiate the S3N filesystem. Use the -files ToolRunner argument to add it.

You can also split the Loader and Verify stages:

Load with:

./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Loader' \
  -files /path/to/hadoop-aws.jar \
  -Dfs.s3n.awsAccessKeyId=<AWS access key> \
  -Dfs.s3n.awsSecretAccessKey=<AWS secret key> \
  /path/to/test-CC-MAIN-2021-10-warc.paths.gz \
  /path/to/tmp/warc-loader-output

Verify with:

./bin/hbase 'org.apache.hadoop.hbase.test.IntegrationTestLoadCommonCrawl$Verify' \
  /path/to/tmp/warc-loader-output