Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-787

Postgresql to HBase table replication example

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.2.0
    • 0.18.0
    • None
    • None

    Description

      It is useful to have an easy way to replicate data from Postgresql tables to a HBase tables.
      I made a simple python tool which does this, called hbrep.

      hbrep is a tool for replicating data from postgresql tables to hbase tables.

      Dependancies:

      • python 2.4
      • hbase 0.2.0
      • skytools 2.1.7
      • postgresql

      It has two main functions.

      • bootstrap, which bootstraps all the data from specified columns of a table
      • play, which processes incoming insert, update and delete events and applies them to hbase.

      Example usage:
      install triggers:
      ./hbrep.py hbrep.ini install schema1.table1 schema2.table2
      now that future updates are queuing, bootstrap the tables.
      ./hbrep.py hbrep.ini bootstrap schema1.table1 schema2.table2
      start pgq ticker (this is part of skytools, it manages event queues and sends the events to registered consumers).
      pgqadm.py pgq.ini ticker
      play our queue consumer to replicate events
      ./hbrep.py hbrep.ini play schema1.table1 schema2.table2

      more details in the readme.
      feedback and improvements appreciated.

      Attachments

        1. hbrep-2008.07.30.tar.gz
          16 kB
          Tim Sell

        Activity

          People

            stack Michael Stack
            tim_s Tim Sell
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: