Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5980 Add load tests for Core Apache Beam operations
  3. BEAM-6291

Make the schema for BQ tables storing metric results more generic (Python)

Details

    • Sub-task
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • None
    • 2.11.0
    • testing
    • None

    Description

      Currently, we keep the metrics results in BQ in tables with a schema like this: 

      timestamp | total_bytes | run_time | (possibly other BQ columns)

      every time we want to add a new column the schema has to be extended. This is not convenient given the fact that any load test can have different metrics stored. This in turn would cause multiple BQ tables each queried differently. 

      We can provide a more generic schema, like so: 

      test_id | timestamp | metric | value

      thanks to that, every metric, whatever it's name is, can be saved in the table as a separate row. This gives more elasticity in storing metrics and is still easy to query and plot.

      Attachments

        Issue Links

          Activity

            People

              kasiak Kasia Kucharczyk
              ŁukaszG Lukasz Gajowy
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 40m
                  1h 40m