Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-5980

Add load tests for Core Apache Beam operations

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • P3
    • Resolution: Unresolved
    • None
    • None
    • testing
    • None

    Description

      This involves adding a suite of load tests described in this proposal: https://s.apache.org/load-test-basic-operations

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. BEAM-6334 Change status of LoadTests after failure on Dataflow

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. BEAM-7536 Fix BigQuery dataset name in collecting Load Tests metrics

          • P2 - Default priority. Will be triaged and planned according to community practices.
          • Resolved
          links to

          Web Link GitHub Pull Request #9286

          1.
          		 Create GroupByKey load test for Java SDK Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 10m
          2.
          		 Create ParDo load test for Java SDK Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 40m
          3.
          		 Create CoGroupByKey load test for Java SDK Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 50m
          4.
          		 Create Side input load test for Java SDK Sub-task Open Unassigned  
          5.
          		 Create Combine load test for Java SDK Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 50m
          6.
          		 Publish metrics from load tests to BigQuery database Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 50m
          7.
          		 Create jenkins jobs to run the load tests for Java SDK Sub-task Resolved Michał Walenia

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 20h 20m
          8.
          		 Provide standard configs to run load tests Sub-task Open Unassigned  
          9.
          		 Runtime and total bytes metrics are not collected properly Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4.5h
          10.
          		 Provide a "Data insertion pipeline" for Python suites Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 20m
          11.
          		 Create unbounded synthetic source Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 6h 10m
          12.
          		 extend "Data insertion Pipeline" with Kafka IO Sub-task Resolved Michał Walenia

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 4h 40m
          13.
          		 Make the schema for BQ tables storing metric results more generic (JAVA) Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 2h 40m
          14.
          		 Make the schema for BQ tables storing metric results more generic (Python) Sub-task Resolved Kasia Kucharczyk

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          15.
          		 Create PostCommit smoke test suite (besides phrase triggered one) Sub-task Open Unassigned  
          16.
          		 Apply windowing in load tests Sub-task Resolved Lukasz Gajowy

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 1h 40m
          17.
          		 Extend "Data insertion pipeline" with KinesisIO Sub-task Resolved Michał Walenia

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 0.5h
          18.
          		 Create GroupByKey streaming load test job for Java SDK Sub-task Resolved Michał Walenia

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 15h 20m
          19.
          		 Create ParDo and Combine streaming load test jobs for Java SDK Sub-task Resolved Michał Walenia

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 17h 50m
          20.
          		 Run Python GBK load tests on portable Flink runner Sub-task Resolved Unassigned

          100%
          Original Estimate - Not Specified Original Estimate - Not Specified
          Time Spent - 11h 10m

          Activity

            People

              Unassigned Unassigned
              ŁukaszG Lukasz Gajowy
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 101.5h
                  101.5h