Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7737

Microbenchmark script do not work consistently

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.14.0
    • Component/s: sdk-py-harness
    • Labels:
      None

      Description

      microbenchmar scripts do not work consistently, at least run into problem for me:

      (beam) ➜  python git:(master) python -m apache_beam.tools.distribution_counter_microbenchmark
      Traceback (most recent call last):
        File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 174, in _run_module_as_main
          "__main__", fname, loader, pkg_name)
        File "/usr/local/Cellar/python@2/2.7.14_3/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
          exec code in run_globals
        File "/Users/rkumar/personal/opensource/beam/sdks/python/apache_beam/tools/distribution_counter_microbenchmark.py", line 69, in <module>
          'apache_beam.transforms.cy_dataflow_distribution_counter')
        File "apache_beam/tools/utils.py", line 41, in check_compiled
          "Profiling uncompiled code.\n"
      RuntimeError: Profiling uncompiled code.
      To compile beam, run 'pip install Cython; python setup.py build_ext --inplace'
      

      Tihs is because this line of check_compiled method
      doesn't work consistently. It every time just get the root module instead of the submodule and fails to identify the the given moudle is compiled with cython.
      Also it is not recommended to use _import_ method because it is python internal and it is meant to be used by the Python interpreter. importlib.import_module is the recommended way of finding the module.

      Pull request: https://github.com/apache/beam/pull/9066

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                rakeshkumar Rakesh Kumar
                Reporter:
                rakeshkumar Rakesh Kumar
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m