Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29553

This problemis about using native BLAS to improvement ML/MLLIB performance

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Not A Problem
    • 2.3.0, 2.4.4
    • None
    • ML, MLlib

    Description

      I use native BLAS to improvement ML/MLLIB performance on Yarn.

      The file spark-env.sh which is modified by SPARK-21305 said that I should set OPENBLAS_NUM_THREADS=1 to disable multi-threading of OpenBLAS, but it does not take effect.

      I modify spark.conf to set  spark.executorEnv.OPENBLAS_NUM_THREADS=1,and the performance improve.
       
       
      I think MKL_NUM_THREADS is the same.
       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Zeyiii WuZeyi
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: