Uploaded image for project: 'Spot (Retired)'
  1. Spot (Retired)
  2. SPOT-136

[ML] Number of executors is set but also dynamic allocation, need to leave only one

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Trivial
    • Resolution: Fixed
    • None
    • 1.0
    • Any

    Description

      As Giacomo originally reported:
      In ml_ops.sh there are:
      --num-executors ${SPK_EXEC} \
      and:
      --conf spark.dynamicAllocation.enabled=true \

      which trigger the warning:
      WARN spark.SparkContext: Dynamic Allocation and num executors both
      set, thus dynamic allocation disabled.

      Shouldn't we remove the "--num-executors" and add instead:
      --conf spark.dynamicAllocation.maxExecutors=${SPK_EXEC} \
      ?

      Still need to determine if we keep it dynamic of fixed number of executors.

      Attachments

        Issue Links

          Activity

            People

              rabarona Ricardo Barona
              rabarona Ricardo Barona
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: