Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7044

[Spark SQL] query would hang when using scripts in SQL statement

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.3.0, 1.4.0
    • 1.3.0, 1.4.0
    • SQL
    • None

    Description

      Query with 'USING' operator like below would hang when using scripts in SQL statement

      INSERT INTO TABLE ${hiveconf:RESULT_TABLE}
      SELECT pid1, pid2, COUNT (*) AS cnt
      FROM (
        --Make items basket
        FROM (
          -- Joining two tables
          SELECT s.ss_ticket_number AS oid , s.ss_item_sk AS pid
          FROM store_sales s
          INNER JOIN item i ON (s.ss_item_sk = i.i_item_sk)
          WHERE i.i_category_id in (${hiveconf:q01_i_category_id_IN})
          AND s.ss_store_sk in (${hiveconf:q01_ss_store_sk_IN})
          CLUSTER BY oid
        ) q01_map_output
        REDUCE q01_map_output.oid, q01_map_output.pid
        USING '${env:BIG_BENCH_JAVA} ${env:BIG_BENCH_java_child_process_xmx} -cp bigbenchqueriesmr.jar de.bankmark.bigbench.queries.q01.Red -ITEM_SET_MAX ${hiveconf:q01_NPATH_ITEM_SET_MAX} '
        AS (pid1 BIGINT, pid2 BIGINT)
      ) q01_temp_basket
      

      Attachments

        Activity

          People

            chenghao Cheng Hao
            jameszhouyi Yi Zhou
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: