Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27509

enable connection in cluster mode for output in client machine

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Trivial
    • Resolution: Incomplete
    • 3.0.0
    • None
    • Spark Submit

    Description

      While working on Spark I implemented a feature where when you submit a Spark job you enable the option that when it finishes the client machine will read from a file that the Spark job wrote to. It enables a quick and easy way to get output of lets say a dynamic Spark SQL query in the client without manually doing a hdfs download command or anything like. There are some pre requisites for it ( like having configurations for the hdfs communication) but I think its completely possible to  implement it and help many users to have an established communication line with a Spark job.

      What do you think about the option? 
      Would be glad to hear any issues with this idea or why it shouldnt be implemented. 

      Attachments

        Activity

          People

            Unassigned Unassigned
            eskimolimun Ilya Brodetsky
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: