Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18352

Parse normal, multi-line JSON files (not just JSON Lines)

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.2.0
    • SQL

    Description

      Spark currently can only parse JSON files that are JSON lines, i.e. each record has an entire line and records are separated by new line. In reality, a lot of users want to use Spark to parse actual JSON files, and are surprised to learn that it doesn't do that.

      We can introduce a new mode (wholeJsonFile?) in which we don't split the files, and rather stream through them to parse the JSON files.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-10840 SparkSQL doesn't work well with JSON

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          New Feature - A new feature of the product, which has yet to be developed. SPARK-17969 I think it's user unfriendly to process standard json file with DataFrame

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-7366 Support multi-line JSON objects

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Task - A task that needs to be done. SPARK-20980 Rename the option `wholeFile` to `multiLine` for JSON and CSV

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-16496 Add wholetext as option for reading text in SQL.

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-7366 Support multi-line JSON objects

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          links to

          Pull request #16386 [Github] Pull Request #16386 (NathanHowell)

          Pull request #17128 [Github] Pull Request #17128 (felixcheung)

          Activity

            People

              NathanHowell Nathan Howell
              rxin Reynold Xin
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: