Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-29339

Support Arrow 0.14 in vectoried dapply and gapply (test it in AppVeyor build)

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.0.0
    • 3.0.0
    • SparkR
    • None

    Description

      dapply and gapply with Arrow optimization and Arrow 0.14 seems failing:

      > collect(dapply(df, function(rdf) { data.frame(rdf$gear + 1) }, structType("gear double")))
Error in readBin(con, raw(), as.integer(dataLen), endian = "big") :
  invalid 'n' argument

      We should fix it and also test it in AppVeyor

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-26924 Fix CRAN hack as soon as Arrow is available on CRAN

          • Major - Major loss of function.
          • Resolved
          relates to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-28215 as_tibble was removed from Arrow R API

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #25993

          Activity

            People

              Unassigned Unassigned
              gurwls223 Hyukjin Kwon
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: