Uploaded image for project: 'CarbonData'
  1. CarbonData
  2. CARBONDATA-4325

Documentation Issue in Github Link: https://github.com/apache/carbondata/blob/master/docs/carbon-as-spark-datasource-guide.md and fix partition table creation with df issue

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • None
    • 2.3.0
    • docs
    • None

    Description

      Scenario 1:

      https://github.com/apache/carbondata/blob/master/docs/carbon-as-spark-datasource-guide.md :

      • Under SUPPORTED Options, mention all supported Table Properties. Following are list of supported Table Properties not mentioned in the document:
        • bucketNumber
        • bucketColumns
        • streaming
        • timestampformat
        • dateformat
        • SPATIAL_INDEX
        • SPATIAL_INDEX_type
        • SPATIAL_INDEX_sourcecolumns
        • SPATIAL_INDEX_originLatitude
        • SPATIAL_INDEX_gridSize
        • SPATIAL_INDEX_conversionRatio
        • SPATIAL_INDEX_class

      Scenario 2:

      Partition Table Creation Using Spark Dataframe Fails with Spatial Index Property.

      Queries:

      val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
            StructField("longitude", LongType, nullable = false),
            StructField("latitude", LongType, nullable = false)))
      val geoDf = sqlContext.read.option("delimiter", ",").option("header", "true").schema(geoSchema).csv("hdfs://hacluster/geodata/geodata.csv")

      sql("drop table if exists source_index_df").show()
      geoDf.write
            .format("carbondata")
            .option("tableName", "source_index_df")
            .option("partitionColumns", "timevalue")
            .option("SPATIAL_INDEX", "mygeohash")
            .option("SPATIAL_INDEX.mygeohash.type", "geohash")
            .option("spatial_index.MyGeoHash.sourcecolumns", "longitude, latitude")
            .option("SPATIAL_INDEX.MyGeoHash.originLatitude", "39.832277")
            .option("SPATIAL_INDEX.mygeohash.gridSize", "50")
            .option("spatial_index.mygeohash.conversionRatio", "1000000")
            .option("spatial_index.mygeohash.CLASS", "org.apache.carbondata.geo.GeoHashIndex")
            .mode(SaveMode.Overwrite)
            .save()

       

      Attachments

        Activity

          People

            Unassigned Unassigned
            PURUJIT PURUJIT CHAUGULE
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 4.5h
                4.5h