Description
Scenario 1:
https://github.com/apache/carbondata/blob/master/docs/carbon-as-spark-datasource-guide.md :
- Under SUPPORTED Options, mention all supported Table Properties. Following are list of supported Table Properties not mentioned in the document:
-
- bucketNumber
- bucketColumns
- streaming
- timestampformat
- dateformat
- SPATIAL_INDEX
- SPATIAL_INDEX_type
- SPATIAL_INDEX_sourcecolumns
- SPATIAL_INDEX_originLatitude
- SPATIAL_INDEX_gridSize
- SPATIAL_INDEX_conversionRatio
- SPATIAL_INDEX_class
Scenario 2:
Partition Table Creation Using Spark Dataframe Fails with Spatial Index Property.
Queries:
val geoSchema = StructType(Seq(StructField("timevalue", LongType, nullable = true),
StructField("longitude", LongType, nullable = false),
StructField("latitude", LongType, nullable = false)))
val geoDf = sqlContext.read.option("delimiter", ",").option("header", "true").schema(geoSchema).csv("hdfs://hacluster/geodata/geodata.csv")
sql("drop table if exists source_index_df").show()
geoDf.write
.format("carbondata")
.option("tableName", "source_index_df")
.option("partitionColumns", "timevalue")
.option("SPATIAL_INDEX", "mygeohash")
.option("SPATIAL_INDEX.mygeohash.type", "geohash")
.option("spatial_index.MyGeoHash.sourcecolumns", "longitude, latitude")
.option("SPATIAL_INDEX.MyGeoHash.originLatitude", "39.832277")
.option("SPATIAL_INDEX.mygeohash.gridSize", "50")
.option("spatial_index.mygeohash.conversionRatio", "1000000")
.option("spatial_index.mygeohash.CLASS", "org.apache.carbondata.geo.GeoHashIndex")
.mode(SaveMode.Overwrite)
.save()