Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Not A Problem
-
1.5.2
-
None
-
PostGIS Server:
Ubuntu-14.04
Postgres 9.4.5
PostGIS 2.2.1
Spark standalone client:
Java 7
Spark 1.5.2
PostGIS JDBC driver built within java directory of PostGIS 2.1.8, see: http://postgis.net/docs/manual-2.1/postgis_installation.html#idp58575520
Description
on server postgis running with a table that has a column of type raster
on client running spark standalone application that pulls data from postgis server using and jdbc driver and spark-sql:
val conf = new SparkConf().setAppName("Simple Application").setMaster("local")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
val jdbcDF = sqlContext.read.format("jdbc").options(Map("url" -> "jdbc:postgresql://postgishost/test?user=postgres&password=userpassword", "dbtable" -> "atlas")).load()
val rows = jdbcDF.take(3)
println(rows(0).toString())
println(rows(1).toString())
println(rows(2).toString())
when running this spark app i get:
java.sql.SQLException: Unsupported type 1111 at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:103) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anonfun$1.apply(JDBCRDD.scala:140) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anonfun$1.apply(JDBCRDD.scala:140) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:139) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRelation.<init>(JDBCRelation.scala:91) at org.apache.spark.sql.execution.datasources.jdbc.DefaultSource.createRelation(DefaultSource.scala:60) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:125) at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) at rdd.BasicRDDTests$$anonfun$1.apply$mcV$sp(BasicRDDTests.scala:65)
Attachments
Issue Links
- duplicates
-
SPARK-10186 Add support for more postgres column types
-
- Resolved
-