Step 1 of 4: Choose Issues

Cancel

T Patch Info Key Summary Assignee Reporter P Status Resolution Created Updated Due Development
Sub-task SPARK-39607

SPARK-22386 DataSourceV2: Distribution and ordering support V2 function in writing

Cheng Pan Cheng Pan Major Resolved Fixed  
Sub-task SPARK-37262

SPARK-22386 Not log empty aggregate and group by in JDBCScan

Huaxin Gao Huaxin Gao Minor Resolved Fixed  
Sub-task SPARK-37220

SPARK-22386 Do not split input file for Parquet reader with aggregate push down

Cheng Su Cheng Su Minor Resolved Fixed  
Sub-task SPARK-37167

SPARK-22386 Add benchmark for aggregate push down

Unassigned Cheng Su Minor Open Unresolved  
Sub-task SPARK-36647

SPARK-22386 Push down filter by partition column for Aggregate (Min/Max/Count) for Parquet

Huaxin Gao Huaxin Gao Minor Resolved Fixed  
Sub-task SPARK-36646

SPARK-22386 Push down group by partition column for Aggregate (Min/Max/Count) for Parquet

Huaxin Gao Huaxin Gao Major Resolved Fixed  
Sub-task SPARK-36645

SPARK-22386 Aggregate (Min/Max/Count) push down for Parquet

Huaxin Gao Huaxin Gao Major Resolved Fixed  
Sub-task SPARK-34960

SPARK-22386 Aggregate (Min/Max/Count) push down for ORC

Cheng Su Cheng Su Minor Resolved Fixed  
Sub-task SPARK-34952

SPARK-22386 DS V2 Aggregate push down

Huaxin Gao Huaxin Gao Major Resolved Fixed  
Sub-task SPARK-34230

SPARK-22386 Let AQE determine the right parallelism in DistributionAndOrderingUtils

Unassigned Anton Okolnychyi Major Open Unresolved  
Sub-task SPARK-34183

SPARK-22386 DataSource V2: Support required distribution and ordering in SS

Anton Okolnychyi Anton Okolnychyi Blocker Resolved Fixed  
Sub-task SPARK-34049

SPARK-22386 DataSource V2: Use Write abstraction in StreamExecution

Anton Okolnychyi Anton Okolnychyi Major Resolved Fixed  
Sub-task SPARK-34026

SPARK-22386 DataSource V2: Inject repartition and sort nodes to satisfy required distribution and ordering

Anton Okolnychyi Anton Okolnychyi Major Resolved Fixed  
Sub-task SPARK-33808

SPARK-22386 DataSource V2: Build logical writes in the optimizer

Anton Okolnychyi Anton Okolnychyi Major Resolved Fixed  
Sub-task SPARK-33807

SPARK-22386 Data Source V2: Remove read specific distributions

Unassigned Anton Okolnychyi Major Open Unresolved  
Sub-task SPARK-33779

SPARK-22386 DataSource V2: API to request distribution and ordering on write

Anton Okolnychyi Anton Okolnychyi Major Resolved Fixed  
Sub-task SPARK-29248

SPARK-22386 Pass in number of partitions to BuildWriter

Ximo Guanter Ximo Guanter Major Resolved Fixed  
Sub-task SPARK-28612

SPARK-22386 DataSourceV2: Add new DataFrameWriter API for v2

Ryan Blue Ryan Blue Major Resolved Done  
Sub-task SPARK-28555

SPARK-22386 Recover options and properties and pass them back into the v1 API

Unassigned Xin Ren Minor Open Unresolved  
Sub-task SPARK-25700

SPARK-22386 Avoid to create a readsupport at write path in Data Source V2

Hyukjin Kwon Hyukjin Kwon Critical Resolved Fixed  
Sub-task SPARK-25460

SPARK-22386 DataSourceV2: Structured Streaming does not respect SessionConfigSupport

Hyukjin Kwon Hyukjin Kwon Major Resolved Fixed  
Sub-task SPARK-25280

SPARK-22386 Add support for USING syntax for DataSourceV2

Unassigned Hyukjin Kwon Major Resolved Incomplete  
Sub-task SPARK-25127

SPARK-22386 DataSourceV2: Remove SupportsPushDownCatalystFilters

Reynold Xin Ryan Blue Major Resolved Fixed  
Sub-task SPARK-24991

SPARK-22386 use InternalRow in DataSourceWriter

Wenchen Fan Wenchen Fan Major Resolved Fixed  
Sub-task SPARK-24990

SPARK-22386 merge ReadSupport and ReadSupportWithSchema

Wenchen Fan Wenchen Fan Major Resolved Fixed  
Sub-task SPARK-24971

SPARK-22386 remove SupportsDeprecatedScanRow

Wenchen Fan Wenchen Fan Major Resolved Fixed  
Sub-task SPARK-24478

SPARK-22386 DataSourceV2 should push filters and projection at physical plan conversion

Ryan Blue Ryan Blue Major Resolved Fixed  
Sub-task SPARK-24130

SPARK-22386 Data Source V2: Join Push Down

Unassigned Jia Li Major Resolved Incomplete  
Sub-task SPARK-24073

SPARK-22386 DataSourceV2: Rename DataReaderFactory to InputPartition.

Ryan Blue Ryan Blue Major Resolved Fixed  
Sub-task SPARK-23889

SPARK-22386 DataSourceV2: Add interfaces to pass required sorting and clustering for writes

Unassigned Ryan Blue Major Resolved Implemented  
Sub-task SPARK-23418

SPARK-22386 DataSourceV2 should not allow userSpecifiedSchema without ReadSupportWithSchema

Ryan Blue Ryan Blue Major Resolved Fixed  
Sub-task SPARK-23398

SPARK-22386 DataSourceV2 should provide a way to get a source's schema.

Unassigned Ryan Blue Major Resolved Fixed  
Sub-task SPARK-23341

SPARK-22386 DataSourceOptions should handle path and table names to avoid confusion.

Wenchen Fan Ryan Blue Major Resolved Fixed  
Sub-task SPARK-23325

SPARK-22386 DataSourceV2 readers should always produce InternalRow.

Ryan Blue Ryan Blue Major Resolved Fixed  
Sub-task SPARK-23323

SPARK-22386 DataSourceV2 should use the output commit coordinator.

Ryan Blue Ryan Blue Major Resolved Fixed  
Sub-task SPARK-23321

SPARK-22386 DataSourceV2 should apply some validation when writing.

Unassigned Ryan Blue Major Resolved Fixed  
Sub-task SPARK-23268

SPARK-22386 Reorganize packages in data source V2

Gengliang Wang Gengliang Wang Major Resolved Fixed  
Sub-task SPARK-23204

SPARK-22386 DataSourceV2 should support named tables in DataFrameReader, DataFrameWriter

Unassigned Ryan Blue Major Resolved Fixed  
Sub-task SPARK-23203

SPARK-22386 DataSourceV2 should use immutable trees.

Ryan Blue Ryan Blue Blocker Resolved Fixed  
Sub-task SPARK-22391

SPARK-22386 add `MetadataCreationSupport` trait to separate data and metadata handling at write path

Unassigned Wenchen Fan Major Resolved Incomplete  
Sub-task SPARK-22390

SPARK-22386 Aggregate push down

Unassigned Wenchen Fan Major Resolved Duplicate  
Sub-task SPARK-22388

SPARK-22386 Limit push down

Unassigned Wenchen Fan Major Resolved Incomplete  

Cancel