Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44101

Support pandas 2

    XMLWordPrintableJSON

Details

    Attachments

      Issue Links

        is related to

        Umbrella - An overarching type made of sub-tasks SPARK-44111 Prepare Apache Spark 4.0.0

        • Critical - Crashes, loss of data, severe memory leak.
        • Open
        1.
        		 Support `isocalendar` Sub-task Resolved Haejoon Lee
        2.
        		 Add `show_counts` parameter for DataFrame.info Sub-task Resolved Unassigned
        3.
        		 Deprecate & remove the APIs that will be removed in pandas 2.0. Sub-task Resolved Haejoon Lee
        4.
        		 Add `inclusive` parameter for (DataFrame|Series).between_time Sub-task Resolved Unassigned
        5.
        		 Add `inclusive` parameter for date_range Sub-task Resolved Unassigned
        6.
        		 Add migration notes for update to supported pandas version. Sub-task Resolved Haejoon Lee
        7.
        		 Upgrade pandas to 2.0.0 Sub-task Resolved Haejoon Lee
        8.
        		 PySpark 3.4.0 cannot convert timestamp-typed objects to pandas with pandas 2.0 Sub-task Resolved Unassigned
        9.
        		 MultiIndex.append not checking names for equality Sub-task Resolved Haejoon Lee
        10.
        		 Fix DatetimeIndex.microsecond to return 'int32' instead of 'int64' type of Index. Sub-task Resolved Haejoon Lee
        11.
        		 Match behavior with DataFrame.reindex with specifying `index`. Sub-task Resolved Unassigned
        12.
        		 Investigate DataFrame.sort_values with pandas behavior. Sub-task Resolved Unassigned
        13.
        		 Generate proper warning on different behavior with numeric_only Sub-task Resolved Unassigned
        14.
        		 Make DataFrameGroupBy.sum support for string type columns Sub-task Resolved Haejoon Lee
        15.
        		 Enable test_to_latex by supporting jinja2>=3.0.0 Sub-task Resolved Haejoon Lee
        16.
        		 Match `GroupBy.nth` behavior with new pandas behavior Sub-task Resolved Haejoon Lee
        17.
        		 Enable GroupBySlowTests.test_value_counts for pandas 2.0.0. Sub-task Closed Unassigned
        18.
        		 Enable GroupBySlowTests.test_split_apply_combine_on_series for pandas 2.0.0. Sub-task Closed Unassigned
        19.
        		 Enable RollingTests.test_rolling_count for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        20.
        		 Enable RollingTests.test_groupby_rolling_count for pandas 2.0.0. Sub-task Resolved Unassigned
        21.
        		 Ignore the names of MultiIndex when axis=1 for concat Sub-task Resolved Haejoon Lee
        22.
        		 Enable SeriesConversionTests.test_to_latex for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        23.
        		 Enable OpsOnDiffFramesGroupByTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        24.
        		 Enable OpsOnDiffFramesGroupByTests.test_groupby_different_lengths for pandas 2.0.0. Sub-task Resolved Unassigned
        25.
        		 Enable SeriesDateTimeTests.test_date_subtraction for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        26.
        		 Enable SeriesTests.test_rank for pandas 2.0.0. Sub-task Resolved Unassigned
        27.
        		 Enable SeriesTests.test_value_counts for pandas 2.0.0. Sub-task Resolved Unassigned
        28.
        		 Enable SeriesTests.test_append for pandas 2.0.0. Sub-task Resolved Unassigned
        29.
        		 Enable SeriesTests.test_astype for pandas 2.0.0. Sub-task Resolved Unassigned
        30.
        		 Enable SeriesTests.test_between for pandas 2.0.0. Sub-task Resolved Unassigned
        31.
        		 Enable SeriesTests.test_mad for pandas 2.0.0. Sub-task Resolved Unassigned
        32.
        		 Enable SeriesTests.test_quantile for pandas 2.0.0. Sub-task Resolved Unassigned
        33.
        		 Enable SeriesStringTests.test_string_replace for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        34.
        		 Enable SeriesStringTests.test_string_rsplit for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        35.
        		 Enable SeriesStringTests.test_string_split for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        36.
        		 Enable SeriesTests.test_iteritems for pandas 2.0.0. Sub-task Resolved Unassigned
        37.
        		 Enable SeriesTests.test_between_time for pandas 2.0.0. Sub-task Resolved Unassigned
        38.
        		 Enable SeriesTests.test_product for pandas 2.0.0. Sub-task Resolved Unassigned
        39.
        		 Enable StatsTests.test_cov_corr_meta for pandas 2.0.0. Sub-task Resolved Unassigned
        40.
        		 Enable StatsTests.test_axis_on_dataframe for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        41.
        		 Enable StatsTests.test_stat_functions_with_no_numeric_columns for pandas 2.0.0. Sub-task Resolved Unassigned
        42.
        		 Enable ArrowTests.test_toPandas_empty_columns for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        43.
        		 Enable MultiIndex test for IndexesTests.test_difference Sub-task Resolved Unassigned
        44.
        		 Enable SeriesTests.test_factorize for pandas 2.0.0. Sub-task Resolved Unassigned
        45.
        		 Enable GroupByTests.test_prod for pandas 2.0.0. Sub-task Resolved Unassigned
        46.
        		 Enable GroupByTests.test_nth for pandas 2.0.0. Sub-task Resolved Unassigned
        47.
        		 Enable GroupByTests.test_mad for pandas 2.0.0. Sub-task Resolved Unassigned
        48.
        		 Enable GroupByTests.test_basic_stat_funcs for pandas 2.0.0. Sub-task Resolved Unassigned
        49.
        		 Enable GroupByTests.test_groupby_multiindex_columns for pandas 2.0.0. Sub-task Resolved Unassigned
        50.
        		 Enable DataFrameSlowTests.test_describe for pandas 2.0.0. Sub-task Resolved Unassigned
        51.
        		 Enable DataFrameSlowTests.test_between_time for pandas 2.0.0. Sub-task Resolved Unassigned
        52.
        		 Enable DataFrameSlowTests.test_product for pandas 2.0.0. Sub-task Resolved Unassigned
        53.
        		 Enable DataFrameSlowTests.test_iteritems for pandas 2.0.0. Sub-task Resolved Unassigned
        54.
        		 Enable DataFrameSlowTests.test_mad for pandas 2.0.0. Sub-task Resolved Unassigned
        55.
        		 Enable DataFrameConversionTests.test_to_latex for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        56.
        		 Enable DataFrameTests.test_append for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        57.
        		 Enable CsvTests.test_read_csv_with_squeeze for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        58.
        		 Enable CategoricalTests.test_factorize for pandas 2.0.0. Sub-task Resolved Unassigned
        59.
        		 Enable CategoricalTests.test_as_ordered_unordered for pandas 2.0.0. Sub-task Resolved Unassigned
        60.
        		 Enable CategoricalTests.test_categories_setter for pandas 2.0.0. Sub-task Resolved Unassigned
        61.
        		 Enable CategoricalIndexTests.test_factorize for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        62.
        		 Enable CategoricalIndexTests.test_categories_setter for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        63.
        		 Enable DateOpsTests.test_rsub for pandas 2.0.0. Sub-task Resolved Unassigned
        64.
        		 Enable DateOpsTests.test_sub for pandas 2.0.0. Sub-task Resolved Unassigned
        65.
        		 Enable CategoricalTests.test_remove_categories for pandas 2.0.0. Sub-task Resolved Unassigned
        66.
        		 Enable IndexesTests.test_index_basic for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        67.
        		 Enable IndexesTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        68.
        		 Enable IndexesTests.test_union for pandas 2.0.0. Sub-task Resolved Unassigned
        69.
        		 Enable CategoricalIndexTests.test_remove_categories for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        70.
        		 Enable DataFramePlotMatplotlibTests.test_area_plot for pandas 2.0.0. Sub-task Resolved Unassigned
        71.
        		 Enable DataFramePlotMatplotlibTests.test_area_plot_stacked_false for pandas 2.0.0. Sub-task Resolved Unassigned
        72.
        		 Enable DataFramePlotMatplotlibTests.test_area_plot_y for pandas 2.0.0. Sub-task Resolved Unassigned
        73.
        		 Enable DataFramePlotMatplotlibTests.test_bar_plot for pandas 2.0.0. Sub-task Resolved Unassigned
        74.
        		 Enable DataFramePlotMatplotlibTests.test_bar_with_x_y for pandas 2.0.0. Sub-task Resolved Unassigned
        75.
        		 Enable DataFramePlotMatplotlibTests.test_barh_plot_with_x_y for pandas 2.0.0. Sub-task Resolved Unassigned
        76.
        		 Enable DataFramePlotMatplotlibTests.test_barh_plot for pandas 2.0.0. Sub-task Resolved Unassigned
        77.
        		 Enable DataFramePlotMatplotlibTests.test_line_plot for pandas 2.0.0. Sub-task Resolved Unassigned
        78.
        		 Enable DataFramePlotMatplotlibTests.test_pie_plot for pandas 2.0.0. Sub-task Resolved Unassigned
        79.
        		 Enable DataFramePlotMatplotlibTests.test_scatter_plot for pandas 2.0.0. Sub-task Resolved Unassigned
        80.
        		 Enable DatetimeIndexTests.test_indexer_between_time for pandas 2.0.0. Sub-task Resolved Unassigned
        81.
        		 Enable TimedeltaIndexTests.test_properties for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        82.
        		 Enable GroupByTests.test_apply_without_shortcut for pandas 2.0.0. Sub-task Resolved Unassigned
        83.
        		 Enable GroupByTests.test_mean for pandas 2.0.0. Sub-task Resolved Unassigned
        84.
        		 Enable GroupByTests.test_apply for pandas 2.0.0. Sub-task Resolved Unassigned
        85.
        		 Enable NamespaceTests.test_date_range for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        86.
        		 Enable DataFramePlotMatplotlibTests.test_hist_plot for pandas 2.0.0. Sub-task Resolved Unassigned
        87.
        		 Enable DataFramePlotMatplotlibTests.test_kde_plot for pandas 2.0.0. Sub-task Resolved Unassigned
        88.
        		 Enable SeriesDateTimeTests.test_day for pandas 2.0.0. Sub-task Resolved Unassigned
        89.
        		 Enable SeriesDateTimeTests.test_dayofweek for pandas 2.0.0. Sub-task Resolved Unassigned
        90.
        		 Enable SeriesDateTimeTests.test_dayofyear for pandas 2.0.0. Sub-task Resolved Unassigned
        91.
        		 Enable SeriesDateTimeTests.test_days_in_month for pandas 2.0.0. Sub-task Resolved Unassigned
        92.
        		 Enable SeriesDateTimeTests.test_daysinmonth for pandas 2.0.0. Sub-task Resolved Unassigned
        93.
        		 Enable SeriesDateTimeTests.test_hour for pandas 2.0.0. Sub-task Resolved Unassigned
        94.
        		 Enable SeriesDateTimeTests.test_microsecond for pandas 2.0.0. Sub-task Resolved Unassigned
        95.
        		 Enable SeriesDateTimeTests.test_minute for pandas 2.0.0. Sub-task Resolved Unassigned
        96.
        		 Enable SeriesDateTimeTests.test_month for pandas 2.0.0. Sub-task Resolved Unassigned
        97.
        		 Enable SeriesDateTimeTests.test_quarter for pandas 2.0.0. Sub-task Resolved Unassigned
        98.
        		 Enable SeriesDateTimeTests.test_second for pandas 2.0.0. Sub-task Resolved Unassigned
        99.
        		 Enable SeriesDateTimeTests.test_weekday for pandas 2.0.0. Sub-task Resolved Unassigned
        100.
        		 Enable SeriesDateTimeTests.test_year for pandas 2.0.0. Sub-task Resolved Unassigned
        101.
        		 Enable FeatureTests.test_standard_scaler for pandas 2.0.0. Sub-task Resolved Weichen Xu
        102.
        		 Enable FeatureTests.test_max_abs_scaler for pandas 2.0.0. Sub-task Resolved Weichen Xu
        103.
        		 Enable SummarizerTests.test_summarize_dataframe for pandas 2.0.0. Sub-task Resolved Weichen Xu
        104.
        		 Enable DataFrameSlowTests.test_cov for pandas 2.0.0. Sub-task Resolved Unassigned
        105.
        		 Enable DataFrameSlowTests.test_quantile for pandas 2.0.0. Sub-task Resolved Unassigned
        106.
        		 Enable DataFrameTests.test_reindex for pandas 2.0.0. Sub-task Resolved Unassigned
        107.
        		 Enable DataFrameTests.test_all for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        108.
        		 Enable CategoricalTests.test_groupby_apply_without_shortcut for pandas 2.0.0. Sub-task Resolved Unassigned
        109.
        		 Enable GroupBySlowTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        110.
        		 Enable SeriesTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        111.
        		 Enable SeriesDateTimeTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        112.
        		 Enable DataFramePlotMatplotlibTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        113.
        		 Enable DataFrameSlowTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        114.
        		 Enable GroupByTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        115.
        		 Enable CategoricalTests for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        116.
        		 Warning for the pandas-related behavior changes in next major release Sub-task Resolved Unassigned
        117.
        		 Warning for the pandas-related behavior changes in next major release Sub-task Resolved Haejoon Lee
        118.
        		 Match behavior with pandas for `SeriesStringTests.test_string_replace` Sub-task Resolved Unassigned
        119.
        		 Enable GroupbyAggregateTests.test_aggregate for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        120.
        		 Support value_counts for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        121.
        		 Support stat functions Sub-task Resolved Haejoon Lee
        122.
        		 Update document related to removed Index object Sub-task Resolved Haejoon Lee
        123.
        		 Support Pandas 2.1.0 Sub-task Resolved Haejoon Lee
        124.
        		 Remove deprecated Index APIs Sub-task Resolved Haejoon Lee
        125.
        		 Remove `inplace` parameter from `Categorical` APIs Sub-task Resolved Haejoon Lee
        126.
        		 Remove `col_space` parameter from `DataFrame.to_latex` Sub-task Resolved Haejoon Lee
        127.
        		 Remove boolean inputs for inclusive from Series.between Sub-task Resolved Haejoon Lee
        128.
        		 Upgrade Pandas to 2.1.1 Sub-task Resolved Haejoon Lee
        129.
        		 Change the default value for `numeric_only`. Sub-task Resolved Haejoon Lee
        130.
        		 Enable `GroupbySplitApplyTests.test_split_apply_combine_on_series` for pandas 2.0.0. Sub-task Resolved Haejoon Lee
        131.
        		 Remove remaining deprecated Pandas APIs from Spark 3.4.0 Sub-task Resolved Haejoon Lee
        132.
        		 Upgrade Pandas to 2.1.2 Sub-task Resolved Haejoon Lee

        Activity

          People

            itholic Haejoon Lee
            itholic Haejoon Lee
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: