Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7723

[Python] StructArray timestamp type with timezone to_pandas convert error

    XMLWordPrintableJSON

Details

    Description

      When a StructArray has a child that is a timestamp with a timezone, the to_pandas conversion outputs an int64 instead of a timestamp

      In [1]: import pyarrow as pa 
         ...: import pandas as pd 
         ...: arr = pa.array([{'start': pd.Timestamp.now(), 'end': pd.Timestamp.now()}]) 
         ...:                                                                                                                      
      
      In [2]: arr.to_pandas()                                                                                   
      Out[2]: 
      0    {'end': 2020-01-29 11:38:02.792681, 'start': 2...
      dtype: object
      
      In [3]: ts = pd.Timestamp.now()                                                                                              
      
      In [4]: arr2 = pa.array([ts], type=pa.timestamp('us', tz='America/New_York'))                                                
      
      In [5]: arr2.to_pandas()                                                                                  
      Out[5]: 
      0   2020-01-29 06:38:47.848944-05:00
      dtype: datetime64[ns, America/New_York]
      
      In [6]: arr = pa.StructArray.from_arrays([arr2, arr2], ['start', 'stop'])                                                    
      
      In [7]: arr.to_pandas()                                                                                   
      Out[7]: 
      0    {'start': 1580297927848944000, 'stop': 1580297...
      dtype: object
      
      

      from https://github.com/apache/arrow/pull/6312

      Attachments

        Issue Links

          Activity

            People

              wesm Wes McKinney
              bryanc Bryan Cutler
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h