Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-18298

[Python] datetime shifted when using pyarrow.Table.from_pandas to load a pandas DateFrame containing datetime with timezone

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Duplicate
    • 9.0.0
    • None
    • Python
    • None
    • MacOS M1, Python 3.8.13

    Description

      Problem:

      When using pyarrow.Table.from_pandas to load a pandas DataFrame which contains a timestamp object with timezone information, the created Table object will shift the datetime, while still keeping the timezone information. Please see my scripts.

       

      Reproduce scripts:

      import pandas as pd
      import pyarrow
      ts = pd.Timestamp("2022-10-21 22:46:17", tz="America/Los_Angeles")
      df = pd.DataFrame({"TS": [ts]})
      table = pyarrow.Table.from_pandas(df)
      
      print(df)
      """
                               TS
      0 2022-10-21 22:46:17-07:00
      """
      
      print(table)
      """
      pyarrow.Table
      TS: timestamp[ns, tz=America/Los_Angeles]
      ----
      TS: [[2022-10-22 05:46:17.000000000]]""" 

      Expected results:

      The table should not shift the datetime when timezone information is provided.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              adam_ling Adam Ling
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: