How to keep dtypes when reading a parquet file(read_parquet()) in pandas?

Question

Code:

In [31]: df = pd.DataFrame({"a": [[{"b": 1}], [{"b": np.nan}]]})

In [32]: df
Out[32]:
              a
0    [{'b': 1}]
1  [{'b': nan}]

In [33]: df.dtypes
Out[33]:
a    object
dtype: object

In [34]: df.to_parquet("a.parquet")

In [35]: pd.read_parquet("a.parquet")
Out[35]:
               a
0   [{'b': 1.0}]
1  [{'b': None}]

As you can see here, [{'b': 1}] becomes [{'b': 1.0}].

How can I keep dtypes even in reading the parquet file?

Answer 1

You can try to use pyarrow.parquet.read_table and pyarrow.Table.to_pandas with integer_object_nulls (see the doc)

import pyarrow.parquet as pq

pq.read_table("a.parquet").to_pandas(integer_object_nulls=True)

	a
0	[{'b': 1}]
1	[{'b': None}]

On the other hand, it looks like pandas.read_parquet with use_nullable_dtypes doesn't work.

df = pd.DataFrame({"a": [[{"b": 1}], [{"b": None}]]})

df.to_parquet("a.parquet")
pd.read_parquet("a.parquet", use_nullable_dtypes=True)

	a
0	[{'b': 1.0}]
1	[{'b': None}]