Python numpy: cannot convert datetime64[ns] to datetime64[D] (to use with Numba)

by

Series.astype converts all date-like objects to datetime64[ns].
To convert to datetime64[D], use values to obtain a NumPy array before calling astype:

dates_input = df["month_15"].values.astype('datetime64[D]')

Note that NDFrames (such as Series and DataFrames) can only hold datetime-like objects as objects of dtype datetime64[ns]. The automatic conversion of all datetime-likes to a common dtype simplifies subsequent date computations. But it makes it impossible to store, say, datetime64[s] objects in a DataFrame column. Pandas core developer, Jeff Reback explains,

“We don’t allow direct conversions because its simply too complicated to keep anything other than datetime64[ns] internally (nor necessary at all).”

Also note that even though df['month_15'].astype('datetime64[D]') has dtype datetime64[ns]:

In [29]: df['month_15'].astype('datetime64[D]').dtype
Out[29]: dtype('<M8[ns]')

when you iterate through the items in the Series, you get pandas Timestamps, not datetime64[ns]s.

In [28]: df['month_15'].astype('datetime64[D]').tolist()
Out[28]: [Timestamp('2010-01-15 00:00:00'), Timestamp('2011-01-15 00:00:00')]

Therefore, it is not clear that Numba actually has a problem with datetime64[ns], it might just have a problem with Timestamps. Sorry, I can’t check this — I don’t have Numba installed.

However, it might be useful for you to try

testf(df['month_15'].astype('datetime64[D]').values)

since df['month_15'].astype('datetime64[D]').values is truly a NumPy array of dtype datetime64[ns]:

In [31]: df['month_15'].astype('datetime64[D]').values.dtype
Out[31]: dtype('<M8[ns]')

If that works, then you don’t have to convert everything to datetime64[D], you just have to pass NumPy arrays — not Pandas Series — to testf.

More Related Contents:

  1. Replacing values represented by 'UN' with NaN
  2. Convert pandas dataframe to NumPy array
  3. Strings in a DataFrame, but dtype is object
  4. What does axis in pandas mean?
  5. FutureWarning: elementwise comparison failed; returning scalar, but in the future will perform elementwise comparison
  6. pandas select from Dataframe using startswith
  7. How to map numeric data into categories / bins in Pandas dataframe
  8. What’s the fastest way in Python to calculate cosine similarity given sparse matrix data?
  9. Restart cumsum and get index if cumsum more than value
  10. How can I map True/False to 1/0 in a Pandas DataFrame?
  11. What does the term “broadcasting” mean in Pandas documentation?
  12. Dropping infinite values from dataframes in pandas?
  13. ValueError: numpy.dtype has the wrong size, try recompiling
  14. Pandas: convert dtype ‘object’ to int
  15. Pandas lookup from one of multiple columns, based on value
  16. Most efficient way to sort an array into bins specified by an index array?
  17. Pandas pd.Series.isin performance with set versus array
  18. Get week start date (Monday) from a date column in Python (pandas)?
  19. Find symmetric pairs quickly in numpy
  20. What is dtype(‘O’), in pandas?
  21. Pass percentiles to pandas agg function
  22. Unpivot Pandas Data
  23. pandas.read_csv from string or package data
  24. Get first and second highest values in pandas columns
  25. ImportError: No module named ‘pandas.indexes’
  26. How to resample a dataframe with different functions applied to each column?
  27. Resample time series in pandas to a weekly interval
  28. Python Pandas: remove entries based on the number of occurrences
  29. Convert a column of timestamps into periods in pandas
  30. How to use numpy to get the cumulative count by unique values in linear time?

Leave a Comment