pandas to_datetime parsing wrong year

by

That seems to be the behavior of the Python library datetime, I did a test to see where the cutoff is 68 – 69:

datetime.datetime.strptime('31-Dec-68', '%d-%b-%y').date()
>>> datetime.date(2068, 12, 31)

datetime.datetime.strptime('1-Jan-69', '%d-%b-%y').date()
>>> datetime.date(1969, 1, 1)

Two digits year ambiguity

So it seems that anything with the %y year below 69 will be attributed a century of 2000, and 69 upwards get 1900

The %y two digits can only go from 00 to 99 which is going to be ambiguous if we start crossing centuries.

If there is no overlap, you could manually process it and annotate the century (kill the ambiguity)

I suggest you process your data manually and specify the century, e.g. you can decide that anything in your data that has the year between 17 and 68 is attributed to 1917 – 1968 (instead of 2017 – 2068).

If you have overlap then you can’t process with insufficient year information, unless e.g. you have some ordered data and a reference

If you have overlap e.g. you have data from both 2016 and 1916 and both were logged as ’16’, that’s ambiguous and there isn’t sufficient information to parse this, unless the data is ordered by date in which case you can use heuristics to switch the century as you parse it.

More Related Contents:

  1. Filtering Pandas DataFrames on dates
  2. Keep only date part when using pandas.to_datetime
  3. Convert pandas timezone-aware DateTimeIndex to naive timestamp, but in certain timezone
  4. How to print pandas DataFrame without index
  5. datetime dtypes in pandas read_csv
  6. How to convert string to datetime format in pandas python?
  7. pandas dataframe groupby datetime month
  8. Find the end of the month of a Pandas DataFrame Series
  9. Pandas: Subtracting two date columns and the result being an integer
  10. How to change the datetime tick label frequency for matplotlib plots
  11. Converting between datetime and Pandas Timestamp objects
  12. Splitting timestamp column into separate date and time columns
  13. Pandas: Datetime Improperly selecting day as month from date [duplicate]
  14. Convert a column of datetimes to epoch in Python
  15. Pandas converting row with unix timestamp (in milliseconds) to datetime
  16. Python pandas convert datetime to timestamp effectively through dt accessor
  17. Specifying date format when converting with pandas.to_datetime
  18. Which is the fastest way to extract day, month and year from a given date?
  19. Locate first and last non NaN values in a Pandas DataFrame
  20. Truncate `TimeStamp` column to hour precision in pandas `DataFrame`
  21. Datetime conversion – How to extract the inferred format?
  22. Pandas custom function to find whether it is the 1st, 2nd etc Monday, Tuesday, etc – all suggestions welcome
  23. What values are valid in Pandas ‘Freq’ tags?
  24. Python: Datetime to season
  25. How do I convert timestamp to datetime.date in pandas dataframe?
  26. Pandas datetimes with different formats in the same column
  27. Convert dates to pd.to_datetime where month could be either a number or month name
  28. Get weekday/day-of-week for Datetime column of DataFrame
  29. Plot pandas dates in matplotlib
  30. Pandas: select all dates with specific month and day

Leave a Comment