Why is pandas.to_datetime slow for non standard time format such as ‘2014/12/31’

by

This is because pandas falls back to dateutil.parser.parse for parsing the strings when it has a non-default format or when no format string is supplied (this is much more flexible, but also slower).

As you have shown above, you can improve the performance by supplying a format string to to_datetime. Or another option is to use infer_datetime_format=True

Apparently, the infer_datetime_format cannot infer when there are microseconds. With an example without those, you can see a large speed-up:

In [28]: d = '2014-12-24 01:02:03'

In [29]: c = re.sub('-', "https://stackoverflow.com/", d)

In [30]: s_c = pd.Series([c]*10000)

In [31]: %timeit pd.to_datetime(s_c)
1 loops, best of 3: 1.14 s per loop

In [32]: %timeit pd.to_datetime(s_c, infer_datetime_format=True)
10 loops, best of 3: 105 ms per loop

In [33]: %timeit pd.to_datetime(s_c, format="%Y/%m/%d %H:%M:%S")
10 loops, best of 3: 99.5 ms per loop

More Related Contents:

  1. How do I read a large csv file with pandas?
  2. Convert DataFrame column type from string to datetime
  3. How to add pandas data to an existing csv file?
  4. Import CSV file as a pandas DataFrame
  5. Changing the formatting of a datetime axis in matplotlib
  6. How to read a file with a semi colon separator in pandas
  7. How to get rid of “Unnamed: 0” column in a pandas DataFrame read in from CSV file?
  8. Pandas split column into multiple columns by comma
  9. How to read file with space separated values in pandas
  10. Convert commas decimal separators to dots within a Dataframe
  11. Pandas: ValueError: cannot convert float NaN to integer
  12. How do I round datetime column to nearest quarter hour
  13. pandas DataFrame output end of csv
  14. Use Multiple Character Delimiter in Python Pandas read_csv
  15. Get pandas.read_csv to read empty values as empty string instead of nan
  16. Panda’s Write CSV – Append vs. Write
  17. UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x96 in position 35: invalid start byte
  18. Loading multiple csv files of a folder into one dataframe
  19. Pandas read_csv dtype leading zeros
  20. pandas.read_csv from string or package data
  21. Pandas ParserError EOF character when reading multiple csv files to HDF5
  22. Pandas groupby to to_csv
  23. Python Pandas – Read csv file containing multiple tables
  24. Is there a way to speed up handling large CSVs and dataframes in python?
  25. How do you remove the column name row when exporting a pandas DataFrame?
  26. Pandas read_csv dtype read all columns but few as string
  27. pandas dataframe with 2-rows header and export to csv
  28. computing the mean for python datetime
  29. start index at 1 for Pandas DataFrame
  30. pandas – convert string into list of strings [duplicate]

Leave a Comment