dask dataframe how to convert column to to_datetime

by

Use astype

You can use the astype method to convert the dtype of a series to a NumPy dtype

df.time.astype('M8[us]')

There is probably a way to specify a Pandas style dtype as well (edits welcome)

Use map_partitions and meta

When using black-box methods like map_partitions, dask.dataframe needs to know the type and names of the output. There are a few ways to do this listed in the docstring for map_partitions.

You can supply an empty Pandas object with the right dtype and name

meta = pd.Series([], name="time", dtype=pd.Timestamp)

Or you can provide a tuple of (name, dtype) for a Series or a dict for a DataFrame

meta = ('time', pd.Timestamp)

Then everything should be fine

df.time.map_partitions(pd.to_datetime, meta=meta)

If you were calling map_partitions on df instead then you would need to provide the dtypes for everything. That isn’t the case in your example though.

More Related Contents:

  1. python dask DataFrame, support for (trivially parallelizable) row apply?
  2. how to parallelize many (fuzzy) string comparisons using apply in Pandas?
  3. GroupBy results to dictionary of lists
  4. How to remove nan value while combining two column in Panda Data frame?
  5. Restart cumsum and get index if cumsum more than value
  6. Remove non-numeric rows in one column with pandas
  7. How to deal with multi-level column names downloaded with yfinance
  8. Replace all occurrences of a string in a pandas dataframe (Python)
  9. Running get_dummies on several DataFrame columns?
  10. ValueError: numpy.dtype has the wrong size, try recompiling
  11. Get business days between start and end date using pandas
  12. How to change the datetime tick label frequency for matplotlib plots
  13. Pandas DataFrame aggregate function using multiple columns
  14. Python Pandas: Is Order Preserved When Using groupby() and agg()?
  15. Pass percentiles to pandas agg function
  16. Capitalize first letter of each word in a dataframe column
  17. Compute the running (cumulative) maximum for a series in pandas
  18. Get total of Pandas column
  19. Sorting columns and selecting top n rows in each group pandas dataframe
  20. Get first and second highest values in pandas columns
  21. ImportError: No module named ‘pandas.indexes’
  22. select columns based on columns names containing a specific string in pandas
  23. Enumerate each row for each group in a DataFrame
  24. Python How to use ExcelWriter to write into an existing worksheet
  25. replace value by using regex to np.nan
  26. Pandas groupby with bin counts
  27. Pandas groupby and qcut
  28. computing the mean for python datetime
  29. How to convert string representation of dictionary in Pandas DataFrame to a new columns?
  30. Pandas explode multiple columns [duplicate]

Leave a Comment