replace value by using regex to np.nan

by

NaN is consistently used as a placeholder for missing, when replacing part of a string with “missing” it can only mean the entire entry is compromised. I’ve heard this called NaN pollution (or similar, will see if I can find some references), in that if NaN touches the data is compromised.

That said, that’s not always the case:

In [11]: s = pd.Series([1, 2, np.nan, 4])

In [12]: s.sum()
Out[12]: 7.0

In [13]: s.sum(skipna=False)
Out[13]: nan

In some languages you’ll see skipna=False as the default behaviour, some vehemently argue that NaN should always pollute all data. Pandas takes a somewhat more pragmatic approach…

The real question is what do you expect it to do in the case of NaN?

More Related Contents:

  1. change column name in pandas, which will be used to change name for 1 column
  2. Convert DataFrame column type from string to datetime
  3. How to save a new sheet in an existing excel file, using Pandas?
  4. Pandas read in table without headers
  5. Constructing pandas DataFrame from values in variables gives “ValueError: If using all scalar values, you must pass an index”
  6. Can pandas automatically read dates from a CSV file?
  7. Pandas cannot open an Excel (.xlsx) file
  8. pandas dataframe str.contains() AND operation
  9. Pandas groupby mean – into a dataframe?
  10. Python Pandas: Check if string in one column is contained in string of another column in the same row
  11. Pandas: Filtering multiple conditions
  12. How to determine the length of lists in a pandas dataframe column
  13. Pandas fill missing values in dataframe from another dataframe
  14. Filter a pandas dataframe using values from a dict
  15. What is the difference between a pandas Series and a single-column DataFrame?
  16. How to convert a Scikit-learn dataset to a Pandas dataset
  17. Get data from pandas into a SQL server with PYODBC
  18. Find the unique values in a column and then sort them
  19. how do I calculate a rolling idxmax
  20. binning a dataframe in pandas in Python [duplicate]
  21. What is the difference between pandas agg and apply function?
  22. Pandas: Return Hour from Datetime Column Directly
  23. How to filter a pandas dataframe based on the length of a entry
  24. How can I group a stacked bar chart?
  25. How is pandas groupby method actually working?
  26. How to rotate x-axis tick labels in a pandas plot
  27. Fixed effect in Pandas or Statsmodels
  28. Resample hourly TimeSeries with certain starting hour
  29. Pandas how to apply multiple functions to dataframe
  30. Pandas, filter rows which column contain another column

Leave a Comment