You can use isnull with mean for threshold and then remove columns by boolean indexing with loc (because remove columns), also need invert condition – so <.8 means remove all columns >=0.8: df = df.loc[:, df.isnull().mean() < .8] Sample: np.random.seed(100) df = pd.DataFrame(np.random.random((100,5)), columns=list(‘ABCDE’)) df.loc[:80, ‘A’] = np.nan df.loc[:5, ‘C’] = np.nan df.loc[20:, ‘D’] = … Read more
Try this: In [107]: pd.isnull(df.iloc[1,0]) Out[107]: True UPDATE: in a newer Pandas versions use pd.isna(): In [7]: pd.isna(df.iloc[1,0]) Out[7]: True
Pandas NaT behaves like a floating-point NaN, in that it’s not equal to itself. Instead, you can use pandas.isnull: In [21]: pandas.isnull(pandas.NaT) Out[21]: True This also returns True for None and NaN. Technically, you could also check for Pandas NaT with x != x, following a common pattern used for floating-point NaN. However, this is … Read more
Assuming your DataFrame is in df: df.Temp_Rating.fillna(df.Farheit, inplace=True) del df[‘Farheit’] df.columns=”File heat Observations”.split() First replace any NaN values with the corresponding value of df.Farheit. Delete the ‘Farheit’ column. Then rename the columns. Here’s the resulting DataFrame:
You can use multiple boolean conditions to test if the current value and previous value are NaN: In [3]: df = pd.DataFrame({‘a’:[1,3,np.NaN, np.NaN, 4, np.NaN, 6,7,8]}) df Out[3]: a 0 1 1 3 2 NaN 3 NaN 4 4 5 NaN 6 6 7 7 8 8 In [6]: df[(df.a.isnull()) & (df.a.shift().isnull())] Out[6]: a 3 … Read more
[Updated to adapt to modern pandas, which has isnull as a method of DataFrames..] You can use isnull and any to build a boolean Series and use that to index into your frame: >>> df = pd.DataFrame([range(3), [0, np.NaN, 0], [0, 0, np.NaN], range(3), range(3)]) >>> df.isnull() 0 1 2 0 False False False 1 … Read more
The reason why your code doesn’t work is because Math.max is expecting each parameter to be a valid number. This is indicated in the documentation as follows: If at least one of arguments cannot be converted to a number, the result is NaN. In your instance you are only providing 1 argument, and that 1 … Read more
You could enable floating point exceptions – see glibc Control Functions – then you’ll get a SIGFPE when your NaN value is produced
You could first find if element is NaN or not by isnull() and then take row-wise sum(axis=1) In [195]: df.isnull().sum(axis=1) Out[195]: 0 0 1 0 2 0 3 3 4 0 5 0 dtype: int64 And, if you want the output as list, you can In [196]: df.isnull().sum(axis=1).tolist() Out[196]: [0, 0, 0, 3, 0, 0] … Read more
isNaN() and Number.isNaN() both test if a value is (or, in the case of isNaN(), can be converted to a number-type value that represents) the NaN value. In other words, “NaN” does not simply mean “this value is not a number”, it specifically means “this value is a numeric Not-a-Number value according to IEEE-754″. The … Read more