Debian based images use only python pip to install packages with .whl format: Downloading pandas-0.22.0-cp36-cp36m-manylinux1_x86_64.whl (26.2MB) Downloading numpy-1.14.1-cp36-cp36m-manylinux1_x86_64.whl (12.2MB) WHL format was developed as a quicker and more reliable method of installing Python software than re-building from source code every time. WHL files only have to be moved to the correct location on the target … Read more
You can use pd.merge: import pandas as pd pd.merge(df1, df2, on=”movie_title”) Only rows are kept for which common keys are found in both data frames. In case you want to keep all rows from the left data frame and only add values from df2 where a matching key is available, you can use how=”left”: pd.merge(df1, … Read more
Solution with MultiLabelBinarizer Assuming sequences is an array of integers with maximum possible value upto dimension-1, we can use MultiLabelBinarizer from sklearn.preprocessing to replicate the behaviour of the function vectorize_sequences from sklearn.preprocessing import MultiLabelBinarizer mlb = MultiLabelBinarizer(classes=range(dimension)) mlb.fit_transform(sequences) Solution with Numpy broadcasting Assuming sequences is an array of integers with maximum possible value upto dimension-1 … Read more
You can use ordered catagorical first: cats = [ ‘Monday’, ‘Tuesday’, ‘Wednesday’, ‘Thursday’, ‘Friday’, ‘Saturday’, ‘Sunday’] df[‘Day of Week’] = df[‘Day of Week’].astype(‘category’, categories=cats, ordered=True) In pandas 0.21.0+ use: from pandas.api.types import CategoricalDtype cat_type = CategoricalDtype(categories=cats, ordered=True) df[‘Day of Week’] = df[‘Day of Week’].astype(cat_type) Or reindex: df_weekday = df.groupby([‘Day of Week’]).sum().reindex(cats)
You can map dict values inside fillna df.B = df.B.fillna(df.A.map(dict)) print(df) A B 0 a 2 1 b 5 2 c 4
For a Kalman filter it is useful to represent the input data with a constant time step. Your sensors send data randomly, so you can define the smallest significant time step for your system and discretize the time axis with this step. For example one of your sensors sends data approximately each 0.2 seconds and … Read more
You could first find if element is NaN or not by isnull() and then take row-wise sum(axis=1) In [195]: df.isnull().sum(axis=1) Out[195]: 0 0 1 0 2 0 3 3 4 0 5 0 dtype: int64 And, if you want the output as list, you can In [196]: df.isnull().sum(axis=1).tolist() Out[196]: [0, 0, 0, 3, 0, 0] … Read more
TL;DR I believe you’re seriously underestimating memory requirements. Even assuming that data is fully cached, storage info will show only a fraction of peak memory required for bringing data back to the driver. First of all Spark SQL uses compressed columnar storage for caching. Depending on the data distribution and compression algorithm in-memory size can … Read more
You can use tz_localize to set the timezone to UTC/+0000, and then tz_convert to add the timezone you want: start = pd.to_datetime(‘2015-02-24’) rng = pd.date_range(start, periods=10) df = pd.DataFrame({‘Date’: rng, ‘a’: range(10)}) df.Date = df.Date.dt.tz_localize(‘UTC’).dt.tz_convert(‘Asia/Kolkata’) print (df) Date a 0 2015-02-24 05:30:00+05:30 0 1 2015-02-25 05:30:00+05:30 1 2 2015-02-26 05:30:00+05:30 2 3 2015-02-27 05:30:00+05:30 3 … Read more
Use shift. df[‘dA’] = df[‘A’] – df[‘A’].shift(-1)