How to custom sort pandas multi-index?

The following code generates the pandas table named out. import pandas as pd import numpy as np df = pd.DataFrame({'Book': ['B1', 'B1', 'B2', 'B3', 'B3', 'B3'], 'Trader': ['T1', 'Z2', 'Z2', 'T1', 'U3', 'T2'], ...
more »

2017-01-11 20:01 (2) Answers

Define trend pandas/python

I have dataset: print (df['price']) 0 0.435 1 -2.325 2 -3.866 ... 58 -35.876 59 -37.746 Name: price, dtype: float64 moving average: m_a = df['price'].rolling(window=5).mean() m_a.plot() print(m_a) 0 NaN 1 NaN 2 ...
more »

2017-01-11 20:01 (2) Answers

Pandas - Compare positive/negative values

I have a dataframe "df": x y 0 1 -1 1 -2 -3 2 3 4 3 4 5 4 9 6 I am trying to determine what percentage of x and y values are in agreement in terms of being positive or negative. So if x is positive and y is positive, that wou...
more »

2017-01-11 18:01 (1) Answers

Pandas: How to make apply on dataframe faster?

Consider this pandas example where I'm calculating column C by multiplying A with B and a float if a certain condition is fulfilled using apply with a lambda function: import pandas as pd df = pd.DataFrame({'A':[1,2,3,4,5,6,7,8,9],'B':[9,8,7,6,5,4,3...
more »

2017-01-11 11:01 (4) Answers

Issue with joining repeated values/rows

New to python, cant seem to understand how to proceed. After using bin and editing my data frame i was able to come up with this : Continents % Renewable Country 0 Asia (15.753, 29.227] China 1 North America (2.212, 15.753] United S...
more »

2017-01-11 08:01 (2) Answers

How to create sparse boolean mask in Pandas?

I have the following code for mask filtering of df : for i, y in enumerate(cols) : dfm = df[y].str.contains(s) mask= dfm if i==0 else np.column_stack((mask, dfm)) df is not sparse, but the filtering results mask is sparse. Storing the mas...
more »

2017-01-11 06:01 (1) Answers

Updating a panda df using a second df

My problem is that I have a dataframe (df1) with a start and stop column and then a counter column. I have a separate dataframe (df2) with a value and a count column. I want to find the row in df1 whose start and stop contains the value of df2 and th...
more »

2017-01-10 22:01 (1) Answers

Bug on astype pandas?

I am working with timedeltas and it seems this code copy_for_U.Time.astype('timedelta64[m]',copy=False); does not change the dataframe - as it should, if I understood correctly from the doc, where it says: Signature: full_df.Time.astype(dtype,...
more »

2017-01-10 11:01 (0) Answers

How to make df.to_sql() create varchar2 object

I have a DataFrame which consists of a column of strings. If I do df.to_sql() to save it as a table into an Oracle database, the column is of CLOB type and I need to convert it. I wonder if I can specify the type (say varchar2) when I create the tabl...
more »

2017-01-10 11:01 (2) Answers

Python - Fast HDF5 Time Series Data Queries

I need to do a lot of successive queries on time series data in specific time spans from a HDF5 database (the data is stord in seconds, not always "continuous", I only know the start and end time). Therefore, I wonder wether there is a faster solutio...
more »

2017-01-09 20:01 (1) Answers

Getting a multidimensional array out of pandas

Hi I am getting started with pandas/numpy and I am running into a few snags. I vectorized an image and stored the data in a pandas column. misc.imresize(misc.imread(path, mode='RGB') The data looks fine, but I just can't get it out in an array ...
more »

2017-01-09 15:01 (2) Answers

Pandas DataFrame subtract cross-section

I have a pandas DataFrame with the data from a 3D-measurement (some 27k rows). I have already created a multi-index consisting of the 3 coordinate columns (x, y, z). The data looks like that (multiple xz-planes along the y-direction): ...
more »

2017-01-09 15:01 (1) Answers