## minimize a loss function using a specific form of the gradient of loss w.r.t its parameters

In machine learning, given a loss function to be minimized, we usually opt to some machine learning library to update parameter. For example, in tensorflow, we usually do the following thing, first write down loss function, self.loss = F(\theta, \et...
more »

## Multiply Series with DataFrame where series.index == dataframe.columns

Hi I have a very simple scenario which I don't know how to solve via a simple method. I have the following dataframes: In [30]: d = pd.DataFrame([1,2,3], index=['A', 'B', 'C']) In [31]: d Out[31]: 0 A 1 B 2 C 3 In [32]: dd = pd.DataFrame({...
more »

## How can I remove duplicates from a list of matrices based on some equivalence relation?

Given some list of symmetric integer matrices, I want to remove all duplicates under the following equivalence relation: Two k x k matrices M1, M2 are equivalent if there is some permutation s on {1,...,k} such that for all i and j in {1,...,k} we h...
more »

## Convert pandas Series with categories as values to DataFrame mask

I have a very simple problem which I'm unsure how to solve with a few simple calls. I have the following Series: In [3]: d = pd.Series(data=['A', 'A', 'B', 'C', 'D', 'D', 'B'], index=range(7)) In [4]: d Out[4]: 0 A 1 A 2 B 3 C 4 D 5...
more »

## How to replace a list of values in a numpy array?

I have an unsorted array of numbers. I need to replace certain numbers (given in a list) with specific alternatives (also given in a corresponding list) I wrote the following code (which seems to works): import numpy as np numbers = np.arange(...
more »

## How do I plot a vector field within an arbitrary plane using Python?

I have a 3d velocity vector field in a numpy array of shape (zlength, ylength, xlength, 3). The '3' contains the velocity components (u,v,w). I can quite easily plot the vector field in the orthogonal x-y, x-z, and y-z planes using quiver, e.g. X, ...
more »

## Get # of contiguous hits and their first/last index in a NumPy array

Here is an itertools solution that returns a list of the lengths of each contiguous block. Here a contiguous block is a run of 1s without breaks in between. Is there a way to also have itertools return the index associated with each block? import ...
more »

## Group by column in pandas dataframe and average arrays

I have a movie dataframe with movie names, their respective genre, and vector representation (numpy arrays). ID Year Title Genre Word Vector 1 2003.0 Dinosaur Planet Documentary [-0.55423898, -0.72544044, 0.33189204, -0.1720... 2 2004.0...
more »

## A masked array indexing issue

I have a numpy array with some NaN values: arr = [ 0, NaN, 2, NaN, NaN, 5, 6, 7 ] Using some logic (outside of the question scope), I generate a mask of the NaN locations: mask = [ True, False, True, False, False, True, True, True ] I use this ...
more »

## AttributeError: 'numpy.ndarray' object has no attribute 'toList'

I'm trying to append certain columns of Pandas Data Frames from CSV files into a numpy array. I have no idea how to instantiate an empty numpy array, so I'm testing it first with a list. def windows(files): x = [] for my_files in files: ...
more »

## Keras custom recall metric based on predicted values

I would like to implement a custom metric in keras that calculates the recall assuming that the top k% most probable y_pred_probs's are true. In numpy I would do it as follows. Sort the y_preds_probs. Then take the value at the kth index. Note k=0.5...
more »

## Converting rows of dataframes into matrices - Python Pandas

I have the following dataframe: Pclass Sex Fare Embarked Title Family 3 0 1.0 0 0 1 1 1 2.0 1 3 1 3 1 1.0 1 2 0 1 ...
more »

## Double dot product with broadcasting in numpy

I have the following operation : import numpy as np x = np.random.rand(3,5,5) w = np.random.rand(5,5) y=np.zeros((3,5,5)) for i in range(3): y[i] = np.dot(w.T,np.dot(x[i],w)) Which corresponds to the pseudo-expression y[m,i,j] = sum( w[k,i] ...
more »

## String matching plus Boolean value reassignment in Pandas

I'm trying to find all the value, which is a more complex query than I'm used to. I'll be changing the values of millions of values potentially, so the most efficient way to find rows, which meet these conditions, and how to change their values would...
more »

## Numpy Array random mutation

I'm coding my first genetic algorithm in Python. I particularly care about the optimization and population scalability. import numpy as np population = np.random.randint(-1, 2, size=(10,10)) Here I make a [10,10] array, with random number between ...
more »

## can we change the vector into matrix in numpy with the elements in the vector repeats in a matrix with m*n dimensions

import numpy as np import numba @numba.vectorize('i4(i4)', target = 'parallel') def mag(b): return b * b def main(): mat_a = np.full((5, 3),2,dtype=np.int32) c = mag(mat_a) d = np.sum(c, axis = 1) print d OUTPUT: [12 12 ...
more »

## print mismatch items in two array

I want to compare two array(4 floating point)and print mismatched items. I used this code: >>> from numpy.testing import assert_allclose as np_assert_allclose >>> x=np.array([1,2,3]) >>> y=np.array([1,0,3]) >>> np...
more »

## Python: Improving Image-processing with numpy

Let there be two big (2000x2000 or higher) .tiff images consisting only of numpy float32 values (no rgb). I call them Image A and B. I want to multiply them in a special way: Find the max value in B and roll it (using numpy.roll) to the upper-left ...
more »

## Numpy vectorized function assignment with a boolean statement

I would like to assign a function that has a boolean evaluation in it, using a fast way. Here is a simple example. I want the following function to be evaluated for arbitrary a and b: a = 0.5 b = 0.6 def func(x): x=max(x,a) if x>b: ...
more »

## In python, how to convert list of dates into number of days since first visit?

I have a list of data showing dates of hospital visits alongside patient ID numbers, data was originally a pandas dataframe which I wrote to a CSV file which now looks similar to this: | Patient | Date | |---------|------------| | 2 | 1...
more »

## Efficiently using Numpy to assign function values to array

I am interested in finding the fastest way of carrying a simple operation in Python3.6 using Numpy. I wish to create a function and from a given array to an array of function values. Here is a simplified code that does that using map: import numpy a...
more »

## How extract numpy array features from spectrogram?

I created a spectrogram with this code: D = librosa.amplitude_to_db(librosa.stft(y), ref=np.max) librosa.display.specshow(D, y_axis='linear', x_axis='time') plt.colorbar(format='%+2.0f dB') plt.title('Log-frequency power spectrogram') plt.savefig('s...
more »

## How does dtype affect row and column operation speed in Numpy?

I'm trying to understand how to best utilize the C-ordering of numpy arrays to write high performance code. My expectation was that operations which traverse rows should be faster than those which traverse columns. Indeed, this was true for the fir...
more »

## Why does complex floating-point division underflow weirdly with NumPy?

Consider this code: import numpy numpy.seterr(under='warn') x1 = 1 + 1j / (1 << 533) x2 = 1 - 1j / (1 << 533) y1 = x1 * 1.1 y2 = x2 * 1.1 z1 = x1 / 1.1 z2 = x2 / 1.1 print(numpy.divide(1, x1)) # 1-3.55641399918e-161j # OK ...
more »

## How to decode Bell 202 signal? (APRS data from International Space Station)

I am having some difficulties in understanding how to decode a Bell 202 encoded signal. How do I do that? I know Bell202 encodes binary data with ones and zeros as 1200 Hz and 2200 Hz. I recorded a radio signal on 145.825 Mhz with a 2Mhz sampling ra...
more »

## Numpy - Find maximum point and value of data points

I just started to learn Numpy (and Scipy). I wrote a program to compute plot points for an f(x) function. (f(x) can't be given explicitly as I have to numerically solve an equation for each point.) I put the values in a 2D array: [[x1, x2, x3,...
more »

## How to identify significant items from a correlation matrix in Python (without inner loop)

I have built a correlation matrix output from a small test set and ended up with the following. True values are those that are greater than a defined value (e.g. results = correlation_matrix > 0.75 ) [[False False False True] [False False True F...
more »

## Trouble merging Scraped data using Pandas and numpy in Python

I am trying to collect information from a lot of different urls and combine the data based on the year and Golfer name. As of now I am trying to write the information to csv and then match using pd.merge() but I have to use a unique name for each dat...
more »

## Pandas Data Frame filtering based off a column condition

I am trying to remove all the rows from a data frame where a certain condition is met in one column. Lets say my data frame is this: index 'Value 1' 'Grade' 1 10 2170A 2 15 2170B 3 10 NCR2170B 4 ...
more »

## Optimal value is outside range when doing bruteforce optimization using Scipy

I'm following the example given in scipy's optimize documentation to do brute-force optimization on a function with 3 parameters. This is the function I wish to optimize: def entropy_of_hyperplane(z): w0, w1, b = z # do some calculations.. ...
more »

## python - 2D numpy array from a pandas dataframe row with delimited range

I am newbie on python and I loaded a big data from a csv into a pandas dataframe. However, I cannot find a method to create a 2d array for each row of the dataframe where each row of the new np array correspond to X range of values. For example, in m...
more »

## Difference of precision/display between numpy.tolist() and list()

This is kind of a follow up to coldspeed's question. (And this is not a duplicate of is floating point math broken ? BTW) I'm converting a list of lists to a numpy array, and then trying to convert it back to a python list of lists. import numpy a...
more »

## Fix precision issues when *displaying* floats in python

I'm reading out a text file with some float numbers using np.loadtxt . This is what my numpy array looks like: x = np.loadtxt(t2) print(x) array([[ 1.00000000e+00, 6.61560000e-13], [ 2.00000000e+00, 3.05350000e-13], [ 3.0000000...
more »

## How exactly does Keras take dimension argumentsfor LSTM / time series problems?

I can't seem to find a concrete answer to the question of how to feed data into Keras. Most examples seem to work off image / text data and have clearly defined data points. I'm trying to feed music into an LSTM neural network. I want the network t...
more »