Numpy arrays are not a replacement for lists that you just drop in and everything runs faster. Numpy arrays are a commitment to doing things a certain way. If used correctly, Numpy arrays can be blazing fast. If used naively, they make things very slow.

The short answer to how to use numpy correctly is to act upon whole arrays in one swoop with universal functions (ufuncs). Rather than looping through the array, a ufunc applies a compiled function directly on the area of memory represented by the array. “+”, “-“, “*’, and “/” are interpreted as ufuncs when applied to an array. Others are listed here. If what you want can’t be done with these operations acting on big sections of memory, then you might consider writing a ufunc in c, numba.vectorize, cython, or just not using numpy.

Anyways, here is an example. Using numpy incorrectly led to the code running 4x slower than not using numpy at all. Using numpy the way it was designed to be used resulted in the code running in 1/30th the time as python lists. Using numpy incorrectly is 100x slower than using it correctly.

First, the results are:


Setting up data for test...
Building huge numpy arrays is fast
Done setting up numpy arrays in 0.13 s
starting tests

numpy working on arrays...
done in 0.07 s


and the code for you to run is:

'''calculate the normals of some lines'''
import numpy as np
 
def numpy_arrays1(segs):
    '''segs = [[x1,y1,z1,x2,y2,z2],...] => normals [[xn,yn,zn],...]
    this function makes a tmp array inorder to reuse memory and not
    create unnecessary intermediate arrays.'''

    result = segs[:,3:] - segs[:,:3]
    tmp = np.empty((len(segs),4),dtype=segs.dtype)
    tmp[:,:3] = np.square(result)
    #noticed that if the result of sum goes to an element
    #of the same array, that element is not part of the sum.
    #tmp[:,3] here is still garbage, so we can write over it
    tmp[:,3] = np.sum(tmp,axis=1,out=tmp[:,3]) # SAFE? seems to be
    tmp[:,3] = np.sqrt(tmp[:,3])
    #lame, why do this in two step?
    result[:,0] = result[:,0] / tmp[:,3]  #TODO use newaxis 
    result[:,1] = result[:,1] / tmp[:,3]  # and do 3 in 1!
    result[:,2] = result[:,2] / tmp[:,3]
    return result
 
def python_loop(segs):
    '''notice the for loop, and appending to the result list.
    otherwise, code is the same'''
    result = []
    for seg in segs:
        source = [seg[3+i]-seg[i] for i in xrange(3)]
        squared = [s**2 for s in source]
        length = sum(squared)**.5
        result.append([s/length for s in source])
    return result

  
 
if __name__ == '__main__':
    from time import time
    from math import cos, sin, sqrt
 
    print "Setting up data for test..."
    print "  Building huge numpy arrays is fast"
    now = time()
    n = 1000000
    ts = np.linspace(0,2*np.pi,n)
    np.seterr(all='ignore') #I might just divide by zero
 
    #an r is [x1,y1,x2,y2]
    segs = np.ones((n,6),dtype=np.float64)
    segs[:,3] = 1+np.cos(ts)
    segs[:,4] = 1+np.sin(ts)
    segs[:,5] = np.arange(n)
 
    print "  Done setting up numpy arrays in %.2f s"%(time()-now)
 
    print "starting tests\n"
    if True:
 
      print "Actually, first build some python lists"
      print "It takes MUCH longer to build the"
      print "   equivilent Python lists..."
 
      now = time()
      pylist = [[1,1,1,1+cos(t),1+sin(t),i] for t,i in zip(ts,xrange(n))]
      print "  done setting up lists in %.2f s"%(time()-now)
      print "Now the tests\n\n"
      print "cpython working on python lists..."
      now = time()
      norm1 = python_loop(pylist)
      print "  done in %.2f s\n"%(time()-now)
 
    if True:
      print "cpython working on numpy arrays..."
      now = time()
      norm2 = python_loop(segs)
      print "  done is %.2f s\n"%(time()-now)
 
    if True:
      print "numpy working on arrays..."
      now = time()
      norm3 = numpy_arrays1(segs)
      a = time()-now
      print "  done in %.2f s"%(a)