How to use pnumpy

pnumpy functions

Pnumpy now calls init at startup to setup the package. This imports NumPy, replacing all the inner loops of UFuncs with wrapped versions. Then you can enable/disable any of the subsystems:

  • threading

    Threading will kick in when the number of elements to be processed is more than 50,000. It will break the operation into chunks. Each chunk will be executed in its own thread.

  • ledger

    The ledger records data on each loop execution to enable more accurate heuristics on memory allocation, threading behavior and reporting for logging and benchmarking.

  • recycler

    Once we can change the NumPy memory allocation strategy, we can use the data from the ledger to create more performant memory caches.

  • atop

    Provide faster implementations of NumPy inner loops.

pnumpy.atop_disable()

disable the atop inner loop implementations.

pnumpy.atop_enable()

enable the atop inner loop implementations.

pnumpy.atop_info()

return dict

pnumpy.atop_isenabled()

returns True if atop enabled, else False

pnumpy.atop_setworkers()

set workers for a func

pnumpy.disable()

Call to disable the atop engine, stop any threads, and unhook numpy functions.

Parameters

None –

Returns

Return type

None

See also

pn.enable(), pn.atop_info()

pnumpy.enable()

Call to enable the atop engine, use threads, and hook numpy functions.

Parameters

None –

Returns

Return type

None

See also

pn.disable(), pn.atop_info()

pnumpy.getitem()

alternative to fancy index or boolean index

pnumpy.lexsort(*args, **kwargs)

Perform an indirect stable sort using a sequence of keys.

Given multiple sorting keys, which can be interpreted as columns in a spreadsheet, lexsort returns an array of integer indices that describes the sort order by multiple columns. The last key in the sequence is used for the primary sort order, the second-to-last key for the secondary sort order, and so on. The keys argument must be a sequence of objects that can be converted to arrays of the same shape. If a 2D array is provided for the keys argument, it’s rows are interpreted as the sorting keys and sorting is according to the last row, second last row etc.

Parameters

  • keys ((k, N) array or tuple containing k (N,)-shaped sequences) – The k different “columns” to be sorted. The last column (or row if keys is a 2D array) is the primary sort key.

  • axis (int, optional) – Axis to be indirectly sorted. By default, sort over the last axis.

Returns

  • indices ((N,) ndarray of ints) – Array of indices that sort the keys along the specified axis.

  • Threading

  • ———

  • Up to 8 threads

See also

argsort()

Indirect sort.

ndarray.sort()

In-place sort.

sort()

Return a sorted copy of an array.

Examples

Sort names: first by surname, then by name.

>>> surnames =    ('Hertz',    'Galilei', 'Hertz')
>>> first_names = ('Heinrich', 'Galileo', 'Gustav')
>>> ind = np.lexsort((first_names, surnames))
>>> ind
array([1, 2, 0])

>>> [surnames[i] + ", " + first_names[i] for i in ind]
['Galilei, Galileo', 'Hertz, Gustav', 'Hertz, Heinrich']

Sort two columns of numbers:

>>> a = [1,5,1,4,3,4,4] # First column
>>> b = [9,4,0,4,0,2,1] # Second column
>>> ind = np.lexsort((b,a)) # Sort by a, then by b
>>> ind
array([2, 0, 4, 6, 5, 3, 1])

>>> [(a[i],b[i]) for i in ind]
[(1, 0), (1, 9), (3, 0), (4, 1), (4, 2), (4, 4), (5, 4)]

Note that sorting is first according to the elements of a. Secondary sorting is according to the elements of b.

A normal argsort would have yielded:

>>> [(a[i],b[i]) for i in np.argsort(a)]
[(1, 9), (1, 0), (3, 0), (4, 4), (4, 2), (4, 1), (5, 4)]

Structured arrays are sorted lexically by argsort:

>>> x = np.array([(1,9), (5,4), (1,0), (4,4), (3,0), (4,2), (4,1)],
...              dtype=np.dtype([('x', int), ('y', int)]))

>>> np.argsort(x) # or np.argsort(x, order=('x', 'y'))
array([2, 0, 4, 6, 5, 3, 1])
pnumpy.recarray_to_colmajor(item, parallel=True)

Converts a numpy record array (void type) to a dictionary of numpy arrays, col major

Returns

Return type

A dictionary of numpy arrays corresponding to the original numpy record array.

Examples

>>> x=np.array([(1.0, 2, 3, 4, 5, 'this is a long test'), (3.0, 4, 5, 6, 7, 'short'), (30.0, 40, 50, 60, 70, '')],
            dtype=[('x', '<f4'), ('y', '<i2'), ('z', 'i8'),('zz','i8'),('yy','i4'),('str','<S20')])
>>> item=np.tile(x,100_000)
>>> mydict = recarray_to_colmajor(item)
pnumpy.sort(a, axis=- 1, kind=None, order=None)

Return a sorted copy of an array.

Parameters

  • a (array_like) – Array to be sorted.

  • axis (int or None, optional) – Axis along which to sort. If None, the array is flattened before sorting. The default is -1, which sorts along the last axis.

  • kind ({'quicksort', 'mergesort', 'heapsort', 'stable'}, optional) –

    Sorting algorithm. The default is ‘quicksort’. Note that both ‘stable’ and ‘mergesort’ use timsort or radix sort under the covers and, in general, the actual implementation will vary with data type. The ‘mergesort’ option is retained for backwards compatibility.

    Changed in version 1.15.0.: The ‘stable’ option was added.

  • order (str or list of str, optional) – When a is an array with fields defined, this argument specifies which fields to compare first, second, etc. A single field can be specified as a string, and not all fields need be specified, but unspecified fields will still be used, in the order in which they come up in the dtype, to break ties.

Returns

  • sorted_array (ndarray) – Array of the same type and shape as a.

  • Threading

  • ———

  • Up to 8 threads

See also

ndarray.sort()

Method to sort an array in-place.

argsort()

Indirect sort.

lexsort()

Indirect stable sort on multiple keys.

searchsorted()

Find elements in a sorted array.

partition()

Partial sort.

Notes

The various sorting algorithms are characterized by their average speed, worst case performance, work space size, and whether they are stable. A stable sort keeps items with the same key in the same relative order. The four algorithms implemented in NumPy have the following properties:

kind

speed

worst case

work space

stable

‘quicksort’

1

O(n^2)

0

no

‘heapsort’

3

O(n*log(n))

0

no

‘mergesort’

2

O(n*log(n))

~n/2

yes

‘timsort’

2

O(n*log(n))

~n/2

yes

Note

The datatype determines which of ‘mergesort’ or ‘timsort’ is actually used, even if ‘mergesort’ is specified. User selection at a finer scale is not currently available.

All the sort algorithms make temporary copies of the data when sorting along any but the last axis. Consequently, sorting along the last axis is faster and uses less space than sorting along any other axis.

The sort order for complex numbers is lexicographic. If both the real and imaginary parts are non-nan then the order is determined by the real parts except when they are equal, in which case the order is determined by the imaginary parts.

Previous to numpy 1.4.0 sorting real and complex arrays containing nan values led to undefined behaviour. In numpy versions >= 1.4.0 nan values are sorted to the end. The extended sort order is:

  • Real: [R, nan]

  • Complex: [R + Rj, R + nanj, nan + Rj, nan + nanj]

where R is a non-nan real value. Complex values with the same nan placements are sorted according to the non-nan part if it exists. Non-nan values are sorted as before.

New in version 1.12.0.

quicksort has been changed to introsort. When sorting does not make enough progress it switches to heapsort. This implementation makes quicksort O(n*log(n)) in the worst case.

‘stable’ automatically chooses the best stable sorting algorithm for the data type being sorted. It, along with ‘mergesort’ is currently mapped to timsort or radix sort depending on the data type. API forward compatibility currently limits the ability to select the implementation and it is hardwired for the different data types.

New in version 1.17.0.

Timsort is added for better performance on already or nearly sorted data. On random data timsort is almost identical to mergesort. It is now used for stable sort while quicksort is still the default sort if none is chosen. For timsort details, refer to CPython listsort.txt. ‘mergesort’ and ‘stable’ are mapped to radix sort for integer data types. Radix sort is an O(n) sort instead of O(n log n).

Changed in version 1.18.0.

NaT now sorts to the end of arrays for consistency with NaN.

Examples

>>> a = np.array([[1,4],[3,1]])
>>> np.sort(a)                # sort along the last axis
array([[1, 4],
       [1, 3]])
>>> np.sort(a, axis=None)     # sort the flattened array
array([1, 1, 3, 4])
>>> np.sort(a, axis=0)        # sort along the first axis
array([[1, 1],
       [3, 4]])

Use the order keyword to specify a field to use when sorting a structured array:

>>> dtype = [('name', 'S10'), ('height', float), ('age', int)]
>>> values = [('Arthur', 1.8, 41), ('Lancelot', 1.9, 38),
...           ('Galahad', 1.7, 38)]
>>> a = np.array(values, dtype=dtype)       # create a structured array
>>> np.sort(a, order='height')                        
array([('Galahad', 1.7, 38), ('Arthur', 1.8, 41),
       ('Lancelot', 1.8999999999999999, 38)],
      dtype=[('name', '|S10'), ('height', '<f8'), ('age', '<i4')])

Sort by age, then height if ages are equal:

>>> np.sort(a, order=['age', 'height'])               
array([('Galahad', 1.7, 38), ('Lancelot', 1.8999999999999999, 38),
       ('Arthur', 1.8, 41)],
      dtype=[('name', '|S10'), ('height', '<f8'), ('age', '<i4')])

Benchmarking

pnumpy.benchmark.benchmark(ctypes=[<class 'numpy.bool_'>, <class 'numpy.int8'>, <class 'numpy.int16'>, <class 'numpy.int32'>, <class 'numpy.int64'>, <class 'numpy.float32'>, <class 'numpy.float64'>], recycle=True, atop=True, thread=True, sizes=[1000000])

Performs a simple benchmark of the ratio of normal numpy (no threading) vs parallel numpy (threaded). The output is formatted to be copied and pasted in a csv file. A result above 1.0 indicates an improvement, below 1.0 indicates worse peformance.

Parameters

  • ctypes (list of numpy dtypes to test, for example [np.int32, npfloat32]) –

  • recycle (True or False) –

  • atop (True or False.  Whether or not the atop engine is used in benchmarking.) –

  • thread (True or False.) –

  • sizes (list of array sizes to test, for example [100_000, 1_000_000]) –

Returns

Return type

output text formatted for a .csv file

Examples

pn.benchmark() pn.benchmark(thread=False) pn.benchmark(sizes=[2**16]) pn.benchmark(ctypes=[np.float32, np.float64])