NDArrays tutorial

The package is inspired by the high-level interface for multi-dimensional arrays provided provided by Matlab, Julia or Python's NumPy package; therefore, this tutorial presents the main features of NDArrays with parallel examples written in Matlab, Julia and Python.

If you are looking for the list of functions, you can find the documentation here: https://hakkelt.github.io/NDArrays/index.html
The source code of the package is available here: https://github.com/hakkelt/NDArrays

Project Structure

The NDArrays library is divided into three namespaces:

Use cases

  1. You want to work with real and complex multi-dimensional arrays in a way similar to Numpy, Matlab or Julia. → You need the reference implementation (Basic NDArrays).
  2. You need a conversion tool between n-dimensional array implementations of different Java packages → You might want to implement a wrapper class for each of them that implements NDArray interface.

In the following, I present the main features of NDArray and ComplexNDArray interfaces through the reference implementation. This tutorial assumes that the reader is familiar with anonymous functions and streams in Java.

Initialization

Construction and initialization

New array filled with zeros and with a constant value

Fill array with position dependent values based on linear indices

Linear index: Index of elements imagining that the array is reshaped into a vector.
Column/row-major indexing:

Note that NDArray, Matlab and Julia uses column-major ordering, while NumPy uses row-major ordering.
Also note that while Matlab and Julia start indexing from 1, NDArray and NumPy uses zero-based indexing.

Fill array with position dependent values based on Cartesian indices

Cartesian index: fancy name for "usual" indexing with separate coordinates for each dimensions.

Other Java-specific ways to initialize NDArrays

Initialize with primitive array

Note that for technical reason, this initialization is done by factory methods (.of() static methods, so new keyword shall not be used)!

Copy constructor

Usually, copy constructor is useful to convert between different types of NDArray.

Stream collector

Sometimes, NDArray collectors might be a little tricky. For instance, in the following code BasicDoubleNDArray.getCollector(4, 5) is capable of receiving only Double values, so first we need to convert int to double (mapToDouble, and (double) casting), and the double to Double (hence we need boxed()). Usually there are more convenient alternatives, but it can be handy when we want to both transform the data and change its type (like squaring integers (transformation) and casting them to double).

Basic operations

Get and set values

Please, be aware that Matlab and Julia start indexing at 1, while NDArrays and NumPy are using zero-based indexing!

Read matrix meta data

Iterate over elements

Piecewise mapping/transformation

Mapping creates a new array and leaves the original intact, while transformation modifies the original array.

Keep in mind that both mapping and transformation runs in parallel, so the anonymous functions given as parameters MUST NOT change variables outside of the function body. In exchange, these operations will be much faster then conventional solutions (for loop + fetch + transform + set) both because they are parallelized and because no boundary checks are needed when fetching/setting elements.

Views

Views are basically wrappers around the original data structure. They are lazy: they don't perform any operations (i.e., don't copy data), instead they reference the specified region its parent array. Therefore, all modifications in the parent array are reflected in the view, and vice versa.

Slice

They reference their parent array giving read-write access to a specific multi-dimensional slice of the parent array.

Basic usage

Copying the slice

To avoid the modification of the original array, we need to create explicitly a copy of the slice.

In Matlab, slicing creates a copy automatically (except when it appears on the left hand side of an assignment).

With NumPy, we also need to create a copy explicitly.

Like Matlab, Julia creates a copy by default when slicing is performed (except when it appears on the left hand side of an assignment):

Special ranges

Ranges starting from beginning, going until end, or selecting entire range.

Negative indexing

  1. Define start or end of range based on the distance from the last index.
  2. Reverse range.

Logical indexing, masking

Assign sub-matrix

Reshaping

Singleton dimensions

Singleton dimension = dimension of size 1

Transposition

More generally: Change the order of axes/dimensions. In 2D, there are only two permutations of the dimensions: normal order, and the transposed order.

Arithmetics

Piecewise arithmetic operations

Accumulation of elements

Norms

Utility functions

This functionality is fully Java-related; therefore, only Java examples are provided.

Let's create a random matrix filled with values between 0 and 1! We will use this array in the following examples.

Stream indices

Alternative for multiple nested loops, and it also makes the parallelization of iterations easy.

Stream Cartesian indices

Example: Let's print the center coordinates of 3×3 blocks in which the average of values is above 0.85

Conventional Java solution:

With NDArray's streaming interface:

Stream Cartesian indices independently from any NDArray

Example: Let's print the center coordinates of 3×3×20 block in which the average of values is above 0.55

These blocks stretch over the entire range along the third dimension, so there is no need to iterate over the third dimension!

Conventional Java solution:

With NDArray's streaming interface:

Stream linear indices

Example: Let's calculate the squared sum of every 10th element.

Slice operations

There are convenience functions to iterate slices over some dimensions and apply functions to them, or even reduce over them.

toArray

We can convert NDArrays into native arrays of both objects and primitive values.

Concatenation

It is possible to concatenate arrays (even different data type!) along a specified dimension:

This is actually a shorthand for the following expressions:

Check for equality

The equals method compares NDArrays elementwise, and returns true only when all elements are pairwise equal.

When two NDArrays with different data types are compared, it first attemps to cast elements on the right hand side to match the data type on the left, and it performs the comparison afterwards. This means that equals() is not always symmetric (a.equals(b) ↔ b.equals(a))!

Similar

Sometimes, we have an array, and we need another one that has same shape and data type, but we don't need to copy all the values. similar() creates such an array, and leaves default values in each elements (which is 0 for basic NDArrays, but might be uninitialized for other implementations).

Serialize and deserialize

We can easily save an NDArray to a file, and read it back. The file stores all the necessary meta data such that the shape of the array and the data type. We can, however, read saved arrays into NDArrays of any data type, and NDArrays performs the necessary type casting.

There is no constraint on the file extension, .nda is only my personal preference.

Complex NDArrays

We use Complex class from Apache math3 library to represent complex values.

Complex NDArrays are no different from simple NDArrays; in fact, ComplexNDArray is a sub-interface of NDArray. Thus, we present only features unique to complex NDArrays. Also, we see no advantage of presenting the parallel solutions in the other languages.

Construction and initialization

New array filled with zeros and with a constant value

Fill array with position dependent values

It is pretty much the same as it was for real NDArrays but with complex values in this case.

Initialize with primitive array(s)

Note that for technical reason, this initialization is done by factory methods (.of() static methods, so new keyword shall not be used)!

Copy constructor

Usually, copy constructor is useful to convert between different types of NDArray.

Create from magnitude and phase arrays

Stream collector

This solution is not recommended as we need explicit type casting (Java can only infer NDArray<Complex> but cannot specialize further).

Get and set

Set range

Assign complex sub-matrix:

Assign real sub-matrix:

Assign real and imaginary parts:

Complex arithmetics

All arithmetical functions (add, addInplace, multiply, multiplyInplace works, of course, for complex NDArrays).
Additionally there are 4 complex-related functions:

Precision of data type

Currently only single and double precision is supported (Float and Double). dtype2() can tell which precision is used by the given array.

Slice operations

Functions applyOnSlices, mapOnSlices and reduceSlices work fine on ComplexNDArray, there is however a little problem with them when used with ComplexNDArrays: the type of the first parameter (slice) is NDArray<Complex> instead of ComplexNDArray<...> meaning that you cannot use complex-specific functions on slices like abs(), argument(), real(), and imaginary(). Therefore, ComplexNDArrays have three additional functions: