Tensor

A tensor is a mathematical object that can be thought of as a generalized type of vector. It has more dimensions than a vector, which allows it to describe higher-order relationships between elements in a data set. Tensors are used in many fields of applied mathematics, including machine learning and deep learning.

In neural networks, tensors are used to represent the data that is fed into a system, as well as the weights and biases that are adjusted during the learning process. Tensor operations such as dot products or matrix multiplications can be used to build up more complex neural network architectures.

PyTorch and TensorFlow are two popular open-source libraries for building machine learning models with tensors. TensorFlow is based on dataflow graphs and PyTorch is based on dynamic computation graphs.

The “shape” of a tensor is represents the number of dimensions, their sizes, and the number of elements in each dimension. The term “rank” specifically refers to the number of dimensions of a tensor. Thus a rank 0 tensor is a single scalar number, a rank 1 tensor is a vector, a rank 2 tensor is a matrix, and so forth.

Cerebras Software Platform (CSoft) automatically supports Variable Tensor Shape (VTS) computations. The unique technology accelerates training of transformer natural language processing models by performing computation on data samples with heterogeneous shapes efficiently, rather than wasting time on padded elements needed by traditional compute architectures.


Further reading

Tensor Shape: Increasing Model Throughput We write machine learning algorithms to fit the data, not pad the data to suit hardware limitations.