Introduction

Are you looking for an efficient and modern framework to create a deep learning model and also which has a pythonic nature? Then you might try considering PyTorch.

PyTorch is an open-source machine-learning framework based on the Torch library. It is primarily developed by Facebook's AI Research lab and used for applications such as computer vision, natural language processing, etc.

Let's look at 5 interesting PyTorch functions.

Torch.tensor_split
torch.nan_to_num
torch.nn.functional.pad
torch.from_numpy
torch.narrow

Before we begin, let's import NumPy and PyTorch

# Import torch and other required modules
import torch
import numpy as np

torch.tensor_split

Parameters:

input (Tensor)
indices_or_sections (Tensor, int or list or tuple of python)
dim (dimension along which to split the tensor)

This function splits a tensor into multiple sub-tensors

This function can be used when we want to split a tensor into unequal subtensors and when we only need a part of the tensor

tensor1=torch.arange(10)
tensor1
>>> tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

In this example, we have created a 1D tensor in the range of 10.

#example 1
torch.tensor_split(tensor1,3,dim=0)
>>> (tensor([0, 1, 2, 3]), tensor([4, 5, 6]), tensor([7, 8, 9]))

The tensor_split function splits the tensor into 3 parts. The function first calculates the input.size(dim)/n. if the tensor is not divisible by n, the sizes of the first int(input.size(dim) % n) sections will have size int(input.size(dim) / n) + 1, and the rest will have a size int(input.size(dim) / n). So here tensor1.size(0)=10 is not divisible by n=3. Therefore the first section will have a size of (10//3)+1=4 and the rest will have a size of 10//3=3.

#example 2
tensor2=torch.randint(1,16,(3,5))
tensor2
>>> tensor([[ 7, 12, 14, 13,  5],
        [ 5,  9,  8,  4,  4],
        [11,  9, 10, 10,  5]])

In this example, we have given a list of indices.

torch.tensor_split(tensor2,[1,2],dim=1)
>>> (tensor([[ 7],
         [ 5],
         [11]]), tensor([[12],
         [ 9],
         [ 9]]), tensor([[14, 13,  5],
         [ 8,  4,  4],
         [10, 10,  5]]))

In this example, we have given a list of indices that indicate the location of the split. The input tensor gets split in the following way:

tensor[: element1],tensor[element1:element2],tensor[element2:].

Here since the dim=1 ie along the columns, therefore indexing would be along the columns.

# Example 3 - breaking
tensor3=torch.arange(8)
torch.tensor_split(tensor3,3,dim=1)
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-19-f88e42abd410> in <module>()
      1 # Example 3 - breaking
      2 tensor3=torch.arange(8)
----> 3 torch.tensor_split(tensor3,3,dim=1)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Summary

This function breaks when specified a 1 dimensional tensor while also specifying a dimension to split along columns because it is 1D tensor.

torch.nan_to_num

Parameters:

input (Tensor)
nan (number, optional) - the values to replace Nan s with
posinf (number, optional) - the values to replace positive infinity values with
neginf (number, optional) - the values to replace negative infinity values with

This function replaces Nan, positive infinty, negative infinty values in input with the values specified in the parameters.

NaN stands for Not A Number and is a common missing data interpretation.It was introduced by the IEEE Standard for Binary Floating-Point for Arithmetic (IEEE 754) way before python was invented. NaNs can bring lot of headaches if not handled properly.
tensor4=torch.tensor([float('nan'),float('inf'),-float('inf'),3.14])
tensor4
>>> tensor([   nan,    inf,   -inf, 3.1400])

Here we are creating a tensor with nan,inf as its values.

# Example 1 
torch.nan_to_num(tensor4)
>>> tensor([ 0.0000e+00,  3.4028e+38, -3.4028e+38,  3.1400e+00])

Here the function replaced all the nan values with zero in float format, inf with a very large positive number, -inf with a very small negative number.

# Example 2
torch.nan_to_num(tensor4,nan=2.0,posinf=1.0)
>>> tensor([ 2.0000e+00,  1.0000e+00, -3.4028e+38,  3.1400e+00])

Here we have replaced the nan values with 2.0 and posinf with 1.0. This can be used when we want an arbitrary value and not a very small/larger floating point value.

Breaking case

This function only works when PyTorch version 1.8+. Any version below this will not be able to identify it and will throw an error.

AttributeError: module 'torch' has not attribute 'nan_to_num'

Summary

PyTorch's nan_to_num function is quite useful when we have to deal with NaN values effortlessly

torch.nn.functional.pad

Parameters:

input (Tensor)
pad (tuple)
mode - 'constant','reflect','replicate' or 'circular'
value (fill value for 'constant' padding)

torch.nn.functional.pad is a very useful function to pad tensors, ie adding zeros around the border of images to convert them to a shape that is amenable to the convolution operation without throwing away any pixel information.

# Example 1
t1d = torch.arange(1,10)
t1d
p1d=(2,2)
t1d_out=torch.nn.functional.pad(t1d,p1d)
print(t1d)
print()
print(t1d_out)
print()
print(t1d_out.size())
>>> tensor([1, 2, 3, 4, 5, 6, 7, 8, 9])

>>> tensor([0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 0])

>>> torch.Size([13])

to pad 1 dimension (padding_left,padding_right)
to pad 2 dimensions (padding_left,padding_right, padding_top,padding_bottom))
to pad 3 dimensions (padding_left,padding_right, padding_top,padding_bottom,padding_front,padding_back)

Here we have a tensor of 1 dimension. Therefore, the padding will occur 2 layers from (left, right).

# Example 2
t2d = torch.ones(3,3)
p2d = (2,2,2, 2)
out = torch.nn.functional.pad(t2d, p2d)
print(t2d) #print the original tensor
print()
print(out) #print the padded tensor
print()
print(out.size()) #print the shape of the tensor
>>> tensor([[1., 1., 1.],
        [1., 1., 1.],
        [1., 1., 1.]])

>>> tensor([[0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 1., 1., 1., 0., 0.],
        [0., 0., 1., 1., 1., 0., 0.],
        [0., 0., 1., 1., 1., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0.]])

>>> torch.Size([7, 7])

Here we have a tensor of 2 dimensions. Therefore, the padding will occur 2 layers from (left, right, top, bottom).

# Example 3 - breaking
tbr=torch.arange(1,10)
pad=(2,2,2)
tbr_out=torch.nn.functional.pad(tbr,pad)
tbr_out
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-92-4220d13680e0> in <module>()
      2 tbr=torch.arange(1,10)
      3 pad=(2,2,2)
----> 4 tbr_out=torch.nn.functional.pad(tbr,pad)
      5 tbr_out

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in _pad(input, pad, mode, value)
   4169     if has_torch_function_unary(input):
   4170         return handle_torch_function(_pad, (input,), input, pad, mode=mode, value=value)
-> 4171     assert len(pad) % 2 == 0, "Padding length must be divisible by 2"
   4172     assert len(pad) // 2 <= input.dim(), "Padding length too large"
   4173     if mode == "constant":

AssertionError: Padding length must be divisible by 2

The pad tuple must contain an even number of elements. Here since we made a tuple of elements, PyTorch threw an error that it can't have three elements.

Summary

This function is quite helpful in convolutions to add border to images without losing important pixel information.

torch.from_numpy

Parameters:

a NumPy array

This function can be used to convert a NumPy array to Pytorch tensor. The returned tensor and the NumPy array share the same memory so any changes we make to the tensor will also be reflected in the NumPy array.

# Example 1
arr=np.random.randint(0,16,(3,3))
tnp=torch.from_numpy(arr)
print(arr)
print()
print(tnp)
>>> [[ 0  0 12]
 [ 2  4  1]
 [14  3 13]]

>>> tensor([[ 0,  0, 12],
        [ 2,  4,  1],
        [14,  3, 13]])

In this example, we created a NumPy array with random integers of the shape 3x3. The function from_numpy converted the array to a tensor.

# Example 2
temp_rain_humid=np.array([[73,67,43],[91,88,64],[87,134,58]])
t2np=torch.from_numpy(temp_rain_humid)
t2np
>>> tensor([[ 73,  67,  43],
        [ 91,  88,  64],
        [ 87, 134,  58]])

In this example, we have created a NumPy array of temperature, rainfall, and humidity of various regions with dummy values. The from_numpy function converted the NumPy array to a PyTorch tensor.

# Example 3 - breaking
arr3=np.array(['1','2','3','4'])
t2np3=torch.from_numpy(arr3)
t2np3
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-110-2bd2c13d6da5> in <module>()
      1 # Example 3 - breaking
      2 arr3=np.array(['1','2','3','4'])
----> 3 t2np3=torch.from_numpy(arr3)
      4 t2np3

TypeError: can't convert np.ndarray of type numpy.str_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

PyTorch currently doesn't accept converting a NumPy array of strings to a tensor and hence the error.

Summary

This function is extremely helpful with people using numpy as their primary array calculations. It can easily convert numpy array to a tensor.

torch.narrow

Parameters:

input (Tensor)
dim (int) - dimension along which to narrow
start (int) - the starting dimension
length (int) - the distance to the ending dimension

The torch.narrow function is used to narrow down a tensor or simply we can say it is used to slice a tensor. It is a good alternative to the traditional slicing method.

# Example 1 - working
tensornt = torch.randint(0,16,(3,3))
print(tensornt)
print()
print(torch.narrow(tensornt,0,0,2))
>>> tensor([[ 6,  4,  8],
        [ 3,  3, 14],
        [ 2,  8, 11]])

>>> tensor([[ 6,  4,  8],
        [ 3,  3, 14]])

In this example, we have sliced the tensor by giving the parameters 0 dimension (rows) start index as 0 and length as 2. The narrow function uses the following syntax to slice a tensor. It starts from start to start+length.Therefore the tensor will be sliced along the row starting from the 0th index and ending at 0+2 ie 1st index.

# Example 2 - working
tensornt2=torch.arange(8).reshape(2,4)
print(tensornt2)
print()
print(torch.narrow(tensornt2,1,1,2))
>>> tensor([[0, 1, 2, 3],
        [4, 5, 6, 7]])

>>> tensor([[1, 2],
        [5, 6]])

In this example, we have sliced the tensor by giving the parameters 1 dimension (rows) start index as 1 and length as 2.Therefore the tensor will be sliced along the row starting from 1st index and ending at 1+2 ie 2nd index.

# Example 3 - breaking
terr=torch.arange(9).reshape(3,3)
torch.narrow(terr,0,1,3)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-131-256a27618082> in <module>()
      1 # Example 3 - breaking
      2 terr=torch.arange(9).reshape(3,3)
----> 3 torch.narrow(terr,0,1,3)

RuntimeError: start (1) + length (3) exceeds dimension size (3)

In this example, the start+end exceeds the maximum index along the row axis ie 3. Hence it throws an error.

Summary

The narrow function can be used in place of traditional slicing operations when the tensors are large and which need a bit of visualization. This function also performs better in some scenarios.

Conclusion

PyTorch has a lot of useful functions. This article only covers a few interesting functions that can be performed on tensors. Deep learning is all about experimentation. To know more about the functions available in PyTorch, the documentation is a good start.

Reference Links

Official documentation for tensor operations: https://pytorch.org/docs/stable/torch.html

Top PyTorch functions you didn't know about

Table of contents

Introduction

torch.tensor_split

Summary

torch.nan_to_num

Breaking case

Summary

torch.nn.functional.pad

Summary

torch.from_numpy

Summary

torch.narrow

Summary

Conclusion

Reference Links