Activation Functions For Deep Learning Using Tensorflow

Table of Contents

• Sigmoid Function
• Tangent Hyperbolic
• RELU Function

This article implements common activation functions for deep learning using Tensorflow.

Tensorflow is a free and open-source library for machine learning and related taks.

Tensorflow is a symbolic math library, and is also used for machine learning applications such as neural networks & deep learning.

In this article we are going to implement sigmoid, tangent hyperbolic and RELU function using Tensorflow.

Rectified Linear Unit (RELU) Function

RELU has become the default activation function for different neural networks application because of its simplicity and power.

Mathematical function for RELU is: $$ f(x) = max(0,x) $$

Derivative for RELU is: $$ f(x) = \begin{cases} \text{1, x>0} \\ \text{0, otherwise} \end{cases} $$

Souce Code: RELU

import tensorflow as tf
import numpy as np
from matplotlib import pyplot as plt

# Generating data to plot
x_data = np.linspace(-6,6,100)
y_data = tf.nn.relu(x_data)

# Plotting
plt.plot(x_data, y_data)
plt.title('RELU')
plt.grid()
plt.show()

Output RELU

Sigmoid Function

Mathematical function for sigmoid is: $$ f(x) = \sigma(x) = \frac{1}{1+e^{-x}} $$

Derivative of sigmoid function is: $$ f'(x) = \sigma(x) ( 1 - \sigma(x) ) $$

Source Code: Sigmoidal Function

import tensorflow as tf
import numpy as np
from matplotlib import pyplot as plt

# Generating data to plot
x_data = np.linspace(-6,6,100)
y_data = tf.math.sigmoid(x_data)

# Plotting
plt.plot(x_data, y_data)
plt.title('Sigmoid Function')
plt.grid()
plt.show()

Sigmoid Output

Tangent Hyperbolic Function

Another useful activation function is tangent hyperbolic.

Formula for Tangent Hyperbolic: $$ f(x) = \frac{e^{x} - e^{-x}}{e^{x} + e^{-x}} $$

Derivative for Tangent Hyperbolic is: $$ f'(x) = ( 1 - g(x)^{2} ) $$

Source Code: Tangent Hyperbolic

import tensorflow as tf
import numpy as np
from matplotlib import pyplot as plt

# Generating data to plot
x_data = np.linspace(-6,6,100)
y_data = tf.math.tanh(x_data)

# Plotting
plt.plot(x_data, y_data)
plt.title('RELU')
plt.grid()
plt.show()

Output