We are going to implement a digit (1 to 5) recognition system using MATLAB through Deep Learning. The Deep Neural Network we are going to train has 25 input nodes, 20 nodes in each hidden layers and 5 output nodes. You may ask why we are taking such kind of architecture. Well the input nodes depend the training data. We will train the network for digits which are consisted of 25 pixels. And in the output there are 5 nodes, because we have to classify 5 digits. If there were 10 digits, then we had to take 10 output nodes. The following figure illustrates the architecture of Deep Neural Network we are about to train –
Here W1, W2, W3, W4 are weight matrices of hidden layers. For hidden layers, we have used ReLU activation function and for output layer, we have used Softmax activation function.
To implement the system in Matlab we have to create 3 functions and 2 scripts. The name of the functions are –
Functions
- ReLU – It is the activation function of hidden layer.
- Softmax – It is the activation function of output layer.
- DeepLearning – This function is to update the weight matrices and train the network.
Script
- TrainingNetwork – This script calls the ‘DeepLearning’ function to train the network
- TestDeepLearning – This script passes the digits to test the network
The ‘DeepLearning’ function uses ReLU and Softmax function. The task of this function is to take the weight matrices (W1, W2, W3, W4), input and correct output when called by the script ‘TrainingNetwork’. It adjusts the weight matrices every time it is called. Inside the ‘TrainingNetwork’ script we have called ‘DeepLearning’ function for 10,000 times. That means we adjusted the weight matrices for 10,000 times. The ‘TrainingNetwork’ generates the trained Deep Neural Network. Finally the ‘TestDeepLearning’ script uses the network trained and saved by ‘TrainingNetwork’ script to identify the digits.
Here are the functions and scripts. You are permitted to copy, modify or distribute it as long as you put the reference of the author.
ReLU Function
function y = ReLU(x) y = max(0,x); end
Softmax Function
function y = Softmax(x) ex = exp(x); y = ex/sum(ex); end
Deep Learning Function
function [w1, w2, w3, w4] = DeepLearning(w1, w2, w3, w4, input_Image, correct_Output) alpha = 0.01; %to control the learning rate N = 5; for k = 1:N reshaped_input_Image = reshape(input_Image(:,:,k), 25, 1); input_of_hidden_layer1 = w1*reshaped_input_Image; output_of_hidden_layer1 = ReLU(input_of_hidden_layer1); input_of_hidden_layer2 = w2* output_of_hidden_layer1; output_of_hidden_layer2 = ReLU(input_of_hidden_layer2); input_of_hidden_layer3 = w3* output_of_hidden_layer2; output_of_hidden_layer3 = ReLU(input_of_hidden_layer3); input_of_output_node = w4* output_of_hidden_layer3; final_output = Softmax(input_of_output_node); correct_Output_transpose = correct_Output(k, :)'; error = correct_Output_transpose - final_output; delta = error; error_of_hidden_layer3 = w4'*delta; delta3 = (input_of_hidden_layer3>0).*error_of_hidden_layer3; error_of_hidden_layer2 = w3'*delta3; delta2 = (input_of_hidden_layer2>0).*error_of_hidden_layer2; error_of_hidden_layer1 = w2'*delta2; delta1 = (input_of_hidden_layer1>0).*error_of_hidden_layer1; adjustment_of_w4 = alpha*delta*output_of_hidden_layer3'; adjustment_of_w3 = alpha*delta3*output_of_hidden_layer2'; adjustment_of_w2 = alpha*delta2*output_of_hidden_layer1'; adjustment_of_w1 = alpha*delta1*reshaped_input_Image'; w1 = w1+ adjustment_of_w1; w2 = w2+ adjustment_of_w2; w3 = w3+ adjustment_of_w3; w4 = w4+ adjustment_of_w4; end end
TrainingNetwork Script
input_Image = zeros(5,5,5);
input_Image(:,:,1) = [1 0 0 1 1;
1 1 0 1 1;
1 1 0 1 1;
1 1 0 1 1;
1 0 0 0 1;
];
input_Image(:,:,2) = [0 0 0 0 1;
1 1 1 1 0;
1 0 0 0 1;
0 1 1 1 1;
0 0 0 0 0;
];
input_Image(:,:,3) = [0 0 0 0 1;
1 1 1 1 0;
1 0 0 0 1;
1 1 1 1 0;
0 0 0 0 0;
];
input_Image(:,:,4) = [1 1 1 0 1;
1 1 0 0 1;
1 0 1 0 1;
0 0 0 0 0;
1 1 1 0 1;
];
input_Image(:,:,5) = [0 0 0 0 0;
0 1 1 1 1;
0 0 0 0 1;
1 1 1 1 0;
0 0 0 0 1;
];
correct_Output = [1 0 0 0 0;
0 1 0 0 0;
0 0 1 0 0;
0 0 0 1 0;
0 0 0 0 1;
];
w1 = 2*rand(20,25)-1;
w2 = 2*rand(20,20)-1;
w3 = 2*rand(20,20)-1;
w4 = 2*rand(5,20)-1;
for epoch = 1:10000
[w1, w2, w3, w4] = DeepLearning(w1, w2, w3, w4, input_Image, correct_Output);
end
save('DeepNeuralNetwork.mat')
TestDeepLearning Script
load('DeepNeuralNetwork.mat');
input_Image = [0 0 0 0 1;
1 1 1 1 0;
1 0 0 0 1;
0 1 1 1 1;
0 0 0 0 0;
];
input_Image = reshape(input_Image, 25, 1);
input_of_hidden_layer1 = w1*input_Image;
output_of_hidden_layer1 = ReLU(input_of_hidden_layer1);
input_of_hidden_layer2 = w2*output_of_hidden_layer1;
output_of_hidden_layer2 = ReLU(input_of_hidden_layer2);
input_of_hidden_layer3 = w3*output_of_hidden_layer2;
output_of_hidden_layer3 = ReLU(input_of_hidden_layer3);
input_of_output_node = w4*output_of_hidden_layer3;
final_output = Softmax(input_of_output_node)
I hope this article will help learners to learn Deep learning easily. When we study the theoretical concepts only, it becomes difficult to understand properly. However, if we study the concept and apply it, then we understand it properly. In this article, I tried to explain Deep Learning using Matlab. And also showed how to train a Deep Neural Network (DNN) using Matlab. So, I believe I have covered both theoretical and practical aspects. My expectation is -anyone, with prior knowledge or without prior knowledge of Deep Learning will be able to follow this article and train networks to do whatever they want.
