We Helped With This MATLAB Programming Homework: Have A Similar One?

CSCI 410 Pattern Recognition

Assignment Four

by Denton Bobeldyk

1.       Using Matlab, create a multi-layer perceptron with 3 layers: input layer, hidden layer, output layer (using a sigmoid function).

2.       Define the learning rate and total iterations learningRate = 0.5; totalIterations = 500;

3.       Define the size of the input layer and the hidden layer:

inputLayerNumber = 2; hiddenLayerNumber = 2;

4.       Define the input and hidden layer:

inputLayer = zeros(inputLayerNumber, 1);

hiddenLayer = zeros(hiddenLayerNumber, 1);

5.       Add the bias to the input and hidden layer:

inputLayerWithBias = zeros(inputLayerNumber + 1, 1); hiddenLayerWithBias = zeros(hiddenLayerNumber + 1, 1);

6.       Define the output layer:

outputLayer = 0;

7.       Randomly assign the weights to the input and hidden layer: inputLayerWeights = rand( (inputLayerNumber + 1) ,hiddenLayerNumber) - .5 ; hiddenLayerWeights = rand( (hiddenLayerNumber + 1), 1) - .5;

8.       Define the input data:

inputLayer = [0 0; 0 1; 1 0; 1 1];

9.       Define the target output for the input layer: ANDtargetOutput = [0; 0; 0; 1]; targetOutput = ANDtargetOutput;

10.   Define the variable `m’ as the number of samples:

m = size(targetOutput, 1);

11.   Create a for loop, that will step through each of the samples one at a time (Note: this is known as online learning)

for iter=1:totalIterations

     for i = 1:m;

        hiddenLayerActivation = inputLayerWithBias(i, :) * inputLayerWeights;         hiddenLayer = sigmoid(hiddenLayerActivation);

        %Add the bias to the hiddenLayer         hiddenLayerWithBias = [1, hiddenLayer];

        outputLayer = sigmoid(hiddenLayerWithBias * hiddenLayerWeights);

%Calculate the error:

deltaOutput = targetOutput(i) - outputLayer;

        deltaHidden(1) = (deltaOutput * hiddenLayerWeights(1)) .* ((hiddenLayerWithBias(1) * (1.0 - hiddenLayerWithBias(1))));         deltaHidden(2) = (deltaOutput * hiddenLayerWeights(2)) .* ((hiddenLayerWithBias(2) * (1.0 - hiddenLayerWithBias(2))));         deltaHidden(3) = (deltaOutput * hiddenLayerWeights(3)) .*

((hiddenLayerWithBias(3) * (1.0 - hiddenLayerWithBias(3)))); 

        % Fixed Step Gradient Descent - Update the weights

        hiddenLayerWeights(1) = hiddenLayerWeights(1) + (learningRate *

(deltaOutput * hiddenLayerWithBias(1)));

        hiddenLayerWeights(2) = hiddenLayerWeights(2) + (learningRate *

(deltaOutput * hiddenLayerWithBias(2)));

        hiddenLayerWeights(3) = hiddenLayerWeights(3) + (learningRate *

(deltaOutput * hiddenLayerWithBias(3)));

        %update each weight according to the part that they played         inputLayerWeights(1,1) = inputLayerWeights(1,1) + (learningRate * deltaHidden(2) * inputLayerWithBias(i, 1));

        inputLayerWeights(1,2) = inputLayerWeights(1,2) + (learningRate * deltaHidden(3) * inputLayerWithBias(i, 1));

        inputLayerWeights(2,1) = inputLayerWeights(2,1) + (learningRate * deltaHidden(2) * inputLayerWithBias(i, 2));

        inputLayerWeights(2,2) = inputLayerWeights(2,2) + (learningRate * deltaHidden(3) * inputLayerWithBias(i, 2));

        inputLayerWeights(3,1) = inputLayerWeights(3,1) + (learningRate * deltaHidden(2) * inputLayerWithBias(i, 3));

        inputLayerWeights(3,2) = inputLayerWeights(3,2) + (learningRate * deltaHidden(3) * inputLayerWithBias(i, 3));  

            end

     end

12.   Create the sigmoid function:

function a = sigmoid(z)

a = 1.0 ./ (1.0 + exp(-z));

  end

13.   Create the cost function:

% This function will only work for NN with just one output (k = 1) function [averageCost] = costFunction(inputLayerWithBias, inputLayerWeights, hiddenLayerWeights, targetOutput)

%Sum of square errors cost function m = 4;

hiddenLayer = sigmoid(inputLayerWithBias * inputLayerWeights);

hiddenLayerWithBias = [ones(m,1) hiddenLayer];

outputLayer = sigmoid(hiddenLayerWithBias * hiddenLayerWeights);

% Step through all of the samples and calculate the cost at each one for i=1:m

    cost(i) = (1/2) * ((outputLayer(i) - targetOutput(i)) .^ 2); end

%Sum up all of the individual costs totalCost = sum(cost);

%average them out

averageCost = totalCost * (1/m);

    end

14. Create a function that will summarize the output of the 4 samples:

 function outputSummary(inputLayerWithBias, inputLayerWeights, hiddenLayerWeights, targetOutput, totalIterations)

cost = costFunction(inputLayerWithBias, inputLayerWeights, hiddenLayerWeights, targetOutput);

hiddenLayer = sigmoid(inputLayerWithBias * inputLayerWeights);

%we have multiple samples, so we need to add the bias to each of them hiddenLayerWithBias = [ones(size(targetOutput,1),1) hiddenLayer];

actualOutput = sigmoid(hiddenLayerWithBias * hiddenLayerWeights);

fprintf('========================================= '); fprintf('Output Summary (after %d iterations): ', totalIterations); fprintf('Total Cost: [%f] ', cost);

    for i=1:length(actualOutput)     if(actualOutput(i) > 0.5)         thresholdedValue = 1;     else

        thresholdedValue = 0;     end     

    if(thresholdedValue == targetOutput(i))

        fprintf('Sample[%d]: Target = [%f] Thresholded Value = [%f] Actual= [%f] ', i, targetOutput(i), thresholdedValue, actualOutput(i));     else  % else print the error in red

        fprintf(2,'Sample[%d]: Target = [%f] Thresholded Value = [%f] Actual= [%f] ', i, targetOutput(i), thresholdedValue, actualOutput(i));     end          end   

fprintf('=========================================');

  end

15.   Attempt to learn the following target outputs:

ANDtargetOutput = [0; 0; 0; 1];

ORtargetOutput = [0; 1; 1; 1];

NANDtargetOutput = [1; 1; 1; 0];

NORtargetOutput = [1; 0; 0; 0];

XORtargetOutput = [0; 1; 1; 0];

16.   Which of the above target outputs does it have the hardest time learning and why?

Turn-in:

1.       A single matlab script that performs the above tasks.

2.       The answer to question 16 output using ‘fprintf’ commands in the above script.

3.       A word document containing the copy and pasted output of the script execution. Please make sure there are no line wrappings (decrease the font if necessary).

References:

Andrew NG, Machine Learning course from Coursera

Bishop, Christopher M. Neural networks for pattern recognition. Oxford university press, 1995.