Multi Layer Neural Networks

8. Multi Layer Neural Networks

Learning outcomes

• Construct a multi-layer neural network that classifies a given dataset in 2D, overcoming the limitation on learning separability.
• Substitute the activation function of the perceptron, with a function amenable to gradient descent.

Perceptron Limitations

• Not being able to classify XOR with a straight line– linear separability is the limitation

Multi-layer Perception

• What we are gonna do is we put neurons in layers, and even just with one more layer we can classify whatever we want.
• 
• So far we’ve been using one layer – perceptron has been a single layer. One per class. One neuron classifies to a yes or no. now we have one more layer of the same thing, each neuron also has a bias input, -1.

MLP and XOR

• We can have more than one line now! And then combine somehow

• By hand, we construct a neural network that does the justification you want and you can see the solution We do this by hand

• Example
  • 
  • Neuron 1: We chose a line and say we want the neuron to output a 1 below the line.
  • Neuron 2: There’s another line that outputs a 1 above the line
  • We want an intersection of this two new layer gives this!
• We can take 2 neurons, each one makes one mistake (bc now this other neuron is misclassifying this point), and if you put them together you can compute the intersection of the two halfplane, and that’s where you want ur MLP to output 1!!!
  • 2 neurons + 1 more layer you can achieve this
  • Outputting 1 = classified correctly
• <-2,-1> and <1,1> is a vector of weights of the two neurons.
• What’s special about the two-half planes?
  • Each one has the side where it outputs 1 that contains our points with 1s.
  • So the first neuron outputs 1 on this side and it contains both the 1s (points), the other neuron outputs 1s on the other side and contains both the 1s. This way we find intersection to find exactly the 1s.

MLP and XOR -

• 
• So, 2 neurons each one has its weights. And this is the first layer of MLP. Now we do it for two!
• Two neurons are the first layer of the neural network.
• The output of the two neurons will become an input of a new layer

MLP and XOR -

• We evaluated two neurons on all four points
• Points in dataset – 00, 01, 10, 11 – input points, aka Boolean values
• Value of the two neurons on 4 pts
  • First neuron (p1)– outputs 1 for 00, 01 and 10
  • Second neuron (p2) – outputs 1 for 01, 10 and 11
  • At this point we are working w 0 and 1 coz it’s the only possible outcome
• We want a function (column o) that outputs 0 on 00, 11 && 1 on 01, 10
• These are the inputs of the second layer coz it was the output of the first layer.
  • P1 p2 was the output of x1x2, but is not an input
• If we focus on last 3 columns, what Boolean function is this?
  • 0 on 10, 1 on 11, and 0 on 01 AND!!!corresponds to intersection
  • Union – OR
• All the second layer has to do is implement AND or OR. Once we go past first layer we have a Boolean function, and all we need to do is do AND or OR depending on CNF or DNF

MLP and XOR -

• 
• This gives a new problem. All possible inputs are 01,10, 11(refer to the table above). And we need a new neuron, given this dataset, that outputs 1 for only the black point (11). And 0 for other pts.

MLP and XOR -

• 
• By composing as many neurons as you want and then doing intersection or the union, we can cover any area of the dataset!! And correctly classify any dataset
• U can do anything but have to be careful of overdo it and overfitting no good algorithm to determine how many neurons we need. It’s a matter of using the validation set to see how much ur overfitting. you want to be good at classifying, and therefore if you see that they are not classifying well, ur underfitting – you can add neuron then!

A Universal Approximator

• Interesting is that it’s been proven ages ago that 2 layers is all you need. you can arbitrarily approximate any function !!
• 
• If you have sth that looks like MLP and you have a function q and any small number epsilon, you will be able to find a number of neurons that gives you an approximation that’s within epsilon
• Deep learning – having many more than 2 layers which solved classification problem. If 2 is all I need, why do more?
  • It’s more of a theoretical thing. To rly approximate sth accurately, you need a hidden layer, if you have only 1 hidden layer, to approximate complex function, hidden layer would have to be ridiculously large!
  • Which used to turn into a computational problem. But is solved by GPU and we have a dataset to support the model
• → General intuition is that by being hierarchical – we can be more efficient!!
• Idea is instead of making hidden layer ridiculously large, we make it short BUT deeper
  • The factor that reduces hidden layer is much more than factor multiplying the neurons
  • If we have 4 hidden layers you can reduce the breadth by more than 4 times

• Can learn low level features in the initial layers that are abstracted more and more as you keep adding layers to more general concepts.

  • Later layers learn more detailed feature – so you see the hierarchy of what they learn

• We like the deep networks, but what should the structure be?

  • Every day people come up w new architecture. Its an experimental evaluation. There are hundreds of new proposals for architecture

Error definition –

• There are a lot of different function you can use
  1. Number of errors on the training set – doesn’t tell you how much ur off by
  2. The perceptron error – fact that error is proportional to the distance and it gives you direction of improvement. BUT only works on perceptron coz its linear.
  3. Mean squared error – always works! Taking the diff btwn output and desired class. It will just cancel each other our but we can force the total error to be positive by squaring. And ½ to cancel out when deriving.
• Problem comes in with the activation function of each neuron – coz derivative is zero everywhere- so it doesn’t help with gradient descent so we gotta replace
• We need a function that is derivable!!!

A different activation function

• V similar to what we wanted, but its rounded sigmoid!
• eq of sigmoid
  • Constant x will control how steep the function is. Can make it vertical although its not, coz then its not differentiable
  • Can make it very small to the sigmoid

The derivative of the sigmoid

• Properties:
  • Derivative of ex is ex
  • Chain rule: (f of g)’s derivative = gx’s derivative * derivative of the whole thing
• 
• So σ’ = σ(1-σ) v imp!!
• So now we get a function that is MSE, but also outputs numbers that are in btwn 0 and 1.

Conclusion

• MLP is a bunch of neuron layers with this activation function instead of the step activation function!!
• Another that’s used a lot is a hyperbolic tangent