4. Neural Network Examples

Neural Network

Neural Networks are made up of simple processor nodes (neurons) connected by weighted links (synapses), each node receives one or more inputs, and each node produces one output which can be sent to other neurons.

A neural network learns through repeated adjustments of weighted links between neurons. They can be used in a variety of situations and can be use to calculate linear regression (and a few other regression models) automatically. It can be used for unsupervised, supervised and reinforcement learning.

Link to original

Deep Neural Network

“Deep Neural Networks” are a subset of neural networks with a significantly large hidden layer count, with layers often used to isolate specific problems.

Link to original

NN Architecture

Neural Network Layers

Each layer has one or more nodes / neurons.

• The input layer provides information from the outside world to the network.
• The output layer transfers information to the outside world.
• The middle layers are called the hidden layers because they have no direct connection with the outside world, we can have $0$ or more hidden layers.

Link to original

There are two main types of NNs:

• Feedforward NN

  Feedforward Network: all information moves in only one direction, forwards, from the input nodes through the hidden nodes (if any) to the output.

  Link to original

• Recurrent NN

  Recurrent Network: nodes have feedback loops (back to themselves / previous nodes)

  Link to original

Neuron

Each neuron (after the input layer) is reliant on an activation function to generate an output.

• Weighted summation value must exceed a threshold.
• Different result in different outputs.

Link to original

Sigmoidal	Hyperbolic Tangent (TanH)	Rectified Linear Unit (ReLU)

NN in practice

The output of a network is effectively a non-linear function of the input values $(x)$ given the output values $(f)$ are the weighted summation of preceding functions in the hidden layers.

Training a Neural Network

Typically we initialise a neural network with random weights, it’s functionally useless and does not give us correct answers. We want to find the optimal set of weights that allow for the network to solve a given task.

We adjust weights using a number of different learning approaches while assessing the current network against the training data and whether it identifies objects successfully or within a level of accuracy.

Backpropagation

Backpropagation

Backpropagation (of errors) uses gradient descent to minimise the failure rate against the training set. Upon testing against every item in the training set, we calculate the error-rate (overall error between desired output and current output). Then we can compute a backward error to adjust weights at each layer and run the process again until error rate stabilises.

$${\theta }^{t+1}={\theta }^t-\alpha \frac{\sigma E(X,{\theta }_t)}{\sigma \theta }$$

Link to original