Artificial Intelligence: Question 9
Syllabus 18.1
Two artificial neural networks are built for the same handwriting-recognition task, and both are trained using back propagation of errors on the same set of labelled handwriting samples.
- Network 1 has a single hidden layer.
- Network 2 has twenty hidden layers, and is described by its developers as using deep learning.
(a) State the specific structural feature that makes Network 2 an example of deep learning, which Network 1 does not have. [1]
(b) Explain one advantage the many hidden layers in Network 2 can give it, compared with Network 1, when learning to recognise complex handwriting patterns. [2]
(c) Explain how training Network 2 differs from training Network 1 in terms of how far the error found at the output must be propagated back through the network. [2]
Show worked solution Hide worked solution
Worked solution
Part (a): The structural feature that defines deep learning
Network 2 has many (twenty) hidden layers, whereas Network 1 has only a single hidden layer. Having many hidden layers between the input and output layers is the specific structural feature that makes a network an example of deep learning. Network 1, with only one hidden layer, is not described as deep learning.
[1 mark] for correctly identifying that having many (twenty) hidden layers, rather than just one, is what makes Network 2 an example of deep learning.
Part (b): Advantage of many hidden layers
With many hidden layers, each layer in Network 2 can learn increasingly complex features, building on the simpler features already learned by the layer before it. For example, early layers might learn to recognise simple strokes, later layers might combine these into letter shapes, and later layers still might combine those into recognisable handwriting patterns.
Network 1, with only a single hidden layer, has far less capacity to build up features in this layered way, so it is generally less able to learn such complex patterns directly from the raw data.
[2 marks]: [1] for stating that additional hidden layers let the network learn increasingly complex features, [1] for explaining that each layer builds on the simpler features learned by the layer before it, giving Network 2 an advantage over Network 1’s single hidden layer.
Part (c): How training differs in terms of back propagation distance
Both networks are trained with the same overall process: the predicted output is compared with the known, correct label to find the error, and this error is passed back through the network, adjusting weights as it goes.
The difference is how far this error has to travel. In Network 1, the error only has to be propagated back through its single hidden layer to reach and adjust every weight in the network. In Network 2, the same error must be propagated back through all twenty hidden layers in turn, adjusting the weights at each layer along the way before it reaches the earliest layers, closest to the input.
[2 marks]: [1] for stating that Network 2’s error must be propagated back through many more (twenty) hidden layers than Network 1’s single hidden layer, [1] for explaining that the weights at each of those layers are adjusted in turn as the error passes back through them.
Final answers
- (a) Network 2 has many (twenty) hidden layers; this is the feature that makes it deep learning
- (b) Additional hidden layers let Network 2 learn increasingly complex features, each layer building on the one before it, an advantage Network 1’s single hidden layer lacks
- (c) Network 2’s error must be propagated back through twenty hidden layers, adjusting weights at each one, compared with just one hidden layer for Network 1