Skip connections, also known as residual connections, play a crucial role in improving the training and performance of deep neural networks. The primary motivation for incorporating skip connections into neural networks is to address the vanishing gradient problem and to facilitate the training of very deep networks. Here's a more detailed explanation of the motivations:

Vanishing Gradient Problem:
- Issue: As neural networks become deeper, the gradients during backpropagation can become very small, approaching zero, especially in the early layers. This is due to the chain rule of differentiation, where gradients are multiplied at each layer. When gradients become extremely small, it hinders the ability of the model to update the weights of early layers effectively.
- Motivation for Skip Connections: Skip connections provide a direct path (shortcut) from one layer to another, allowing gradients to flow more easily. This mitigates the vanishing gradient problem because gradients can travel both through the skip connections and the regular layer-to-layer connections.
Ease of Training:
- Issue: Very deep networks without skip connections are challenging to train because they often suffer from convergence issues and poor generalization.
- Motivation for Skip Connections: Skip connections make it easier to train deep networks. By providing shortcuts for gradients and information flow, they enable the network to learn both high-level and low-level features effectively. This makes it possible to train much deeper networks than previously possible.
Network Depth and Representational Power:
- Issue: Deeper networks can potentially capture more complex and hierarchical features from the data. However, without skip connections, there's a limit to how deep you can effectively train a network.
- Motivation for Skip Connections: Skip connections allow you to build very deep networks while maintaining the ability to capture both low-level and high-level features. This increases the representational power of the network, enabling it to model more complex relationships in the data.
Residual Learning:
- Idea: Skip connections introduce the concept of "residual learning." Instead of trying to directly learn the mapping from the input to the output, the network learns the residual (difference) between the input and the output. This is expressed as F(x) = H(x) - x, where H(x) is the learned mapping.
- Motivation for Skip Connections: Residual learning simplifies the learning process, as the network focuses on learning the difference between the input and output rather than trying to model the entire mapping. This often leads to faster convergence and better generalization.

In summary, skip connections address training challenges associated with very deep neural networks by facilitating gradient flow, easing training, increasing representational power, and introducing the concept of residual learning. They have been a critical innovation in the development of deep convolutional neural networks (CNNs) and have significantly contributed to breakthroughs in computer vision and other domains.

Ref: https://theaisummer.com/skip-connections/