Convolutional Neural Networks: From LeNet to Modern Architectures

Convolutional Neural Networks (CNNs) remain the backbone of computer vision. Understanding their evolution from LeNet-5 (1998) to modern architectures like ConvNeXt reveals fundamental principles of deep learning design.

Core Building Blocks

A CNN stacks three types of layers:

Convolutional layers apply learnable filters (kernels) that slide across the input, detecting local patterns. A $3 \times 3$ kernel with stride 1 on a $28 \times 28$ image produces a $26 \times 26$ feature map. Each filter learns to detect a specific pattern — edges, textures, shapes — with increasing abstraction at deeper layers.

Pooling layers reduce spatial dimensions. Max pooling takes the maximum value in each $2 \times 2$ window, halving width and height. This provides translation invariance and reduces computation. Average pooling and global average pooling (GAP) are common alternatives.

Fully connected layers at the end flatten the feature maps and map them to output classes. Modern architectures often replace FC layers with GAP to reduce parameters.

CNN Architecture (simplified)
==============================

Input Image [3 x 224 x 224]
     │
     ▼
┌─────────────────┐
│ Conv 3x3, 64    │──► ReLU ──► BatchNorm
│ Conv 3x3, 64    │──► ReLU ──► BatchNorm
└────────┬────────┘
         ▼
    MaxPool 2x2         [64 x 112 x 112]
         │
         ▼
┌─────────────────┐
│ Conv 3x3, 128   │──► ReLU ──► BatchNorm
│ Conv 3x3, 128   │──► ReLU ──► BatchNorm
└────────┬────────┘
         ▼
    MaxPool 2x2         [128 x 56 x 56]
         │
         ▼
    ... (deeper blocks)
         │
         ▼
    Global Avg Pool     [512 x 1 x 1]
         │
         ▼
    FC ──► Softmax      [num_classes]

Architecture Evolution

The ResNet Revolution

ResNet's skip connections solved the degradation problem — deeper networks were harder to train, not because of vanishing gradients, but because learning identity mappings is hard for stacked nonlinear layers. The residual formulation $\mathcal{F}(x) + x$ makes identity the default, letting additional layers only learn the residual.

Residual Block
===============

Input x ──────────────────────────┐
    │                             │
    ▼                             │
 Conv 3x3 ──► BN ──► ReLU        │ (skip / shortcut)
    │                             │
    ▼                             │
 Conv 3x3 ──► BN                  │
    │                             │
    ▼                             │
  (+) ◄───────────────────────────┘
    │
    ▼
  ReLU
    │
    ▼
 Output: F(x) + x

Modern Techniques

Depthwise separable convolutions (MobileNet): factor a standard convolution into a depthwise (per-channel) and pointwise (1x1) convolution. Reduces computation by $\sim 8$-$9\times$ with minimal accuracy loss. Essential for mobile and edge deployment.

Attention in CNNs: Squeeze-and-Excitation (SE) blocks recalibrate channel responses. CBAM adds spatial attention. These mechanisms help the network focus on relevant features.

Data augmentation at scale: RandAugment, CutMix, MixUp, and AugMax have become standard. They act as regularizers and can improve top-1 accuracy by 1-3% without architectural changes.

Knowledge distillation: train a small "student" CNN to mimic a large "teacher" model's output distribution. The soft targets carry more information than hard labels, enabling compact models that retain most of the teacher's performance.

CNNs vs Vision Transformers

Vision Transformers (ViT) process images as sequences of patches using self-attention. They outperform CNNs on large datasets but require more data and compute. The landscape in 2026:

The trend is convergence: ConvNeXt brings Transformer design principles to CNNs, while models like CoAtNet and EfficientFormer combine both.