Neural Networks Basics

Artificial Neural Networks

Neural networks are the foundation of deep learning. They're inspired by the structure of the human brain — layers of interconnected neurons that process information. Each neuron receives inputs, applies weights and a bias, and passes the result through an activation function. Stack enough of these neurons in layers, and the network can learn incredibly complex patterns.

Structure of a Neural Network

  ┌─────────────────────────────────────────────────┐
  │            NEURAL NETWORK ARCHITECTURE           │
  │                                                 │
  │   Input       Hidden Layers      Output         │
  │   Layer                            Layer         │
  │                                                 │
  │    (x₁)      ┌──────┐           ┌──────┐       │
  │   ╱    ╲────►│ h₁₁  │──╲    ╱──►│  y₁  │       │
  │              └──────┘   ╲  ╱    └──────┘       │
  │    (x₂)      ┌──────┐   ╲╱     ┌──────┐       │
  │   ╱    ╲────►│ h₁₂  │──╱╲╲────►│  y₂  │       │
  │              └──────┘   ╱  ╲    └──────┘       │
  │    (x₃)      ┌──────┐ ╱    ╲                   │
  │   ╱    ╲────►│ h₁₃  │╱      ╲                  │
  │              └──────┘        ╲                  │
  │                                                 │
  │  Each connection has a weight                   │
  │  Each neuron has an activation function         │
  └─────────────────────────────────────────────────┘

How a Single Neuron Works

  Inputs     Weights    Sum + Bias    Activation    Output
  ────       ──────     ──────────    ──────────    ──────

  x₁ ──w₁──╲
            ╲
  x₂ ──w₂──► Σ (wᵢxᵢ) + b ──── σ(·) ────► output
            ╱
  x₃ ──w₃──╱

  Σ = weighted sum of inputs + bias
  σ = activation function (introduces non-linearity)

Activation Functions

• Sigmoid — Squashes values between 0 and 1. Used in output layers for binary classification.
• ReLU (Rectified Linear Unit) — Returns max(0, x). The most popular hidden layer activation. Simple and effective.
• Tanh — Squashes values between -1 and 1. Zero-centered, which can help training.
• Softmax — Converts outputs to probabilities that sum to 1. Used in multi-class classification output layers.

Training: Forward and Backward Pass

Training happens in two phases:

1. Forward Pass — Input flows through the network layer by layer to produce a prediction.
2. Backward Pass (Backpropagation) — The error is calculated, and gradients flow backward to update weights. Gradient descent adjusts each weight to reduce the error.

This process repeats for many epochs (complete passes through the training data) until the model converges.

Why Deep Learning Works

The "deep" in deep learning refers to the multiple hidden layers. Each layer learns increasingly abstract features. For image recognition: layer 1 detects edges, layer 2 detects shapes, layer 3 detects object parts, layer 4 detects whole objects. This hierarchical feature learning is what makes deep networks so powerful — and so different from traditional ML where you had to engineer features manually.