Skip to main content
NeuroLearn AIACADEMY
Interactive AI Lesson• 2 min

Gradient Descent & the Mathematical Mechanics of Learning

How neural networks minimize loss through gradient descent and backpropagation.

Live — computed in your browser
Yesz=4.284.3%
Noz=2.110.3%
Maybez=1.03.4%
Perhapsz=0.41.9%
Low temperature sharpens the distribution towards the highest-scoring token; high temperature flattens it, making unlikely tokens reachable. At T=0 the model becomes deterministic.

The Mathematical Mechanics of Learning

A deep neural network is a parameterized differentiable function f(x; W, b) that maps input vectors x to predictions y_hat.

The Training Loop:

  1. Forward Propagation:
    • z = W * x + b
    • a = sigma(z) (Non-linear activation)
  2. Loss Calculation:
    • Loss = -sum( y_i * log(y_hat_i) ) (Cross-Entropy Loss)
  3. Backpropagation:
    • Compute partial derivatives of Loss with respect to weights using the Chain Rule of calculus.
  4. Gradient Descent Update:
    • W = W - eta * (dL / dW) (where eta is the learning rate).
Example — python
Run it in Colab
import numpy as np

# Single Step Gradient Descent Demonstration
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def sigmoid_derivative(x):
    s = sigmoid(x)
    return s * (1 - s)

X = np.array([[0.5, 0.2]])
y = np.array([[1.0]])
W = np.random.randn(2, 1)
lr = 0.1

# Forward Pass
z = np.dot(X, W)
pred = sigmoid(z)
loss = 0.5 * np.sum((pred - y)**2)

# Backward Pass (Chain Rule)
d_loss_pred = pred - y
d_pred_z = sigmoid_derivative(z)
d_z_W = X.T
gradient = np.dot(d_z_W, d_loss_pred * d_pred_z)

# Weight Update
W -= lr * gradient
print("Updated Weights:
", W)
Knowledge Checkpoint

In deep neural network optimization, what does the gradient vector represent?

Read to the end of the lesson