Tech

Deep learning and LLMs

Count parameters and memory in your head, explain what tokens, attention and caches cost, and build LLM (large language model) features that are grounded, tested and safe to put in front of clients.

Read first: Machine learning in practice

7 concepts8 min read25-question test, from a bank of 33Updated

Concept 1 of 71 min

Parameters, softmax and cross-entropy

The rule
A dense layer has inputs × outputs weights plus one bias per output, and softmax turns logits into probabilities with ez ÷ Σez.
Worked examples

A dense layer

Given
Inputs300
Outputs100
  1. Weights = 300 × 100 = 30,000
  2. Parameters = 30,000 + 100 = 30,100

The 100 biases are the part people forget.

A 5 by 5 convolution

Given
Filter5 by 5
Channels in32
Channels out64
  1. Per filter = 52 × 32 + 1 = 801
  2. Parameters = 64 × 801 = 51,264

The image size does not enter, because the same filter slides over the whole image.

Softmax of 2 logits

Given
Logits1 and 0
e2.718
  1. P(first) = 2.718 ÷ (2.718 + 1) ≈ 0.731

Softmax exponentiates before dividing, so 1 and 0 give 0.731, not 1.

The picture
Formula
Dense=nin × nout+nout
convolution=(k × k × cin + 1) × cout
softmax_i=ezi ÷ Σ ezj
loss=−ln p(correct class)
Why it holds

A convolution has (kernel height × kernel width × input channels + 1) parameters per output channel, whatever the image size. Cross-entropy is −ln of the probability given to the right class. Forgetting the biases is the classic slip in parameter counts.

A convolution slides the same filter over the whole image, which is why the image size does not enter. Dividing logits by their sum skips the exponential and gives the wrong probabilities.

Cross-entropy punishes confident mistakes heavily: a right class given 1% costs far more than one given 50%; using log base 10 gives a smaller, wrong number.

The loss is about 4.61 when the right class gets 0.01; 2.30 when it gets 0.1; 0.693 at 0.5; 0.105 at 0.9.

Watch out
  • Forgetting the biases.
  • Multiplying a convolution's parameters by the image size.
  • Normalising logits without the exponential.
In the interview
  1. Why does a convolutional network need far fewer parameters than a dense network on images?

    A complete answer covers

    • A convolution reuses the same small filter at every position.
    • Its parameters depend on kernel size and channels, not image size.
    • A dense layer would connect every pixel to every unit.
    • Sharing weights also builds in that a pattern means the same thing anywhere.

Also in our technical interviews, on camera, with a score.

Concept 2 of 71 min

Training deep networks that learn

The rule
Sigmoid derivatives of at most 0.25 shrink gradients layer after layer, which ReLU, residual connections and normalisation prevent.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 3 of 71 min

Memory: weights, training, experts and caches

The rule
Weights take parameters × bytes per parameter, 2 bytes in 16-bit and half a byte in 4-bit.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 4 of 71 min

Tokens, attention, temperature and word order

The rule
LLMs are billed and limited in tokens, about 0.75 English words each, and self-attention's cost grows with the square of the context length.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 5 of 71 min

Retrieval, embeddings and vector search

The rule
Retrieval-augmented generation answers from retrieved source text, which cuts invented facts, and a few sharp passages beat many pages.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 6 of 71 min

Shipping LLM features: alignment, evaluation and safety

The rule
Preference training such as RLHF (reinforcement learning from human feedback) aligns behaviour after pretraining, and text inside data can carry prompt injection and must not steer actions.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Concept 7 of 71 min

AdvancedThe key-value cache: why serving runs out of memory

The rule
A transformer keeps a key and a value per past token in each layer, 2 × layers × hidden size × bytes per number.

The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.

See the plansSign in

Recap

  • Dense layer: inputs × outputs + outputs; convolution: (k × k × channels in + 1) × channels out, whatever the image size.
  • Weights take parameters × bytes: 2 bytes in 16-bit, half a byte in 4-bit; training with Adam in 32-bit needs about 16 bytes a parameter.
  • About 0.75 English words a token; attention cost grows with the square of the context; temperature near 0 gives the likeliest answer.
  • Ground answers in retrieved text, keep the context short and relevant, search embeddings with an ANN (approximate nearest neighbour) index and compare them by cosine.
  • Validate outputs, test prompts on a fixed set, treat text inside data as untrusted, and distrust public benchmark scores.
Now test it

25 questions on Deep learning and LLMs

Timed, like the real one, drawn from a bank of 33. Every answer explained.

Test yourself →