Concept 1 of 71 min
Parameters, softmax and cross-entropy
A dense layer
| Inputs | 300 |
|---|---|
| Outputs | 100 |
- Weights = 300 × 100 = 30,000
- Parameters = 30,000 + 100 = 30,100
The 100 biases are the part people forget.
A 5 by 5 convolution
| Filter | 5 by 5 |
|---|---|
| Channels in | 32 |
| Channels out | 64 |
- Per filter = 52 × 32 + 1 = 801
- Parameters = 64 × 801 = 51,264
The image size does not enter, because the same filter slides over the whole image.
Softmax of 2 logits
| Logits | 1 and 0 |
|---|---|
| e | 2.718 |
- P(first) = 2.718 ÷ (2.718 + 1) ≈ 0.731
Softmax exponentiates before dividing, so 1 and 0 give 0.731, not 1.
- Natural log
A convolution has (kernel height × kernel width × input channels + 1) parameters per output channel, whatever the image size. Cross-entropy is −ln of the probability given to the right class. Forgetting the biases is the classic slip in parameter counts.
A convolution slides the same filter over the whole image, which is why the image size does not enter. Dividing logits by their sum skips the exponential and gives the wrong probabilities.
Cross-entropy punishes confident mistakes heavily: a right class given 1% costs far more than one given 50%; using log base 10 gives a smaller, wrong number.
The loss is about 4.61 when the right class gets 0.01; 2.30 when it gets 0.1; 0.693 at 0.5; 0.105 at 0.9.
- Forgetting the biases.
- Multiplying a convolution's parameters by the image size.
- Normalising logits without the exponential.
Why does a convolutional network need far fewer parameters than a dense network on images?
A complete answer covers
- A convolution reuses the same small filter at every position.
- Its parameters depend on kernel size and channels, not image size.
- A dense layer would connect every pixel to every unit.
- Sharing weights also builds in that a pattern means the same thing anywhere.
Also in our technical interviews, on camera, with a score.
Concept 2 of 71 min
Training deep networks that learn
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 3 of 71 min
Memory: weights, training, experts and caches
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 4 of 71 min
Tokens, attention, temperature and word order
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 5 of 71 min
Retrieval, embeddings and vector search
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 6 of 71 min
Shipping LLM features: alignment, evaluation and safety
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inConcept 7 of 71 min
AdvancedThe key-value cache: why serving runs out of memory
The why, the charts, the worked example and the interview questions are in Essentials, with every quiz.
See the plansSign inRecap
- Dense layer: inputs × outputs + outputs; convolution: (k × k × channels in + 1) × channels out, whatever the image size.
- Weights take parameters × bytes: 2 bytes in 16-bit, half a byte in 4-bit; training with Adam in 32-bit needs about 16 bytes a parameter.
- About 0.75 English words a token; attention cost grows with the square of the context; temperature near 0 gives the likeliest answer.
- Ground answers in retrieved text, keep the context short and relevant, search embeddings with an ANN (approximate nearest neighbour) index and compare them by cosine.
- Validate outputs, test prompts on a fixed set, treat text inside data as untrusted, and distrust public benchmark scores.
25 questions on Deep learning and LLMs
Timed, like the real one, drawn from a bank of 33. Every answer explained.