DSE ICT | Artificial Intelligence and Machine Learning Lesson-Prep Guide (Tutor Edition) | Confusion Matrix, Data Splits and Overfitting

2026-11-02

Tutor lesson-prep guides · Secondary ICT

Students have heard of AI, but ask how accurate a model is and they answer that it is very accurate. This lesson splits accuracy into three numbers.

Objectives — Students can: (1) compute accuracy, precision, recall and F1 from the four cells of a confusion matrix; (2) explain why a 95% accurate model can be useless, using class imbalance; (3) tell overfitting from underfitting and state the correct use of training and test sets with three remedies.

01 Prerequisites (3-minute check)

If any of these fails, fix it before the lesson starts.

Skill	Quick check	Pass mark
Percentages	What percentage is 174 out of 200	87%
Confusion matrix	What do TP, FP, FN and TN mean	Correct positive, false alarm, missed case, correct negative
Averages	How is a root mean square error built	Square, average, then take the root

02 Lesson flow (60 minutes)

Time	Stage	Tutor does	Student does
0-10 min	AI versus ML	Rules written by people versus rules learned from data	One example each
10-22 min	Supervised pipeline	Label, split, train, evaluate	Write 7,000, 1,500, 1,500
22-36 min	The four cells	TP 72, FP 8, FN 18, TN 102	Fill all four correctly
36-46 min	Three metrics	Accuracy 87%, precision 90%, recall 80%	Compute F1 of 84.7%
46-56 min	Overfitting and remedies	Train 99.2% against test 71.5%, a 27.7-point gap	Give three remedies
56-60 min	Exit check	Three questions marked on the spot	All three right

03 Teaching script (read it out)

Scene 1: Machine learning finds rules in data rather than having them written

Classic programs are hand-written conditions; machine learning is given 10,000 labelled samples and finds the boundary itself.
Samples split three ways: 70% training, 15% validation, 15% test, which is 7,000, 1,500 and 1,500. The test set is used once, at the end; tuning against it is like practising on the exam paper, and scores inflate.
The simplest classifier is k nearest neighbours: with k = 5 and neighbours split 3 cats to 2 dogs the answer is cat with confidence 3 / 5 = 60%. Larger k smooths the boundary, smaller k is dragged by one noisy point.

Ask students: why the test set must not be watched during training (it becomes practice, so it no longer estimates new data).

Scene 2: A 95% accurate model can be completely useless

Take 200 test cases: TP 72, FP 8, FN 18, TN 102, and the four cells must total 200, the first check to make.
Accuracy is (72 + 102) / 200 = 87%; precision is 72 / 80 = 90% (alarms that were real); recall is 72 / 90 = 80% (real cases caught); F1 is 2 x 0.9 x 0.8 / 1.7 = 84.7%.
Now take 1,000 cases with only 5% positive: a model that always answers negative scores 95% accuracy and 0% recall. With imbalance, report recall and F1, never accuracy alone.

Ask students: why a hospital screen prefers false alarms to missed cases (a missed case sends a sick patient home).

Scene 3: Overfitting is memorising, not learning

Training 99.2% against test 71.5% is a gap of 27.7 percentage points, the classic overfitting signal: the model has stored noise in the training set.
A tree of depth 8 can hold 2 to the power 8 = 256 leaves, and more leaves let each hold one or two samples, which is copying the answers. Three remedies: more data, pruning or a shallower tree, and regularisation that limits weights.
Underfitting is the opposite, with 60% on both sets, needing a stronger model or better features. For regression, residuals 0.3, -0.2, 0.5 and 0.1 give RMSE = root of 0.0975 = 0.312, in the same unit as the target so it can be interpreted.

Ask students: what does a high training score with a low test score tell you (the model memorised and does not generalise).

04 Question types and worked steps

Type	Example	Steps	Answer
Calculation	Accuracy and recall	Accuracy is correct over total; recall is TP over TP plus FN	87% and 80%
Calculation	F1	Twice precision times recall over their sum	84.7%
Calculation	Data split	Total times 70, 15 and 15 percent	7,000, 1,500, 1,500
Explanation	Detecting overfitting	Compare train and test, then give three remedies	A 27.7-point gap

Sanity check (teach this): The four cells must add to the test total, so 200 means 200; anything else is a copied cell.
Precision divides by everything the model flagged, recall divides by everything truly positive: two different denominators.
F1 always sits between precision and recall; a value outside that range means the formula was misapplied.
A percentage cannot exceed 100, and a training score above the test score is normal; the reverse points to data leakage.
RMSE carries the unit of the target and cannot be compared with a percentage.

05 Independent practice (3 questions)

Question	Answer	What to watch
Accuracy for TP 72, FP 8, FN 18, TN 102	87%	174 out of 200
F1 for the same matrix	84.7%	Precision 90 and recall 80 into 2PR/(P+R)
Split of 10,000 samples 70/15/15	7,000, 1,500, 1,500	The three parts add to 10,000

06 Common mistakes and what to say

Mistake	Student writes	Say this
Reporting accuracy alone	Answering that a 95% model is reliable	Only 5% are positive, so always answering no scores 95%; report recall.
Swapping precision and recall	Dividing TP by the same number twice	Precision divides by flags raised, recall by real cases.
Tuning on the test set	Watching the test score while adjusting parameters	Tune on validation and use the test set once.
Equating a high score with a good model	Shipping at 99.2% training accuracy	Check the test score of 71.5%: a 27.7-point gap means overfitting.

07 Homework (paper, section and question numbers)

Secondary ICT chapter index — All chapters and practice
All lesson-prep guides — Compare other subjects

08 Exit check (last 5 minutes)

Question	Answer	Decision
Accuracy and recall	87% and 80%	All three right: move to system development
F1 value	84.7%	One wrong: reteach the denominators
Evidence of overfitting	A 27.7-point train-test gap	Two wrong: return to act three

09 Tutor FAQ

Must students learn the mathematics of neural networks?

No. The syllabus tests concepts and metric arithmetic, not deep mathematics.

Why keep a separate validation set?

Tuning uses validation so the test set stays untouched until the final estimate.

Is higher accuracy always better?

Not with imbalance: report recall, precision and F1 and explain the business cost.

Find a tutor — free Tutor Registration →

A TutorZone lesson-prep guide. Written by the TutorZone team; questions and lesson flow are original and are not taken from any school or publisher material.