Class 8 · Unit 1: The AI project lifecycle · Lesson 1.4 · 40 min
How good is it? Evaluating and refining
“Ninety-five per cent!” On the photos it studied. On twenty it had never seen: thirteen right.
Today you will: Test on photos the model never saw · Find what the mistakes have in common · Change one thing, and test again
Story
—: “Ninety-five per cent. I showed it the training photos and it got nearly all of them.” (Bilal announces.)
Hiba: “Of course it did. It studied those.” (She opens the folder nobody has touched: test, twenty photos, the answers written down a week ago and not looked at since.)
They go through them one at a time. Wrapper — correct. Peel — correct. Bottle — correct. Then a greasy paper plate, which the model calls dry with 88% confidence, and a tetra pack, which it calls wet.
Thirteen right out of twenty. Sixty-five per cent.
Bilal stares at the screen. "So it's rubbish."
Hiba: “No. Now we know something. Look — both wrong ones are food-stained paper. That's not rubbish. That's a clue.” (writing the two mistakes down.)
Watch
Accuracy, and what it hides — — scan code 8.1.4 in the printed book.
Warm up your fingers
Skill: the number row and the % sign (6 min)
- On typing.com, open the numbers and symbols lessons. Reach for the number row without moving your wrists.
- Then type this line five times, exactly:
13 of 20 correct = 65% accuracy - Term 1 goal: 28–32 WPM at 93% accuracy. You will be typing numbers all lesson.
On the laptop
You need: Teachable Machine · LibreOffice Calc · waste/test/ · answers.ods · model-scores.ods
Mission 1: Test it honestly (10 min, in pairs)
- ☐1
Predict: will the test score be higher or lower than the training score? By how much?
- ☐2
Upload each of the twenty test photos. Record: Photo · Model's answer · Confidence.
- ☐3
Only now open
answers.odsand fill True label. In Right?:=IF(B2=C2;1;0), fill down. Accuracy:=SUM(E2:E21)/20*100.
Mission 2: Read the mistakes (8 min)
- ☐4
Filter to Right? = 0. What do the wrong ones have in common — food-stained paper? shine? shade?
- ☐5
Look at the confidence on the wrong ones. Confidently wrong is the dangerous kind.
Mission 3: One change only (6 min)
- ☐6
Choose one fix: add more data · remove incorrect data · change the settings · retrain. Make it, retrain, and test the same twenty. Up, down, or the same? Even worse is a real result.
No laptop today?
Use the Lesson 1.3 picture cards: the volunteer sorts twenty new cards, the class works out the percentage, then lays the wrong cards in a row — what do they share? — and tests again after five more examples.
Now you know
- Stage 4 is evaluation and refinement. Test on data the model has never seen.
- Accuracy = correct ÷ total × 100 and it's only one number.
- High training score, low test score = memorised, not learned.
- The mistakes are the information. Group them; find what they share.
- Change one thing at a time, then test again — or you can't tell what worked.
🕌 From our heritage
Al-Battānī (c. 858–929) inherited Ptolemy's astronomical tables, which had been trusted for seven hundred years. He did not simply copy them: he checked their predictions against his own observations of the sky, and corrected the values he found to be wrong — including showing that the Sun's apogee, which Ptolemy had declared fixed, had in fact moved since Ptolemy's time. Testing an inherited answer against fresh evidence, and refining it, is the stage you are working in today.
Source: al-Battānī, al-Zīj al-Ṣābiʾ (early 10th century).
Debate it
The waste model is 90% accurate. Then you look at the mistakes: it is right about every wrapper and peel, and wrong about almost every hazardous item — batteries, a broken bulb, a medicine strip.
- Is 90% a fair description of this model?
- Which mistake costs more: calling a peel dry, or calling a battery dry?
- What would you change first, and what would you measure to know whether it worked?
Check yourself — practice, not a test
1. Accuracy is calculated as:
- ○ Total answers ÷ correct answers
- ○ Correct answers ÷ total answers × 100
- ○ Confidence score × 100
- ○ The number of training photos
2. A model that scores 95% on its training photos is proven to work well.
- ○ True
- ○ False
3. CBSE's ways to refine a model include: (choose all that apply)
- ○ Add more data
- ○ Remove incorrect data
- ○ Delete the test set
- ○ Change the settings and retrain
4. The data kept back before training, used to check the model afterwards, is called the ______ dataset.
5. Why should you change only one thing before retraining?
- ○ It is faster
- ○ The tool allows only one change
- ○ So you know which change caused the difference
- ○ To keep the accuracy high
Remember
Test on what it has never seen. Then read the mistakes, and change one thing.