Class 10 · Unit 3: Evaluating models · Lesson 3.1 · 40 min

Why evaluate, and the train-test split

The leaf app scored 99% at the exhibition. In the orchard it was wrong again and again. Both were true.

Today you will: Split data into training and testing · Catch a model that memorised · Say why a test must use unseen data

Story

Last year's science exhibition had one star: a laptop app that looked at a photo of an apple leaf and said healthy or scab. The sign beside it said 99% accurate, and it was true.

In spring, Hiba's uncle tried it on his own trees. It called a scabbed leaf healthy. Then another. Then it called a healthy one scab.

Uncle: “The sign said ninety-nine.”

Hiba finds the team's notes. They took 200 photos, trained the app on them, and then tested it — on the same 200 photos.

Bilal: “So it wasn't lying.”

Hiba: “No. It was answering questions it had already seen the answers to.”

Watch

The report card for a model — — scan code 10.3.1 in the printed book.

Warm up your fingers

Skill: a lookup formula (5 min)

On typing.com: the symbols lessons.
Type three times, eyes on the screen:

=VLOOKUP(B2;$Training.$B$2:$F$29;5;0)
A dollar sign fixes a reference so it doesn't move when you fill down. Goal this term: 38–44 WPM at 95%.

On the laptop

You need: LibreOffice Calc · datasets/apples.csv (with the Training and Testing sheets you made in Lesson 2.1)

Mission 1: A model that learned (10 min, in pairs)

☐1
Open your apples file with its two sheets: Training (A01–A28) and Testing (A29–A36). Unhide variety in Testing.
☐2
In both sheets, put your best rule from Lesson 2.2 in column G, and score it with =IF(G2=F2;1;0) in column H.
☐3
Predict: will the rule score better on Training or on Testing? Write it in J1.
☐4
Add up both. Write each score as right out of how many.

Mission 2: A model that memorised (10 min)

☐5
Now build a cheat. In Training, column I: =VLOOKUP(B2;$Training.$B$2:$F$29;5;0). It finds a training apple with exactly the same weight, and copies its variety. Fill down and score it. It does very well. Why?
☐6
Put the same formula in Testing, column I. Fill down. What happens?
☐7
Fill in the table in J5:

Training score Testing score
The model that learned
The model that memorised

Mission 3: The sign at the exhibition (3 min)

☐8
In J10, rewrite the exhibition sign so that it is honest. Keep it under 15 words.

Finished early?

Move 4 apples from Training to Testing and score both models again. Does a bigger testing set change your trust in the scores?

No laptop today?

Your teacher writes 10 apples on the board and hides 3 of them. The class writes a rule using the 7, then tests it on the hidden 3. Then one student "memorises" the 7 aloud and tries to answer the hidden 3 from memory — and can't.

Now you know

36 apples split into 28 for training and 8 for testing. A model that learned scores 27 of 28 and 7 of 8; a model that memorised scores 28 of 28 and cannot answer new apples.

Evaluation is the model's report card. Metrics tell you how well it works, so you can improve it — and know how far to trust it before anyone relies on it.
The train-test split. Learn from the training set. Measure on a testing set the model has never seen. That score estimates how it will do on new data.
Never test on training data. A model can remember the training set and answer it perfectly. That is overfitting: memory that looks like skill.
The training score can't tell them apart. Both models scored 27 out of 28 on training. Only the test showed which one had learned.
The honest number is the test number, even when it is lower — and it usually is.

Debate it

The exhibition team were not cheating on purpose. They were proud of a real number.

Who should have asked "tested on what?" — the team, the judges, or the visitors?
Hiba's uncle used the app on his own trees. What harm could a wrong healthy have done?

Check yourself — practice, not a test

1. Why is the testing set kept away from the model during training?
- ○ It is too small to learn from
- ○ So its score shows how the model does on data it has never seen
- ○ Testing data has no labels at all
- ○ Models can only read one set at a time
2. A model that scores 100% on its own training data is certain to work well on new data.
- ○ True
- ○ False
3. When a model remembers its training data instead of learning a pattern, it is called ______.
4. Put the train-test procedure in order:
- 1. Split the labelled data into training and testing sets
- 2. Compare its predictions with the true labels
- 3. Train the model on the training set
- 4. Ask the model to predict the testing set
5. CBSE compares model evaluation to:
- ○ A driving licence
- ○ A report card
- ○ A recipe
- ○ A map

Remember

Believe the score on data the model has never seen.

Story

Last year's science exhibition had one star: a laptop app that looked at a photo of an apple leaf and said healthy or scab. The sign beside it said 99% accurate, and it was true.

In spring, Hiba's uncle tried it on his own trees. It called a scabbed leaf healthy. Then another. Then it called a healthy one scab.

Uncle: “The sign said ninety-nine.”

Hiba finds the team's notes. They took 200 photos, trained the app on them, and then tested it — on the same 200 photos.

Bilal: “So it wasn't lying.”

Hiba: “No. It was answering questions it had already seen the answers to.”

On the laptop

You need: LibreOffice Calc · datasets/apples.csv (with the Training and Testing sheets you made in Lesson 2.1)

Mission 1: A model that learned (10 min, in pairs)

☐1
Open your apples file with its two sheets: Training (A01–A28) and Testing (A29–A36). Unhide variety in Testing.
☐2
In both sheets, put your best rule from Lesson 2.2 in column G, and score it with =IF(G2=F2;1;0) in column H.
☐3
Predict: will the rule score better on Training or on Testing? Write it in J1.
☐4
Add up both. Write each score as right out of how many.

Mission 2: A model that memorised (10 min)

☐5
Now build a cheat. In Training, column I: =VLOOKUP(B2;$Training.$B$2:$F$29;5;0). It finds a training apple with exactly the same weight, and copies its variety. Fill down and score it. It does very well. Why?
☐6
Put the same formula in Testing, column I. Fill down. What happens?
☐7
Fill in the table in J5:

Training score Testing score
The model that learned
The model that memorised

Mission 3: The sign at the exhibition (3 min)

☐8
In J10, rewrite the exhibition sign so that it is honest. Keep it under 15 words.

Finished early?

Move 4 apples from Training to Testing and score both models again. Does a bigger testing set change your trust in the scores?

No laptop today?

Now you know

Evaluation is the model's report card. Metrics tell you how well it works, so you can improve it — and know how far to trust it before anyone relies on it.

The train-test split. Learn from the training set. Measure on a testing set the model has never seen. That score estimates how it will do on new data.

Never test on training data. A model can remember the training set and answer it perfectly. That is overfitting: memory that looks like skill.

The training score can't tell them apart. Both models scored 27 out of 28 on training. Only the test showed which one had learned.

The honest number is the test number, even when it is lower — and it usually is.

Check yourself — practice, not a test

1. Why is the testing set kept away from the model during training?

○ It is too small to learn from
○ So its score shows how the model does on data it has never seen
○ Testing data has no labels at all
○ Models can only read one set at a time

2. A model that scores 100% on its own training data is certain to work well on new data.

○ True
○ False

3. When a model remembers its training data instead of learning a pattern, it is called ______.

4. Put the train-test procedure in order:

1. Split the labelled data into training and testing sets
2. Compare its predictions with the true labels
3. Train the model on the training set
4. Ask the model to predict the testing set

5. CBSE compares model evaluation to:

○ A driving licence
○ A report card
○ A recipe
○ A map