Unit 3: Evaluating Models
Evaluation is the stage used to check how well an AI model performs. It helps us compare models, understand their strengths and weaknesses, and improve them using feedback from metrics.
Model Evaluation and Its Importance
Model evaluation is the process of using different evaluation metrics to understand the performance of a machine learning model. It is an important part of model development because it helps us find the best model for the data and understand how well the chosen model may work in the future.
An AI model improves through constructive feedback. A model is built, its performance is checked using metrics, improvements are made, and the process continues until a desirable accuracy is reached.
Model evaluation works like a report card. A report card uses grades, percentage, percentiles, and ranks to show academic performance. In the same way, evaluation metrics show where an AI model is doing well and where it needs improvement.
Model evaluation helps in understanding:
- The strengths of the AI model
- The weaknesses of the AI model
- The suitability of the model for the task
- The reliability of the model for use in the future
Different evaluation techniques can be used depending on the type and purpose of the model.
Everyday example: If two students prepare answers for the same task, checking their work helps decide which answer is better and what needs improvement.
Train-Test Split for Evaluation
Train-test split is a technique used to evaluate the performance of a machine learning algorithm. It can be used for any supervised learning algorithm.
In train-test split, a dataset is divided into two subsets:
- The training dataset
- The testing dataset
The train-test procedure is suitable when a sufficiently large dataset is available.
The training dataset is used to make the model learn. After training, the input elements of the test dataset are given to the trained model. The model makes predictions. These predicted values are then compared with the expected values.
The objective of train-test split is to estimate how the machine learning model performs on new data. New data means data that was not used to train the model.
This is close to how a model is expected to work in practice. The model is fitted on available data with known inputs and outputs. Later, it makes predictions on new examples where the expected output or target values are not available.
It is not recommended to evaluate a model using the same data that was used to build it. If the same training data is used, the model may simply remember the whole training set. It may then always predict the correct label for any point in the training set. This is known as overfitting.
Everyday example: A student should not practise only the exact same sums and then be judged only on those same sums, because remembering them does not show how well the student can solve new sums.
Accuracy and Error
Accuracy is an evaluation metric that measures the total number of predictions a model gets right. The accuracy of the model and the performance of the model are directly proportional. This means that when the model performs better, its predictions are more accurate.
Error is an action that is inaccurate or wrong. In Machine Learning, error is used to see how accurately a model can predict data it uses to learn new, unseen data. Based on error, the machine learning model that performs best for a particular dataset is chosen.
Error refers to the difference between a model's prediction and the actual outcome. It shows how often the model makes mistakes.
The Model Evaluation stands on the two pillars of accuracy and error.
Important formulas:
- Absolute error = Abs(Actual − Predicted)
- Error rate = Error / Actual
- Accuracy = 1 − Error rate
- Accuracy% = (Accuracy * 100)%
Abs means the absolute value. It means only the magnitude of the difference, without any negative sign.
The goal is to minimize error and maximize accuracy. Real-world data can be messy, and even the best models make mistakes. Sometimes focusing only on accuracy may not be ideal. In medical diagnosis, a model with slightly lower accuracy but a strong focus on avoiding incorrectly identifying a healthy person as sick might be preferable.
Choosing the right error or accuracy metric depends on the specific task and its requirements. Understanding both error and accuracy is important for evaluating and improving AI models.
Everyday example: If the actual price of a ticket is Rs 500 and one person guesses Rs 550, the difference shows the error in the estimate.
Classification
Classification refers to a problem where a specific type of class label is the result to be predicted from the given input field of data.
A simple way to understand classification is the supermarket example. Items can be placed into two classes:
- fruits and vegetables
- grocery
A vegetable-grocery-classifier model predicts whether an item in the supermarket is a vegetable or a grocery item.
Classification can have two classes or more than two classes. In classification, the model predicts the class label for the given input.
Popular metrics used for classification model are:
- Confusion matrix
- Classification accuracy
- Precision
- Recall
F1 Score is also used to combine precision and recall into a single measure.
Everyday example: Sorting notebooks into two piles, completed and not completed, is a simple classification task.
Confusion Matrix
A confusion matrix is a handy presentation of the accuracy of a model with two or more classes. It is used to understand the predictions made by a classification model.
In a confusion matrix:
- The actual values are shown on the y-axis
- The predicted values are shown on the x-axis
- Each cell shows the number of predictions made by a machine learning algorithm that fall into that category
For example, in a disease classifier, the output can be 1 if the person is infected and 0 if the person is not infected. The model can predict 0 or 1, and each prediction may actually be 0 or 1.
The confusion matrix has four important outcomes:
- True Positive
- True Negative
- False Positive
- False Negative
True Positive (TP) is the outcome of the model correctly predicting the positive class. Any class can be assumed as a positive class, and the rest can be assumed as negative. If class 1 is assumed as the positive class, then correctly predicting class 1 is a True Positive.
True Negative (TN) is the outcome of the model correctly predicting the negative class. If class 1 is assumed as the positive class, class 0 is assumed as the negative class. Correctly predicting class 0 is a True Negative.
False Positive (FP) is the outcome of the model wrongly predicting the negative class as positive class. When class 0 is predicted as class 1, it falls into the FP cell.
False Negative (FN) is the outcome of the model wrongly predicting the positive class as the negative class. When class 1 is predicted as class 0, it falls into the FN cell.
Examples from the disease classifier:
- True Positive: The model predicts Yes, and the person does have the disease
- True Negative: The model predicts No, and the person does not have the disease
- False Positive: The model predicts Yes, but the person does not have the disease
- False Negative: The model predicts No, but the person does have the disease
Everyday example: If a result is predicted as pass or not pass, the confusion matrix helps count correct and wrong predictions for both classes.
Classification Accuracy from Confusion Matrix
Classification accuracy is the number of correct predictions made as a ratio of all predictions made.
The correct predictions are:
- True Positive
- True Negative
The total predictions include:
- True Positive
- True Negative
- False Positive
- False Negative
Formula:
- Accuracy = (TP + TN) / (TP + TN + FP + FN)
Accuracy is useful, but it cannot be used all the time. It is suitable when there are an equal number of observations in each class, which means a balanced dataset. It also assumes that all predictions and prediction errors are equally important, which is often not the case.
In an unbalanced dataset, accuracy can give a misleading idea of model performance. For example, if a model predicts only one class again and again, it may still show high accuracy when most test data belongs to that class. This does not mean the model is good for the task.
In cases of unbalanced data, other metrics such as Precision, Recall or F1 score should be used.
Everyday example: If most items in a box are red, a model that always says red may look accurate, but it is not really identifying the other items well.
Precision
Precision is the ratio of the total number of correctly classified positive examples and the total number of predicted positive examples.
Formula:
- Precision = TP / (TP + FP)
Precision tells us how often the model is correct when it predicts the positive class. For example, if a model predicts that a patient has heart disease, precision tells how often that prediction is correct.
Precision is generally used for unbalanced datasets when dealing with False Positives becomes important. It is used when the model needs to reduce the FPs as much as possible.
A use case given is predicting a good day based on weather conditions to launch a satellite. A day with favorable weather condition is considered the Positive class. A day with non-favorable weather condition is considered the Negative class. Missing a good weather day is acceptable, but predicting a bad weather day as a good weather day can be disastrous. So, False Positives need to be reduced as much as possible.
Everyday example: If a box is labelled as containing fresh fruits, precision focuses on how many items in that labelled box are actually fresh fruits.
Recall
Recall is the measure of a model correctly identifying True Positives. For all patients who actually have heart disease, recall tells how many were correctly identified as having heart disease.
Recall is also called Sensitivity or True Positive Rate.
Formula:
- Recall = TP / (TP + FN)
Recall is generally used for unbalanced dataset when dealing with False Negatives becomes important. It is used when the model needs to reduce the FNs as much as possible.
A use case given is a covid-19 prediction classifier. Detection of a covid-19 affected case is considered the positive class. Detection of a covid-19 non-affected case is considered the negative class. If a covid-19 affected person is falsely predicted as non-affected, the person may not get treatment and may also infect other persons. So, False Negatives need to be reduced as much as possible.
Another given use case is flagging fraudulent transactions. It is acceptable to classify a legit transaction as fraudulent because it can be re-verified through additional checks. It is not acceptable to classify a fraudulent transaction as legit. So, False Negatives should be reduced as much as possible, and Recall is more important.
Everyday example: If the task is to find all broken chairs in a room, recall focuses on finding as many broken chairs as possible.
F1 Score
F1 Score provides a way to combine both precision and recall into a single measure. It captures both properties.
Formula:
- F1 Score = (2 x Precision x Recall) / (Precision + Recall)
F1 Score is useful in cases where the dataset is unbalanced and it is difficult to decide whether FP is more important or FN. In such use cases, F1 Score should be used as the suitable metric.
F1 Score helps when both precision and recall matter together.
Everyday example: If a class list must include the correct students and also avoid missing students, one combined check is helpful.
Choosing the Right Metric
Different evaluation metrics are used in different scenarios. The suitable metric depends on the dataset and the type of error that matters more in the situation.
Accuracy is suitable when:
- The dataset is balanced
- All predictions are equally important
- All prediction errors are equally important
Precision is suitable when:
- The dataset is unbalanced
- False Positives are important
- The model needs to reduce FPs as much as possible
Recall is suitable when:
- The dataset is unbalanced
- False Negatives are important
- The model needs to reduce FNs as much as possible
F1 Score is suitable when:
- The dataset is unbalanced
- It is not clear whether FP or FN is more important
- Precision and recall need to be combined into a single measure
For a satellite launch weather prediction model, Precision is important because predicting a bad weather day as a good weather day can be disastrous.
For a covid-19 prediction classifier, Recall is important because predicting an affected person as non-affected can lead to no treatment and infection of other persons.
For a fraudulent transaction detection model, Recall is important because classifying a fraudulent transaction as legit is not acceptable.
Everyday example: If the task is checking permission slips, the metric should match what matters more, wrongly accepting a slip or wrongly rejecting one.
Ethical Concerns Around Model Evaluation
While evaluating an AI model, the following ethical concerns need to be kept in mind:
- Bias
- Transparency
- Accuracy
About these notes
These notes and questions were written by Syllabax from the official CBSE Class X Artificial Intelligence (417) Facilitator Handbook and the 2026-27 curriculum. This is not an official CBSE publication.