Detection of Labelling Issue in CIFAR-10 Dataset using RagaAI Platform

CIFAR-10 is a popular image classification dataset that consists of 60,000 32x32 color images categorized into 10 different classes, with 6,000 images per class. It is commonly used for benchmarking image classification models.

Labelling Inconsistency in the RagaAI Platform

RagaAI is a powerful platform for testing and evaluating AI applications. One of the features of RagaAI is its ability to detect labelling inconsistencies in datasets. Labelling inconsistency refers to inconsistencies or errors in the labelling of data points in a dataset. It can occur when different annotators have different interpretations or when data points are incorrectly labelled.

Using the RagaAI Platform to Detect Issues in CIFAR-10 Dataset

RagaAI's labelling inconsistency feature was used to detect and identify issues in the labelling of the CIFAR-10 dataset. By uploading the dataset to the platform, RagaAI automatically analyzed the labels assigned to each image and identified inconsistencies and errors in the labelling.

Images with labelling mismatch: 26

Example of Issue

For example, let's assume that in the CIFAR-10 dataset, some images that belong to the "cat" class have been incorrectly labelled as "dog". This inconsistency in the labelling can significantly impact the performance of machine learning models trained on this dataset. RagaAI can detect and highlight such inconsistencies, making it easier for data scientists to identify and correct the labelling errors.

The following image is labelled as ‘cat’ in the dataset, and the platform is detecting that the label is incorrect with a mistake score of 1:

Mistake Score:
A relative score between images of the dataset, which ranges from value 0 to 1. Values closer to 1, have a higher probability of the label being incorrect. 

Importance of Resolving Labelling Issues

Resolving labelling inconsistencies is crucial for increasing the performance and accuracy of machine learning models. When training models on datasets with accurate and consistent labels, the models can learn more effectively and make better predictions. By leveraging the labelling inconsistency feature of RagaAI, data scientists can identify and correct these issues, ensuring that models are trained on high-quality datasets.

This is part of the blog series in which we use the RagaAI Testing Platform on open-source data and models. In the next blog, we will talk about Failure Mode Analysis on Image Classification.