Using RagaAI Catalyst to Evaluate LLM Applications

Wondering what RagaAI Catalyst is and how it can elevate your AI development? Dive into this comprehensive beginner's guide to Catalyst, where we’ll explore what it offers, how it functions, and how you can get started with its cutting-edge features to prototype and test language models with ease.

The world of LLMs is rapidly evolving, with chatbots, virtual assistants, content generators, and language understanding systems transforming the way we interact with technology. RagaAI’s Catalyst equips you with the tools to navigate this dynamic landscape, ensuring your models are not just functional, but truly exceptional. From stress-testing for robustness to pinpointing areas for improvement, we've got you covered every step of the way.

What is RagaAI Catalyst?

RagaAI Catalyst offers an automated Test and Fix platform for LLM applications - ranging from LLMs and RAGs to Agentic Applications. With its state of the art automated metrics, it helps identify hallucinations, safety and security vulnerabilities as well as cost concerns with GenAI applications. It empowers data science and developer teams with the tools and recommendations to identify issues in real-time and address them seamlessly.

Key Features -

1. Comprehensive and Actionable Metrics: RagaAI Catalyst excels in providing an unparalleled set of comprehensive and actionable metrics for LLM evaluation from applications like Chatbots, Code Generation to Marketing Content Generation.

2. SOTA Evaluation: RagaAI Catalyst offers users state-of-the-art LLM evaluation methods, with an industry leading 93% alignment with human evaluation out of the box. This involves detecting issues with all parts of an application, be it prompt quality, context precision, context recall, hallucinations, response completeness and others.

3. Safety and Security: Ensuring GenAI safety and security in production environments is a constant challenge. RagaAI offers proprietary guardrails to identify safety issues like PII (Personally Identifiable Information) leakage, toxicity, biased outputs as well as adversarial attacks like prompt injection optimised for real time inference.

How Does it Work?

RagaAI Catalyst integrates right into your LLMOps workflow, allowing for both - offline and online testing capabilities. The former allows users to upload existing data (CSV, Pandas dataframe, etc.) easily from their personal systems, while the latter allows for real-time logging of inferences by enabling Catalyst’s Tracer function from the users’ Python environments.

In doing so, Catalyst enables all levels of users and enterprises to gain confidence by seeing results on their own data, and get to deployment faster by eliminating the need to worry about real-time security and reliability issues.

Get Started with RagaAI Catalyst: A Quick Tutorial

This guide will help you get started with RagaAI Catalyst using any sample dataset, straight from your Python environment.

Steps to Use RagaAI Catalyst's Public Sandbox

The above video demonstrates all the steps required to perform your first evaluation on the Catalyst software. All commands shown in the video have been documented below.

1. Sign Up and Authentication

1.1 Sign Up

• Start by signing up for an account at https://catalyst.raga.ai/

1.2 Install Python Package

pip install ragaai_catalyst -U

1.3 Authenticate Using Keys

• Authenticate using your access key and secret key:

import os
from ragaai_catalyst import RagaAICatalyst, Experiment, Tracer, Dataset

catalyst = RagaAICatalyst(
    access_key="YOUR_ACCESS_KEY",
    secret_key="YOUR_SECRET_KEY",
    base_url="https://catalyst.raga.ai/api"
)

• You can find Catalyst Access & Secret Keys in the UI by navigating to Settings -> Authenticate and use the "Generate New Key" button to create a fresh pair:

• A valid API key will be required to run evaluations on the public sandbox. You can enter your API key on the Catalyst platform as follows:

2. Create New Project

• Create a new project for testing your LLM Application:

new_project = catalyst.create_project(
    project_name="your_project_name",
    description="Your project description"
)

User can also create the project via UI by clicking the "Create New Project" button on the platform homepage:

• List your projects:

projects = catalyst.list_projects(num_projects=2)
print(f"Projects: {projects}")

• Note: 'num_projects' is optional and is used to list the latest 'n' projects (here, 2). Passing list_projects() without any arguments will list all the projects available.

3. Upload Dataset

RagaAI Catalyst enables users to ingest data in two broad ways: real-time tracing and static uploads of data (CSV, Pandas, etc.).

3.1 Upload Dataset via CSV

Once your project is created, you can upload datasets to it. Here's how you can upload a dataset via a CSV file:

Create New Dataset in a Project

1. Select your newly created project from the project list.

2. Navigate to the Dataset tab within the project.

3. Click on the Create New Dataset button.

4. In the upload window, select the Upload CSV tab.

5. Click on the upload area and browse/drag and drop your local CSV file. Ensure the file size does not exceed 1GB.

6. Enter a suitable name for your dataset.

7. Click Next to proceed.

Next, you will be directed to map your dataset schema with Catalyst's inbuilt schema, so that your column headings don't require editing:

Set Dataset Schema

Here is a list of Catalyst's inbuilt schema elements (definitions are for reference purposes and may vary slightly based on your use case):

4. Create Experiments and Run Metrics

• Set up an experiment and add metrics for evaluation:

from ragaai_catalyst import Experiment

experiment_manager = Experiment(
    project_name="your_project_name",
    experiment_name="Your Experiment Name",
    experiment_description="Experiment description",
    dataset_name="your_dataset_name"
)

response = experiment_manager.add_metrics(
    metrics=[
        {"name": "Hallucination", "config": {"model": "gpt-4o-mini", "reason": True, "provider": "OpenAI"}},
        {"name": "Faithfulness", "config": {"model": "azure/azure-gpt-4o-mini", "reason": True, "provider": "Azure"}}
    ]
)

print("Metric Response:", response)

• Check out the list of supported metrics [Learn More].

• You can view the experiment on the platform.

View Experiments

Users can view completed experiments by navigating to the Catalyst project screen and clicking on the name of the project they created. Users can click on “View Result” against the experiment they wish to analyse further. This will reveal project and inference level analyses as shown below:

Users can drill down into individual prompts to see more granular information like reasoning for each metric calculated, documents fetched for context, etc.:

Conclusion

As you embark on your journey with our LLM testing software, remember that success in AI development often hinges on experimentation and iteration. Every experiment, every test case, every insight you uncover helps push the boundaries of what's possible.

Our platform is here to support you every step of the way, offering powerful tools to analyse and refine your models. By leveraging our intuitive interface and real-time feedback capabilities, you'll be well-equipped to tackle complex challenges and push the boundaries of what your models can achieve. We encourage you to explore, test, and innovate—your next breakthrough is just a few clicks away. Welcome aboard, and happy testing!