Basics

How to specify the dataset to test on

Testing LLM apps would be incomplete without the scenarios to test on. You can specify these scenarios in your test dataset.

Each dataset consists of samples. Each sample has

• id (string; optional): Unique identifier for the sample
• inputs (object): Input parameters that define a scenario. The key is the name of the parameter, and the value is the value of the parameter for the sample
• expected (string; optional): Used as the ground truth for this sample

Datasets can be specified in the configuration file directly, or imported from a file or HTTP endpoint.

Specify dataset directly

In this example, the LLM is asked to extract the user’s name from an incoming message. The message is provided under the user_message parameter in the dataset.

The LLM prompt can use this value in the prompt through the {{user_message}} placeholder.

"dataset": {
  "samples": [
    {
      "inputs": {
        "user_message": "Hi my name is John Doe."
      },
      "expected": "John Doe"
    },
    {
      "inputs": {
        "user_message": "This is Alice. Is anybody here?"
      },
      "expected": "Alice"
    }
  ]
}

Import from JSONL file

Specify a path to the JSONL file. Each line of the file should be a valid JSON object. On import, the keys of this JSON will be converted into inputs of the sample.

If using relative paths, the path is treated relative to the configuration file.

"dataset": {
  "path": "HumanEval.jsonl"
}

Import from CSV

Specify a path to the CSV file in the empiricalrc.json. If using relative paths, the path is treated relative to the configuration file.

"dataset": {
  "path": "foo.csv"
}

The CSV file should contain headers. The lines of the file are converted into dataset inputs with column header names as the name of the parameter. For example:

foo.csv

name,age
John,25

The above CSV gets converted into the following dataset object:

"dataset": {
  "samples": [
    {
      "inputs": {
        "name": "John",
        "age": "25"
      }
    }
  ]
}

The above conversion enables you to create a prompt with placeholders. For example:

{
  "prompt": "Your name is {{name}} and you are a helpful assistant..."
}

Import from Google Sheets

Specify a path to the Google sheet in the empiricalrc.json file.

empiricalrc.json

"dataset": {
  "path": "https://docs.google.com/spreadsheets/d/1AsMekKCG74m1PbBZQN_sEJgaW0b9Xarg4ms4mhG3i5k"
}

Refer to our chatbot example which uses this dataset.

The sheet should contain column headers. The rows of the file are converted into dataset inputs with column header names as the name of the parameter. For example:

| name | age |
| ---- | --- |
| John | 25  |

The above table in the sheet gets converted into the following dataset object:

"dataset": {
  "samples": [
    {
      "inputs": {
        "name": "John",
        "age": "25"
      }
    }
  ]
}

The above conversion enables you to create prompt with placeholders. For example:

empiricalrc.json

{
  "prompt": "Your name is {{name}} and you are a helpful assistant..."
}

If you wish to extract data from a specific sheet of Google Sheet, make sure to navigate to the desired sheet and copy the browser URL into empiricalrc.json.

Import Empirical JSON format

If your dataset follows the Empirical JSON format, you can import that from a file or HTTP endpoint.

"dataset": {
  "path": "https://assets.empirical.run/datasets/json/spider-tiny.json"
}