Agent Evaluation input schema

This article explains the input schema required by Agent Evaluation to assess your application’s quality, cost, and latency.

• During development, evaluation takes place offline, and an evaluation set is a required input to Agent Evaluation.
• When an application is in production, all inputs to Agent Evaluation come from your inference tables or production logs.

The input schema is identical for both online and offline evaluations.

For general information about evaluation sets, see Evaluation sets.

Evaluation input schema

The following table shows Agent Evaluation’s input schema. The last two columns of the table refer to how input is provided to the mlflow.evaluate() call. See How to provide input to an evaluation run for details.

expected_facts guidelines

The expected_facts field specifies the list of facts that is expected to appear in any correct model response for the specific input request. That is, a model response is deemed correct if it contains these facts, regardless of how the response is phrased.

Including only the required facts, and leaving out facts that are not strictly required in the answer, enables Agent Evaluation to provide a more robust signal on output quality.

You can specify at most one of expected_facts and expected_response. If you specify both, an error will be reported. Databricks recommends using expected_facts, as it is a more specific guideline that helps Agent Evaluation judge more effectively the quality of generated responses.

expected_response guidelines

The expected_response field contains a fully formed response that represents a reference for correct model responses. That is, a model response is deemed correct if it matches the information content in expected_response. In contrast, expected_facts lists only the facts that are required to appear in a correct response and is not a fully formed reference response.

Similar to expected_facts, expected_response should contain only the minimal set of facts that is required for a correct response. Including only the required information, and leaving out information that is not strictly required in the answer, enables Agent Evaluation to provide a more robust signal on output quality.

You can specify at most one of expected_facts and expected_response. If you specify both, an error will be reported. Databricks recommends using expected_facts, as it is a more specific guideline that helps Agent Evaluation judge more effectively the quality of generated responses.

Schema for request

The request schema can be one of the following:

• A messages field that follows the OpenAI chat completion schema and can encode the full conversation.
• A plain string. This format supports single-turn conversations only. Plain strings are converted to the messages format before being passed to your agent.
• A query string field for the most recent request and an optional history field that encodes previous turns of the conversation.

For multi-turn chat applications, use the second or third option above.

The following example shows all three options in the same request column of the evaluation dataset:

import pandas as pd

data = {
  "request": [

      # Plain string. Plain strings are transformed to the `messages` format before being passed to your agent.
      "What is the difference between reduceByKey and groupByKey in Spark?",

      # Using the `messages` field for a single- or multi-turn chat
      {
          "messages": [
              {
                  "role": "user",
                  "content": "How can you minimize data shuffling in Spark?"
              }
          ]
      },

      # Using the query and history fields for a single- or multi-turn chat
      {
          "query": "Explain broadcast variables in Spark. How do they enhance performance?",
          "history": [
              {
                  "role": "user",
                  "content": "What are broadcast variables?"
              },
              {
                  "role": "assistant",
                  "content": "Broadcast variables allow the programmer to keep a read-only variable cached on each machine."
              }
          ]
      }
  ],

  "expected_response": [
    "expected response for first question",
    "expected response for second question",
    "expected response for third question"
  ]
}

eval_dataset = pd.DataFrame(data)

Schema for arrays in evaluation input

The schema of the arrays expected_retrieved_context and retrieved_context is shown in the following table:

Computed metrics

The columns in the following table indicate the data included in the input, and ✓ indicates that the metric is supported when that data is provided.

For details about what these metrics measure, see How quality, cost, and latency are assessed by Agent Evaluation.