Queue Dataset Goldens

(v2)

POST

Adds goldens to the dataset as unfinalized goldens, for review on the platform before they are used in evaluations.

POST/v2/datasets/{datasetId}/queue

curl -X POST "https://api.confident-ai.com/v2/datasets/{datasetId}/queue" \
  -H "CONFIDENT_API_KEY: <PROJECT-API-KEY>" \
  -H "Content-Type: application/json" \
  -d '{
  "goldens": [
    {
      "input": "What is the capital of France?",
      "actualOutput": "The capital of France is Paris.",
      "expectedOutput": "Paris.",
      "context": [
        "Paris is the capital of France."
      ],
      "retrievalContext": [
        "Paris is the capital and largest city of France."
      ],
      "toolsCalled": [
        {
          "name": "get_landmark_info",
          "type": "FUNCTION",
          "description": "This tool gives information about a mountain.",
          "inputParameters": {
            "mountain": "Everest"
          },
          "output": "8,848 metres",
          "reasoning": "The user asked for the height of a mountain."
        }
      ],
      "expectedTools": [
        {
          "name": "get_landmark_info",
          "type": "FUNCTION",
          "description": "This tool gives information about a mountain.",
          "inputParameters": {
            "mountain": "Everest"
          },
          "output": "8,848 metres",
          "reasoning": "The user asked for the height of a mountain."
        }
      ],
      "tokenCost": 0.002,
      "inputTokenCount": 12,
      "outputTokenCount": 3,
      "additionalMetadata": {
        "source": "faq"
      },
      "comments": "Reviewed by the support team.",
      "sourceFile": "capitals.csv",
      "sourceFiles": [
        "capitals.csv"
      ],
      "finalized": true,
      "customColumnKeyValues": {
        "difficulty": "easy"
      },
      "imagesMapping": {
        "map": {
          "url": "https://example.com/paris.png",
          "local": false
        }
      },
      "tags": [
        "geography"
      ]
    }
  ]
}'

200

{
  "success": true,
  "data": {
    "id": "<DATASET-ID>"
  },
  "link": "https://app.confident-ai.com/project/<PROJECT-ID>/datasets/<DATASET-ID>",
  "deprecated": false
}

Headers

CONFIDENT_API_KEYstringRequired
The API key of your Confident AI project.

Path parameters

datasetIdstringRequired
The unique id of the dataset.

Request body

goldenslist of object | objectRequired
The goldens to queue for review. Every golden in one request must be of the same kind and match the dataset's multiTurn. They are stored unfinalized, whatever each golden's own finalized says.
Show 2 variantsHide 2 variants
- Single-Turn Golden Requestobject
  A single-turn golden to write: one input to your LLM application and the outputs expected of it.
  Show 18 propertiesHide 18 properties
  - inputstringRequired
    This is the input to your LLM application.
  - actualOutputstring | null
    This is the actual output of your LLM application.
  - expectedOutputstring | null
    This is the expected output of your LLM application, which is the ideal actual output.
  - contextarray | null
    This is the ideal retrieval context of your LLM application.
  - retrievalContextarray | null
    This is the retrieval context of your LLM application.
  - toolsCalledarray | null
    This is the tools called by your LLM application.
    Show 6 propertiesHide 6 properties
    namestringRequired
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
  - expectedToolsarray | null
    This is the expected tools to be called by the LLM application.
    Show 6 propertiesHide 6 properties
    namestringRequired
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
  - tokenCostnumber | null
    This is the cost of the tokens used to produce the actual output.
  - inputTokenCountinteger | null
    This is the number of input tokens passed to the LLM model.
  - outputTokenCountinteger | null
    This is the number of output tokens generated by the LLM model.
  - additionalMetadataobject | null
    Additional metadata to associate with the golden.
  - commentsstring | null
    Comments to associate with the golden.
  - sourceFilestring | null
    The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.
  - sourceFileslist of strings
    The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.
  - finalizedboolean
    Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.
  - customColumnKeyValuesobject
    Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.
  - imagesMappingobject
    The media this golden refers to, keyed by the id inside each placeholder. Put [DEEPEVAL:IMAGE:<id>] or [DEEPEVAL:PDF:<id>] in a text field where the media belongs, and the platform substitutes the entry with a matching key.
  - tagslist of strings
    Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.
- OR
- Multi-Turn Golden Requestobject
  A multi-turn golden to write: the scenario of a conversation with your LLM application and, optionally, its turns.
  Show 13 propertiesHide 13 properties
  - scenariostringRequired
    This is a description of the conversation context.
  - expectedOutcomestring | null
    This describes the expected outcome, or ideal conversation flow, of the conversation.
  - userDescriptionstring | null
    This is the description of the user in the conversation.
  - turnsarray | null
    This is the list of turns in the conversation.
    Show 6 propertiesHide 6 properties
    idstring
    The id of a turn assigned by Confident AI.
    roleenumRequired
    The role of the turn, either user or assistant.
    Show 2 enum valuesHide 2 enum values
    user
    assistant
    contentstringRequired
    The message content of the turn.
    userIdstring
    The user ID associated with the turn.
    retrievalContextarray | null
    The contexts retrieved to generate the LLM response for this turn.
    toolsCalledarray | null
    The tools called to generate the LLM response for this turn.
    Show 6 propertiesHide 6 properties
    namestringRequired
    This is the name of the tool.
    typeenum
    The type of the tool call, either a function or an MCP tool.
    Show 2 enum valuesHide 2 enum values
    FUNCTION
    MCP
    descriptionstring
    This is the description of the tool.
    inputParametersobject | null
    This is the input parameters that are passed to the tool.
    outputany
    This is the output of the tool.
    reasoningstring
    This is the reasoning your LLM provided for the tool call.
  - contextarray | null
    This is the context of the conversation.
  - additionalMetadataobject | null
    Additional metadata to associate with the golden.
  - commentsstring | null
    Comments to associate with the golden.
  - sourceFilestring | null
    The source file the golden was retrieved from. Like tags and customColumnKeyValues, this is left unchanged when the request omits it; send null to clear it.
  - sourceFileslist of strings
    The source files the golden was retrieved from. Like tags and customColumnKeyValues, these are left unchanged when the request omits them; send an empty array to clear them.
  - finalizedboolean
    Whether the golden is ready to use in evaluations. When pushing or queueing a list of goldens the request decides this for every golden and this field is ignored.
  - customColumnKeyValuesobject
    Custom dataset column values keyed by column name. A column that does not exist in the dataset yet is created.
  - imagesMappingobject
    The media this golden refers to, keyed by the id inside each placeholder. Put [DEEPEVAL:IMAGE:<id>] or [DEEPEVAL:PDF:<id>] in a text field where the media belongs, and the platform substitutes the entry with a matching key.
  - tagslist of strings
    Tags to associate with the golden, which is useful for grouping and filtering goldens. A tag that does not exist in the dataset yet is created.

Response

Queue Dataset Goldens succeeded.

successboolean
Indicates if the request was successful.
dataobject
Show 1 propertyHide 1 property
- idstring
  This is the unique id of the dataset.
linkstring
This is the URL of the resource on the Confident AI platform.
deprecatedboolean
Indicates if this endpoint is deprecated.

Built byConfident AI