VLM – Agentic Datasets

Instruction Datasets Retrieval Datasets Ximilar Platform

This page is part of the Custom Vision Language Models (VLM) API reference. See the overview for task modes, key concepts, the end-to-end workflow and the list of all endpoints.

Agentic Datasets

Agentic datasets teach the model to solve a task over several turns: look at the images, think, call one of your tools, read the tool result and finally answer. Each sample is a trajectory, an ordered list of steps. Tools are defined once per workspace and attached to the datasets that may use them.

Building an agentic sample

Create the dataset with mode: agentic and a system prompt (the task also needs a user prompt).
Create the tools the model may call and attach them to the dataset.
Create a sample, then create its steps in order: the user question, assistant thoughts and tool calls, tool results, and the final answer.
Attach images to the user step (max 10 images and 10 detection objects per step).
Validate the sample.

Step Roles and Types

Every step has a role (who speaks) and a type (what kind of message it is). Only these combinations are allowed:

`role`	allowed `type`	`content`
`system`	`text`	string; only before the first user step
`user`	`text`	string, the question; images are attached to this step
`assistant`	`thought`	string, internal reasoning shown before a tool call or answer
`assistant`	`tool_call`	object `{"name": "<tool name>", "arguments": {...}}`; requires `tool` and `tool_call_id`
`assistant`	`answer`	string or object, the final answer of the turn
`tool`	`tool_result`	any JSON, the output of the tool; requires `tool` and the matching `tool_call_id`; `is_error: true` marks a failed call

order is the position of the turn in the conversation. Steps that share the same order form one logical turn and are sequenced by sub_order (used for parallel tool calls). Simplest approach: give every step its own order (1, 2, 3, ...) and leave sub_order at 0.

Trajectory Rules

A sample is valid when its steps form this sequence:

optional system steps
a user step (with at least one image, detection object or media attachment)
optional assistant / thought
either assistant / answer (turn complete) or assistant / tool_call followed by the matching tool / tool_result (same tool_call_id), then back to step 3
optionally another user step after an answer (multi-turn), repeating from step 2
the trajectory must end with an assistant / answer

Typical validation errors (error_type: reason):

empty_agentic_steps: Agentic sample has no steps.
invalid_start: Agentic trajectory must start with optional system step(s), then a user step.
forbidden_assistant_text: Assistant type='text' is ambiguous; use thought, tool_call, or answer.
tool_response_without_tool_call: Tool response has no preceding open tool call.
tool_response_id_mismatch: Tool response id 'call_2' does not match open tool call 'call_1'.
dangling_thought: Assistant thought is only allowed before a tool call or answer.
ends_after_user / ends_after_tool_response: the trajectory has no final answer.
invalid_tool_call: Tool call has no tool name / Tool-call arguments must be a JSON object.
media_role_not_allowed: Media attachments are only supported on user and tool-result steps.

GET/v2/tool/

List Tools

List the tool definitions in your workspace, optionally only those attached to a dataset.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.

Optional attributes

Name
dataset
Type
string
Description
Only tools attached to the given dataset UUID.
Name
search
Type
string
Description
Search tools by name.

Request

GET

/v2/tool/

curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     'https://api.ximilar.com/vlm/v2/tool/?dataset=__DATASET_ID__'

Response

{
  "count": 1,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": "3c1f6a2b-9d4e-4f70-8b21-5e6a7c8d9e0f",
      "name": "check_db",
      "description": "Check the card database for a record by barcode.",
      "parameters_schema": {
        "type": "object",
        "properties": {"query": {"type": "string"}},
        "required": ["query"]
      },
      "datasets": ["8797c273-b1d3-4e6f-82bb-adfb719415fe"],
      "datasets_count": 1,
      "workspace": "748e50e4-d081-4924-b9e7-f500aac6a71d"
    }
  ]
}

POST/v2/tool/

Create Tool

Define a tool the model can call. The description tells the model when to use the tool, parameters_schema is a JSON Schema of the arguments. GET, PATCH and DELETE /v2/tool/{tool_id}/ read, update and delete a tool.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
name
Type
string
Description
Tool name, unique within the workspace (letters, numbers and underscores, not starting with a number). Used as name in tool calls.
Name
description
Type
string
Description
What the tool does and when the model should call it.
Name
parameters_schema
Type
object
Description
JSON Schema describing the arguments object.

Optional attributes

Name
datasets
Type
array
Description
UUIDs of agentic datasets that may use this tool. Can also be set later with Attach Tool to Datasets.

Request

POST

/v2/tool/

curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{
       "name": "check_db",
       "description": "Check the card database for a record by barcode.",
       "parameters_schema": {
         "type": "object",
         "properties": {"query": {"type": "string", "description": "Search query, e.g. barcode:501234567890"}},
         "required": ["query"]
       },
       "datasets": ["__DATASET_ID__"]
     }' \
     https://api.ximilar.com/vlm/v2/tool/

Response

{
  "id": "3c1f6a2b-9d4e-4f70-8b21-5e6a7c8d9e0f",
  "name": "check_db",
  "description": "Check the card database for a record by barcode.",
  "parameters_schema": {
    "type": "object",
    "properties": {"query": {"type": "string", "description": "Search query, e.g. barcode:501234567890"}},
    "required": ["query"]
  },
  "datasets": ["8797c273-b1d3-4e6f-82bb-adfb719415fe"],
  "datasets_count": 1,
  "workspace": "748e50e4-d081-4924-b9e7-f500aac6a71d"
}

POST/v2/tool/{tool_id}/add-datasets/

Attach Tool to Datasets

Allow the samples of the given datasets to call this tool. A step can only reference a tool that is attached to the sample's dataset. POST /v2/tool/{tool_id}/remove-datasets/ with the same body detaches it.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
tool_id
Type
string
Description
UUID of the tool.
Name
dataset_ids
Type
array
Description
List of dataset UUIDs.

Request

POST

/v2/tool/{tool_id}/add-datasets/

curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{"dataset_ids": ["__DATASET_ID__"]}' \
     https://api.ximilar.com/vlm/v2/tool/__TOOL_ID__/add-datasets/

Response

{
  "added": 1
}

GET/v2/step/

List Steps

List the steps of a sample in conversation order (order, then sub_order). Returns paginated results.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.

Optional attributes

Name
sample
Type
string
Description
UUID of the sample whose steps to list (recommended).
Name
search
Type
string
Description
Search by sample name.
Name
page_size
Type
integer
Description
Number of results per page.

Request

GET

/v2/step/

curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     'https://api.ximilar.com/vlm/v2/step/?sample=__SAMPLE_ID__&page_size=100'

POST/v2/step/

Create Step

Add one step to an agentic sample. Create the steps in conversation order and give each its own order. GET, PATCH and DELETE /v2/step/{step_id}/ read, update and delete a step (the sample cannot be changed).

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
sample
Type
string
Description
UUID of the sample (must belong to an agentic dataset).
Name
role
Type
string
Description
system, user, assistant or tool.
Name
type
Type
string
Description
text, thought, tool_call, tool_result or answer (see the allowed combinations above).

Optional attributes

Name
content
Type
any
Description
Message text (string) or, for tool_call and tool_result, a JSON object.
Name
tool
Type
string
Description
UUID of the tool for tool_call and tool_result steps. The tool must be attached to the sample's dataset.
Name
tool_call_id
Type
string
Description
Identifier linking a tool_call to its tool_result, e.g. call_1.
Name
is_error
Type
boolean
Description
For tool_result: whether the tool execution failed (default false).
Name
order
Type
integer
Description
Position of the turn in the conversation (default 0).
Name
sub_order
Type
integer
Description
Position within the turn (default 0), for parallel tool calls.

Returns

The step with the read-only fields tool_name, images_count, detection_objects_count, media_count, step_images and step_detection_objects.

Errors

{"sample": "Sample steps are only valid for agentic datasets."}
{"tool": "Tool must be attached to the sample's dataset."}
{"tool": "Tool must belong to the same workspace as the sample."}

Request

POST

/v2/step/

curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{
       "sample": "__SAMPLE_ID__",
       "role": "user",
       "type": "text",
       "content": "Is this card real?",
       "order": 1
     }' \
     https://api.ximilar.com/vlm/v2/step/

Response

{
  "id": "5d6e7f80-91a2-4b3c-8d4e-5f6a7b8c9d0e",
  "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
  "role": "assistant",
  "type": "tool_call",
  "content": {"name": "check_db", "arguments": {"query": "barcode:501234567890"}},
  "tool": "3c1f6a2b-9d4e-4f70-8b21-5e6a7c8d9e0f",
  "tool_name": "check_db",
  "tool_call_id": "call_1",
  "is_error": false,
  "images_count": 0,
  "detection_objects_count": 0,
  "media_count": 0,
  "step_images": [],
  "step_detection_objects": [],
  "order": 3,
  "sub_order": 0
}

PATCH/v2/step/{step_id}/

Update Step

Change any field of a step except sample. Use it to fix content, reorder steps or link a tool call to its result.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
step_id
Type
string
Description
UUID of the step.

Optional attributes

Any of role, type, content, tool, tool_call_id, is_error, order, sub_order.

Request

PATCH

/v2/step/{step_id}/

curl -v -XPATCH \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{"content": {"found": false}, "is_error": true}' \
     https://api.ximilar.com/vlm/v2/step/__STEP_ID__/

POST/v2/step/{step_id}/add-images/

Add Images to Step

Attach images to a step. In agentic samples images live on the steps (usually the user step), not on the sample. A step can hold at most 10 images. POST /v2/step/{step_id}/remove-images/ detaches them.

Detection objects work the same way with add-detection-objects/, remove-detection-objects/ and a body {"detection_object_ids": [...]}.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
step_id
Type
string
Description
UUID of the step.
Name
image_ids
Type
array
Description
List of image UUIDs to add.

Errors

Cannot add 3 images. Step already has 8/10 images.
Some images were not found or not accessible in this workspace.

Request

POST

/v2/step/{step_id}/add-images/

curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{"image_ids": ["__IMAGE_ID_1__", "__IMAGE_ID_2__"]}' \
     https://api.ximilar.com/vlm/v2/step/__STEP_ID__/add-images/

Response

{
  "added": 2
}

GET/v2/step/{step_id}/step-images/

List Step Images

List the images attached to a step in order. PATCH /v2/step/{step_id}/step-images/{step_image_id}/ with text and/or order updates one attached image. Detection objects are listed with GET /v2/step/{step_id}/step-detection-objects/ and updated with PATCH .../step-detection-objects/{step_object_id}/.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
step_id
Type
string
Description
UUID of the step.

Returns (each item)

Name
id
Type
string
Description
UUID of the step-image relation.
Name
image_id
Type
string
Description
UUID of the image.
Name
img_path
Type
string
Description
Full URL of the image.
Name
text
Type
string
Description
Optional text label shown to the model before this image.
Name
order
Type
integer
Description
Order of the image within the step.

Request

GET

/v2/step/{step_id}/step-images/

curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     https://api.ximilar.com/vlm/v2/step/__STEP_ID__/step-images/

Response

[
  {
    "id": "7a8b9c0d-1e2f-4a3b-8c4d-5e6f7a8b9c0d",
    "image_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "img_path": "https://images.ximilar.com/...",
    "text": "front",
    "order": 0,
    "workspace": "748e50e4-d081-4924-b9e7-f500aac6a71d"
  }
]

Agentic Walkthrough

A complete example: a card authenticity agent with one tool and one annotated conversation.

# 1. Prompts, task and dataset (mode: agentic)
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Authenticity system prompt", "type": "system", "content": "You are a card authentication agent. Use the check_db tool when a barcode is visible. Answer with JSON."}' \
     https://api.ximilar.com/vlm/v2/prompt/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Authenticity instruction", "type": "user", "content": "Decide whether the card in the image is real."}' \
     https://api.ximilar.com/vlm/v2/prompt/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Card authenticity agent", "mode": "agentic", "system_prompt_id": "__SYSTEM_PROMPT_ID__", "user_prompt_id": "__USER_PROMPT_ID__"}' \
     https://api.ximilar.com/vlm/v2/task/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Authenticity conversations", "mode": "agentic", "system_prompt_id": "__SYSTEM_PROMPT_ID__"}' \
     https://api.ximilar.com/vlm/v2/dataset/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset_id": "__DATASET_ID__"}' \
     https://api.ximilar.com/vlm/v2/task/__TASK_ID__/add-dataset/

# 2. Tool, attached to the dataset
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "check_db", "description": "Check the card database for a record by barcode.", "parameters_schema": {"type": "object", "properties": {"query": {"type": "string"}}, "required": ["query"]}, "datasets": ["__DATASET_ID__"]}' \
     https://api.ximilar.com/vlm/v2/tool/

# 3. Image and sample
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -F 'img_path=@card.jpg' https://api.ximilar.com/recognition/v2/training-image/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset": "__DATASET_ID__", "name": "Card 0001"}' \
     https://api.ximilar.com/vlm/v2/sample/

# 4. Steps, in order; the image goes on the user step
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "role": "user", "type": "text", "content": "Is this card real?", "order": 1}' \
     https://api.ximilar.com/vlm/v2/step/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"image_ids": ["__IMAGE_ID__"]}' \
     https://api.ximilar.com/vlm/v2/step/__USER_STEP_ID__/add-images/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "role": "assistant", "type": "thought", "content": "I see a barcode. Let me check the database.", "order": 2}' \
     https://api.ximilar.com/vlm/v2/step/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "role": "assistant", "type": "tool_call", "content": {"name": "check_db", "arguments": {"query": "barcode:501234567890"}}, "tool": "__TOOL_ID__", "tool_call_id": "call_1", "order": 3}' \
     https://api.ximilar.com/vlm/v2/step/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "role": "tool", "type": "tool_result", "content": {"found": true, "status": "genuine"}, "tool": "__TOOL_ID__", "tool_call_id": "call_1", "order": 4}' \
     https://api.ximilar.com/vlm/v2/step/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "role": "assistant", "type": "answer", "content": {"type": "real", "probability": 95}, "order": 5}' \
     https://api.ximilar.com/vlm/v2/step/

# 5. Validate, then (after 20+ samples) train
curl -XPOST -H 'Authorization: Token __API_TOKEN__' https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/validate/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' https://api.ximilar.com/vlm/v2/task/__TASK_ID__/train/