VLM – Instruction Datasets

SamplesAgentic DatasetsXimilar Platform

This page is part of the Custom Vision Language Models (VLM) API reference. See the overview for task modes, key concepts, the end-to-end workflow and the list of all endpoints.

Instruction Datasets

Instruction datasets teach the model to answer a fixed instruction with a structured result. The dataset's result template (a template prompt) contains {{placeholders}}; each placeholder must be declared as a variable of the dataset. A sample holds one or more images (or detection objects, i.e. crops of images) and one value per variable. During training the model sees the system prompt, the user prompt with the images, and learns to produce the template filled with the sample's values.

Building an instruction sample

  1. Create the dataset with mode: instruction, a system prompt, a user prompt and a result template such as {"grade": {{grade}}, "explain": "{{explain}}"}.
  2. Create a variable for every placeholder (grade as float, explain as string).
  3. Upload images with Upload Training Image and note their ids.
  4. Create a sample and add the images (max 10 images and 10 detection objects per sample).
  5. Set the variable values and optionally input_meta_data for placeholders in the user prompt.
  6. Validate the sample. A valid instruction sample has at least one image or object, a value for every required variable and no unresolved placeholders.

Supported Variable Types

TypeDescription
stringText values with optional min_length / max_length and a comma-separated list of allowed choices
integerWhole numbers with optional min_value / max_value
floatDecimal numbers with optional min_value / max_value and step_size
booleantrue / false
arrayJSON list of values with optional min_length / max_length
objectNested JSON object

Variable names must start with a letter or underscore and contain only letters, numbers and underscores.

GET/v2/variable/

List Variables

List all variables, optionally filtered by dataset.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

Optional attributes

  • Name
    dataset
    Type
    string
    Description

    Filter variables by dataset ID.

  • Name
    search
    Type
    string
    Description

    Search by variable or dataset name.

  • Name
    page_size
    Type
    integer
    Description

    Number of results per page.

Request

GET
/v2/variable/
curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     'https://api.ximilar.com/vlm/v2/variable/?dataset=__DATASET_ID__'

Response

{
  "count": 2,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": "f82b01ed-e65d-4730-a458-2966cbf86994",
      "dataset": "8797c273-b1d3-4e6f-82bb-adfb719415fe",
      "dataset_name": "Grading dataset",
      "name": "grade",
      "type": "float",
      "required": true
    },
    {
      "id": "7b661f0e-7c00-4b59-b334-aca2a0249e2f",
      "dataset": "8797c273-b1d3-4e6f-82bb-adfb719415fe",
      "dataset_name": "Grading dataset",
      "name": "explain",
      "type": "string",
      "required": false
    }
  ]
}
GET/v2/variable/{variable_id}/

Get Variable

Get details of a specific variable by its ID, including all constraints.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    variable_id
    Type
    string
    Description

    UUID of the variable.

Request

GET
/v2/variable/{variable_id}/
curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     https://api.ximilar.com/vlm/v2/variable/__VARIABLE_ID__/

Response

{
  "id": "f82b01ed-e65d-4730-a458-2966cbf86994",
  "dataset": "8797c273-b1d3-4e6f-82bb-adfb719415fe",
  "dataset_name": "Grading dataset",
  "name": "grade",
  "type": "float",
  "required": true,
  "choices": null,
  "min_length": null,
  "max_length": null,
  "min_value": 0.0,
  "max_value": 10.0,
  "step_size": 0.5,
  "description": "Overall grade of the card",
  "default_value": null
}
POST/v2/variable/

Create Variable

Create a new variable for a dataset. Create one variable for every {{placeholder}} used in the result template.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    dataset
    Type
    string
    Description

    UUID of the dataset this variable belongs to. Cannot be changed later.

  • Name
    name
    Type
    string
    Description

    Variable name, unique within the dataset (letters, numbers and underscores, not starting with a number).

  • Name
    type
    Type
    string
    Description

    Variable type: string, integer, float, boolean, array, or object.

Optional attributes

  • Name
    required
    Type
    boolean
    Description

    Whether every sample must provide a value (default: true).

  • Name
    description
    Type
    string
    Description

    Human-readable description.

  • Name
    choices
    Type
    string
    Description

    Comma-separated list of allowed values (for string type), e.g. "mint,near_mint,excellent".

  • Name
    min_value
    Type
    number
    Description

    Minimum value (for numeric types).

  • Name
    max_value
    Type
    number
    Description

    Maximum value (for numeric types).

  • Name
    step_size
    Type
    number
    Description

    Step size (for float type).

  • Name
    min_length
    Type
    integer
    Description

    Minimum length (for string/array types).

  • Name
    max_length
    Type
    integer
    Description

    Maximum length (for string/array types).

  • Name
    default_value
    Type
    any
    Description

    Default value if not provided.

Request

POST
/v2/variable/
curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{
       "dataset": "__DATASET_ID__",
       "name": "condition",
       "type": "string",
       "required": false,
       "description": "Condition of the item",
       "choices": "mint,near_mint,excellent,good,poor"
     }' \
     https://api.ximilar.com/vlm/v2/variable/
PATCH/v2/variable/{variable_id}/

Update Variable

Update the constraints or description of a variable. The dataset cannot be changed. Existing sample values are not re-validated.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    variable_id
    Type
    string
    Description

    UUID of the variable.

Optional attributes

Any of name, type, required, description, choices, min_value, max_value, step_size, min_length, max_length, default_value.

Request

PATCH
/v2/variable/{variable_id}/
curl -v -XPATCH \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{"required": false, "max_value": 10.5}' \
     https://api.ximilar.com/vlm/v2/variable/__VARIABLE_ID__/
DELETE/v2/variable/{variable_id}/

Delete Variable

Delete a variable and all values annotated for it. Remove the corresponding placeholder from the result template as well, otherwise the dataset becomes invalid.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    variable_id
    Type
    string
    Description

    UUID of the variable.

Request

DELETE
/v2/variable/{variable_id}/
curl -v -XDELETE \
     -H 'Authorization: Token __API_TOKEN__' \
     https://api.ximilar.com/vlm/v2/variable/__VARIABLE_ID__/
POST/v2/sample/{sample_id}/add-images/

Add Images to Sample

Attach images to an instruction sample. Images must already be uploaded to your workspace (see Upload Training Image). A sample can hold at most 10 images. New images are appended after the existing ones (their order continues the sequence).

POST /v2/sample/{sample_id}/remove-images/ with the same body detaches images again.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    sample_id
    Type
    string
    Description

    UUID of the sample.

  • Name
    image_ids
    Type
    array
    Description

    List of image UUIDs to add.

Errors

  • Sample can have at most 10 images (currently 9, trying to add 2).
  • One or more images not found or not accessible
  • Sample-level images are only valid for instruction datasets. (use step or retrieval item images instead)

Request

POST
/v2/sample/{sample_id}/add-images/
curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{"image_ids": ["__IMAGE_ID_1__", "__IMAGE_ID_2__"]}' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/add-images/

Response

{
  "added": 2
}
GET/v2/sample/{sample_id}/sample-images/

List Sample Images

List all images attached to a sample with their metadata (text, order, resize settings).

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    sample_id
    Type
    string
    Description

    UUID of the sample.

Returns (each item)

  • Name
    id
    Type
    string
    Description

    UUID of the sample-image relation (use it for Update Sample Image).

  • Name
    image_id
    Type
    string
    Description

    UUID of the image.

  • Name
    img_path
    Type
    string
    Description

    Full URL of the image.

  • Name
    thumb
    Type
    string
    Description

    URL of the image thumbnail.

  • Name
    text
    Type
    string
    Description

    Optional text label shown to the model before this image (e.g. front, back).

  • Name
    order
    Type
    integer
    Description

    Order of the image within the sample.

  • Name
    resize
    Type
    boolean
    Description

    Whether the image is resized before it is fed to the model (default true).

  • Name
    meta_data
    Type
    object
    Description

    Additional metadata for this sample image.

Request

GET
/v2/sample/{sample_id}/sample-images/
curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/sample-images/

Response

[
  {
    "id": "d4e5f6a7-b8c9-0123-def4-567890123456",
    "image_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "img_path": "https://images.ximilar.com/...",
    "thumb": "https://images.ximilar.com/.../thumb",
    "text": "front",
    "order": 0,
    "resize": true,
    "meta_data": {},
    "workspace": "748e50e4-d081-4924-b9e7-f500aac6a71d"
  }
]
PATCH/v2/sample/{sample_id}/sample-images/{sample_image_id}/

Update Sample Image

Change the text label, order or resize flag of one attached image. GET on the same URL returns a single sample image. Images and detection objects of a sample share one order sequence, so an order that is already used by another attachment is rejected.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    sample_id
    Type
    string
    Description

    UUID of the sample.

  • Name
    sample_image_id
    Type
    string
    Description

    UUID of the sample-image relation (the id from List Sample Images).

Optional attributes

  • Name
    text
    Type
    string
    Description

    Text label for this image.

  • Name
    order
    Type
    integer
    Description

    New position of the image (0-based).

  • Name
    resize
    Type
    boolean
    Description

    Whether to resize the image for the model.

Request

PATCH
/v2/sample/{sample_id}/sample-images/{sample_image_id}/
curl -v -XPATCH \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{"text": "back side", "order": 1}' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/sample-images/__SAMPLE_IMAGE_ID__/
POST/v2/sample/{sample_id}/add-objects/

Add Detection Objects to Sample

Attach detection objects (bounding boxes of the Object Detection API) to an instruction sample. The model then sees the cropped region instead of the whole image. A sample can have at most 10 detection objects.

POST /v2/sample/{sample_id}/remove-objects/ with the same body detaches objects again.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    sample_id
    Type
    string
    Description

    UUID of the sample.

  • Name
    detection_object_ids
    Type
    array
    Description

    List of detection object UUIDs to add.

Errors

  • Sample can have at most 10 detection objects (currently 10, trying to add 1).
  • One or more detection objects not found or not accessible

Request

POST
/v2/sample/{sample_id}/add-objects/
curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{"detection_object_ids": ["__OBJECT_ID_1__", "__OBJECT_ID_2__"]}' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/add-objects/

Response

{
  "added": 2
}
GET/v2/sample/{sample_id}/sample-objects/

List Sample Detection Objects

List all detection objects attached to a sample with their bounding boxes and label information. PATCH /v2/sample/{sample_id}/sample-objects/{sample_object_id}/ with text and/or order updates one attached object.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    sample_id
    Type
    string
    Description

    UUID of the sample.

Returns (each item)

  • Name
    id
    Type
    string
    Description

    UUID of the sample-object relation.

  • Name
    detection_object_id
    Type
    string
    Description

    UUID of the detection object.

  • Name
    image_id
    Type
    string
    Description

    UUID of the source image.

  • Name
    image_url
    Type
    string
    Description

    Full URL of the source image.

  • Name
    thumb_url
    Type
    string
    Description

    URL of the detection object thumbnail (cropped region).

  • Name
    label_name
    Type
    string
    Description

    Name of the detection label.

  • Name
    label_color
    Type
    string
    Description

    Color of the detection label.

  • Name
    bbox
    Type
    array
    Description

    Bounding box [xmin, ymin, xmax, ymax].

  • Name
    text
    Type
    string
    Description

    Optional text label shown to the model before this crop.

  • Name
    order
    Type
    integer
    Description

    Order of the object within the sample.

  • Name
    meta_data
    Type
    object
    Description

    Additional metadata for this sample object.

Request

GET
/v2/sample/{sample_id}/sample-objects/
curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/sample-objects/

Response

[
  {
    "id": "e5f6a7b8-c9d0-1234-ef56-789012345678",
    "detection_object_id": "f6a7b8c9-d0e1-2345-f678-901234567890",
    "image_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "image_url": "https://images.ximilar.com/...",
    "thumb_url": "https://images.ximilar.com/.../thumb",
    "label_name": "card",
    "label_color": "#FF0000",
    "bbox": [100, 150, 400, 350],
    "text": null,
    "order": 0,
    "meta_data": {},
    "workspace": "748e50e4-d081-4924-b9e7-f500aac6a71d"
  }
]
POST/v2/sample/{sample_id}/add-variable-value/

Add Variable Value to Sample

Set (or replace) the value of one variable for a sample. This is how you annotate instruction training data. The value is checked against the variable definition (type, range, length, choices). Sending null or an empty string deletes the value.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

  • Name
    sample_id
    Type
    string
    Description

    UUID of the sample.

  • Name
    dataset_variable
    Type
    string
    Description

    UUID of the variable to set (must belong to the sample's dataset).

  • Name
    value
    Type
    any
    Description

    The value: a number, string, boolean, array or object matching the variable type.

Returns

The stored sample variable (id, sample, dataset_variable, variable_name, variable_type, value), or {"deleted": true} when the value was cleared.

Errors

  • Dataset variable not found or does not belong to the sample's dataset
  • grade must be <= 10.0, condition must be one of: mint, near_mint, ..., tags must be an array
  • Sample variable values are only valid for instruction datasets.

Request

POST
/v2/sample/{sample_id}/add-variable-value/
curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{
       "dataset_variable": "__GRADE_VARIABLE_ID__",
       "value": 8.5
     }' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/add-variable-value/

Response

{
  "id": "9a1f3c2e-5b6d-4e7f-8a9b-0c1d2e3f4a5b",
  "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
  "dataset_variable": "f82b01ed-e65d-4730-a458-2966cbf86994",
  "variable_name": "grade",
  "variable_type": "float",
  "value": 8.5
}
GET/v2/sample_variable/

List Sample Variable Values

List annotated variable values, optionally filtered by sample. The same values are embedded in Get Sample as sample_variables.

Required attributes

  • Name
    Authorization
    Type
    string
    Description

    Unique API token for authentication.

Optional attributes

  • Name
    sample
    Type
    string
    Description

    Filter by sample UUID.

  • Name
    search
    Type
    string
    Description

    Search by variable name or dataset name.

Request

GET
/v2/sample_variable/
curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     'https://api.ximilar.com/vlm/v2/sample_variable/?sample=__SAMPLE_ID__'

Instruction Walkthrough

A complete example: a card grading dataset with two variables, one sample with two images and its annotation.

# 1. Prompts
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Grading system prompt", "type": "system", "content": "You are an expert card grader. Answer in JSON."}' \
     https://api.ximilar.com/vlm/v2/prompt/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Grading instruction", "type": "user", "content": "Grade this {{category}} from 1 to 10 and explain the grade."}' \
     https://api.ximilar.com/vlm/v2/prompt/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Grading template", "type": "template", "format": "json", "content": "{\"grade\": {{grade}}, \"explain\": \"{{explain}}\"}"}' \
     https://api.ximilar.com/vlm/v2/prompt/

# 2. Task and dataset (mode: instruction)
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Card grading", "mode": "instruction", "system_prompt_id": "__SYSTEM_PROMPT_ID__", "user_prompt_id": "__USER_PROMPT_ID__"}' \
     https://api.ximilar.com/vlm/v2/task/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Grading dataset", "mode": "instruction", "system_prompt_id": "__SYSTEM_PROMPT_ID__", "user_prompt_id": "__USER_PROMPT_ID__", "result_template_id": "__TEMPLATE_PROMPT_ID__", "result_format": "json", "default_input_meta_data": {"category": "sports card"}}' \
     https://api.ximilar.com/vlm/v2/dataset/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset_id": "__DATASET_ID__"}' \
     https://api.ximilar.com/vlm/v2/task/__TASK_ID__/add-dataset/

# 3. Variables (one per placeholder in the template)
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset": "__DATASET_ID__", "name": "grade", "type": "float", "min_value": 1, "max_value": 10, "step_size": 0.5}' \
     https://api.ximilar.com/vlm/v2/variable/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset": "__DATASET_ID__", "name": "explain", "type": "string"}' \
     https://api.ximilar.com/vlm/v2/variable/

# 4. Images (recognition API), then the sample
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -F 'img_path=@front.jpg' https://api.ximilar.com/recognition/v2/training-image/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -F 'img_path=@back.jpg'  https://api.ximilar.com/recognition/v2/training-image/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset": "__DATASET_ID__", "name": "Card 0001", "type": "multi_ordered"}' \
     https://api.ximilar.com/vlm/v2/sample/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"image_ids": ["__FRONT_IMAGE_ID__", "__BACK_IMAGE_ID__"]}' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/add-images/

# 5. Annotation
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset_variable": "__GRADE_VARIABLE_ID__", "value": 8.5}' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/add-variable-value/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset_variable": "__EXPLAIN_VARIABLE_ID__", "value": "Sharp corners, light surface wear on the back."}' \
     https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/add-variable-value/

# 6. Validate, then (after 20+ samples) train
curl -XPOST -H 'Authorization: Token __API_TOKEN__' https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/validate/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' https://api.ximilar.com/vlm/v2/task/__TASK_ID__/train/

Was this page helpful?