VLM – Retrieval Datasets

Agentic Datasets Ximilar Platform

This page is part of the Custom Vision Language Models (VLM) API reference. See the overview for task modes, key concepts, the end-to-end workflow and the list of all endpoints.

Retrieval Datasets

Retrieval datasets train an embedding model: the model maps images and text into vectors so that matching items end up close to each other. Use it for visual search, product matching or RAG over image collections. Retrieval tasks and datasets use no prompts.

Each sample is a small relevance graph made of retrieval items and relevance edges:

an anchor item is the query side (a customer photo, a text query, a crop of an object)
a document item is a candidate result (a catalogue product photo with its title, a reference image)
an edge connects one anchor with one document and carries the relevance: "1.0000" for a positive (matching) document, "0.0000" for a negative one

Every item holds free text and/or up to 10 images (and 10 detection objects); text and images of one item are fused into a single embedding.

Building a retrieval sample

Create the dataset with mode: retrieval (no prompts needed) and connect it to a retrieval task.
Create a sample.
Create an anchor item and one or more document items; attach images and/or set their text.
Create an edge from every anchor to every document with relevance 1.0000 (positive) or 0.0000 (negative).
Validate the sample.

Rules for a valid retrieval sample

at least one anchor and at least one document, each with text, an image, a detection object or a media attachment
an edge between every anchor and every document (a complete bipartite graph)
relevance is binary: exactly 0.0000 or 1.0000
a document has the same relevance for all anchors of the sample
a sample only counts for training when at least one edge is positive; the task needs at least 20 such anchors across its datasets

Typical validation errors (error_type: reason):

missing_anchor / missing_document: Retrieval sample has no anchor item / no document item.
empty_retrieval_item: Retrieval item must contain text, an image, a detection object, video, or audio media.
missing_retrieval_edge: Retrieval document ... is missing relevance edges for anchor(s): ...
invalid_retrieval_relevance: Relevance must be 0.0000 or 1.0000.
inconsistent_retrieval_relevance: Retrieval document ... must have the same relevance for every anchor.

GET/v2/retrieval_item/

List Retrieval Items

List the items of a sample ordered by item_type and order. Returns paginated results.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.

Optional attributes

Name
sample
Type
string
Description
UUID of the sample whose items to list (recommended).
Name
search
Type
string
Description
Search by sample name.
Name
page_size
Type
integer
Description
Number of results per page.

Request

GET

/v2/retrieval_item/

curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     'https://api.ximilar.com/vlm/v2/retrieval_item/?sample=__SAMPLE_ID__'

Response

{
  "count": 3,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": "aa11bb22-cc33-4dd4-8ee5-ff6677889900",
      "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
      "item_type": "anchor",
      "text": "red floral summer dress",
      "images_count": 1,
      "detection_objects_count": 0,
      "item_images": [
        {
          "id": "0a1b2c3d-4e5f-4a6b-8c7d-8e9f0a1b2c3d",
          "image_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
          "img_path": "https://images.ximilar.com/...",
          "text": null,
          "order": 0,
          "workspace": "748e50e4-d081-4924-b9e7-f500aac6a71d"
        }
      ],
      "item_detection_objects": [],
      "order": 0
    },
    {
      "id": "bb22cc33-dd44-4ee5-8ff6-001122334455",
      "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
      "item_type": "document",
      "text": "Summer Maxi Dress - red floral print",
      "images_count": 2,
      "detection_objects_count": 0,
      "item_images": ["..."],
      "item_detection_objects": [],
      "order": 0
    },
    {
      "id": "cc33dd44-ee55-4ff6-8007-112233445566",
      "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
      "item_type": "document",
      "text": "Navy pinstripe office blazer",
      "images_count": 1,
      "detection_objects_count": 0,
      "item_images": ["..."],
      "item_detection_objects": [],
      "order": 1
    }
  ]
}

POST/v2/retrieval_item/

Create Retrieval Item

Add an anchor or document item to a retrieval sample. Set text here and attach images with Add Images to Retrieval Item. GET, PATCH and DELETE /v2/retrieval_item/{item_id}/ read, update and delete an item.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
sample
Type
string
Description
UUID of the sample (must belong to a retrieval dataset).
Name
item_type
Type
string
Description
anchor (query side) or document (candidate side). Cannot be changed once the item has an edge.

Optional attributes

Name
text
Type
string
Description
Text content of the item, e.g. a search query or a product title. Combined with the attached images into one embedding.
Name
order
Type
integer
Description
Position within the items of the same type (default 0).

Errors

{"sample": "Retrieval items require a retrieval-mode sample."}
{"item_type": "Item type cannot change after a relevance edge is created."}

Request

POST

/v2/retrieval_item/

curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{
       "sample": "__SAMPLE_ID__",
       "item_type": "anchor",
       "text": "red floral summer dress",
       "order": 0
     }' \
     https://api.ximilar.com/vlm/v2/retrieval_item/

Response

{
  "id": "aa11bb22-cc33-4dd4-8ee5-ff6677889900",
  "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
  "item_type": "anchor",
  "text": "red floral summer dress",
  "images_count": 0,
  "detection_objects_count": 0,
  "media_count": 0,
  "item_images": [],
  "item_detection_objects": [],
  "order": 0
}

POST/v2/retrieval_item/{item_id}/add-images/

Add Images to Retrieval Item

Attach images to an anchor or document item (max 10 per item). POST /v2/retrieval_item/{item_id}/remove-images/ detaches them. Detection objects work the same way with add-detection-objects/, remove-detection-objects/ and a body {"detection_object_ids": [...]}.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
item_id
Type
string
Description
UUID of the retrieval item.
Name
image_ids
Type
array
Description
List of image UUIDs to add.

Errors

Cannot add 2 images. Item already has 9/10 images.
Some images were not found or not accessible in this workspace.

Request

POST

/v2/retrieval_item/{item_id}/add-images/

curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{"image_ids": ["__IMAGE_ID_1__", "__IMAGE_ID_2__"]}' \
     https://api.ximilar.com/vlm/v2/retrieval_item/__ITEM_ID__/add-images/

Response

{
  "added": 2
}

GET/v2/retrieval_item/{item_id}/item-images/

List Retrieval Item Images

List the images attached to an item in order. PATCH /v2/retrieval_item/{item_id}/item-images/{item_image_id}/ with text and/or order updates one attached image. Detection objects are listed with GET /v2/retrieval_item/{item_id}/item-detection-objects/ and updated with PATCH .../item-detection-objects/{item_object_id}/.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
item_id
Type
string
Description
UUID of the retrieval item.

Returns (each item)

Name
id
Type
string
Description
UUID of the item-image relation.
Name
image_id
Type
string
Description
UUID of the image.
Name
img_path
Type
string
Description
Full URL of the image.
Name
text
Type
string
Description
Optional text label shown to the model before this image.
Name
order
Type
integer
Description
Order of the image within the item.

Request

GET

/v2/retrieval_item/{item_id}/item-images/

curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     https://api.ximilar.com/vlm/v2/retrieval_item/__ITEM_ID__/item-images/

Response

[
  {
    "id": "0a1b2c3d-4e5f-4a6b-8c7d-8e9f0a1b2c3d",
    "image_id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "img_path": "https://images.ximilar.com/...",
    "text": "front view",
    "order": 0,
    "workspace": "748e50e4-d081-4924-b9e7-f500aac6a71d"
  }
]

GET/v2/retrieval_edge/

List Retrieval Edges

List the relevance edges of a sample. Returns paginated results.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.

Optional attributes

Name
sample
Type
string
Description
UUID of the sample whose edges to list (recommended).
Name
page_size
Type
integer
Description
Number of results per page.

Request

GET

/v2/retrieval_edge/

curl -v -XGET \
     -H 'Authorization: Token __API_TOKEN__' \
     'https://api.ximilar.com/vlm/v2/retrieval_edge/?sample=__SAMPLE_ID__'

Response

{
  "count": 2,
  "next": null,
  "previous": null,
  "results": [
    {
      "id": "dd44ee55-ff66-4007-8118-223344556677",
      "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
      "anchor_item": "aa11bb22-cc33-4dd4-8ee5-ff6677889900",
      "document_item": "bb22cc33-dd44-4ee5-8ff6-001122334455",
      "relevance": "1.0000"
    },
    {
      "id": "ee55ff66-0077-4118-8229-334455667788",
      "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
      "anchor_item": "aa11bb22-cc33-4dd4-8ee5-ff6677889900",
      "document_item": "cc33dd44-ee55-4ff6-8007-112233445566",
      "relevance": "0.0000"
    }
  ]
}

POST/v2/retrieval_edge/

Create Retrieval Edge

Set the relevance between one anchor and one document of the same sample. Create one edge for every anchor-document pair. PATCH /v2/retrieval_edge/{edge_id}/ with {"relevance": "0.0000"} changes a positive into a negative (and vice versa); DELETE removes the edge.

Required attributes

Name
Authorization
Type
string
Description
Unique API token for authentication.
Name
sample
Type
string
Description
UUID of the sample.
Name
anchor_item
Type
string
Description
UUID of an item with item_type: anchor of this sample.
Name
document_item
Type
string
Description
UUID of an item with item_type: document of this sample.
Name
relevance
Type
string
Description
"1.0000" for a positive (matching) document, "0.0000" for a negative one. Sent and returned as a decimal string with four decimals.

Errors

{"anchor_item": "Anchor item must have item_type='anchor'."}
{"document_item": "Document item must have item_type='document'."}
{"anchor_item": "Anchor item must belong to the edge sample."}
400 when an edge for the same anchor and document already exists (use PATCH instead).

Request

POST

/v2/retrieval_edge/

curl -v -XPOST \
     -H 'Authorization: Token __API_TOKEN__' \
     -H 'Content-Type: application/json' \
     -d '{
       "sample": "__SAMPLE_ID__",
       "anchor_item": "__ANCHOR_ITEM_ID__",
       "document_item": "__POSITIVE_DOCUMENT_ID__",
       "relevance": "1.0000"
     }' \
     https://api.ximilar.com/vlm/v2/retrieval_edge/

Response

{
  "id": "dd44ee55-ff66-4007-8118-223344556677",
  "sample": "c3d4e5f6-a7b8-9012-cdef-345678901234",
  "anchor_item": "aa11bb22-cc33-4dd4-8ee5-ff6677889900",
  "document_item": "bb22cc33-dd44-4ee5-8ff6-001122334455",
  "relevance": "1.0000"
}

Retrieval Walkthrough

A complete example: a customer photo (anchor) matched against a positive and a negative catalogue product (documents).

# 1. Task and dataset (mode: retrieval, no prompts)
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Fashion product search", "mode": "retrieval"}' \
     https://api.ximilar.com/vlm/v2/task/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"name": "Dress catalogue pairs", "mode": "retrieval"}' \
     https://api.ximilar.com/vlm/v2/dataset/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset_id": "__DATASET_ID__"}' \
     https://api.ximilar.com/vlm/v2/task/__TASK_ID__/add-dataset/

# 2. Images
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -F 'img_path=@customer_photo.jpg' https://api.ximilar.com/recognition/v2/training-image/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -F 'img_path=@catalogue_dress.jpg' https://api.ximilar.com/recognition/v2/training-image/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -F 'img_path=@catalogue_blazer.jpg' https://api.ximilar.com/recognition/v2/training-image/

# 3. Sample with one anchor and two documents
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"dataset": "__DATASET_ID__", "name": "Dress 0001"}' \
     https://api.ximilar.com/vlm/v2/sample/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "item_type": "anchor", "text": "red floral summer dress"}' \
     https://api.ximilar.com/vlm/v2/retrieval_item/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"image_ids": ["__CUSTOMER_PHOTO_ID__"]}' \
     https://api.ximilar.com/vlm/v2/retrieval_item/__ANCHOR_ITEM_ID__/add-images/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "item_type": "document", "text": "Summer Maxi Dress - red floral print", "order": 0}' \
     https://api.ximilar.com/vlm/v2/retrieval_item/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"image_ids": ["__CATALOGUE_DRESS_ID__"]}' \
     https://api.ximilar.com/vlm/v2/retrieval_item/__POSITIVE_DOCUMENT_ID__/add-images/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "item_type": "document", "text": "Navy pinstripe office blazer", "order": 1}' \
     https://api.ximilar.com/vlm/v2/retrieval_item/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"image_ids": ["__CATALOGUE_BLAZER_ID__"]}' \
     https://api.ximilar.com/vlm/v2/retrieval_item/__NEGATIVE_DOCUMENT_ID__/add-images/

# 4. One edge per anchor-document pair: positive and negative
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "anchor_item": "__ANCHOR_ITEM_ID__", "document_item": "__POSITIVE_DOCUMENT_ID__", "relevance": "1.0000"}' \
     https://api.ximilar.com/vlm/v2/retrieval_edge/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' -H 'Content-Type: application/json' \
     -d '{"sample": "__SAMPLE_ID__", "anchor_item": "__ANCHOR_ITEM_ID__", "document_item": "__NEGATIVE_DOCUMENT_ID__", "relevance": "0.0000"}' \
     https://api.ximilar.com/vlm/v2/retrieval_edge/

# 5. Validate, then (after 20+ anchors with a positive document) train
curl -XPOST -H 'Authorization: Token __API_TOKEN__' https://api.ximilar.com/vlm/v2/sample/__SAMPLE_ID__/validate/
curl -XPOST -H 'Authorization: Token __API_TOKEN__' https://api.ximilar.com/vlm/v2/task/__TASK_ID__/train/