Advanced OCR system
This page describes API for OCR (Optical Character Recognition) service and combination of OCR on images and ChatGPT (large language models applied on ocr results). The API follows the general rules of Ximilar API as described in Section First steps.
This service API has two endpoints running at this URL:
https://api.ximilar.com/ocr/v2/read
https://api.ximilar.com/ocr/v2/read_gpt
Sign-up via app.ximilar.com
In order to get access to the OCR service, please register at https://app.ximilar.com and then contact us at tech@ximilar.com. Be aware that using ChatGPT in combination of OCR requires at least business plan.
How to use ChatGPT & OCR?
If you are looking for a more detailed analysis of texts, you can use /v2/read_gpt endpoint.
There are many use-cases that you can use this endpoint:
- You can for example datamine information from invoices.
- You want to automatically read texts from trading cards, posters, comics, ...
- You want to read nutrition information from labels on food products, ...
Just specift the prompt field (str) in record and this prompt will be send along with full_text (readed text by OCR) from image to ChatGPT API.
OCR reading endpoint: /v2/read
Given a list of image records, this method returns readed text with OCR system on the images. For each image it predict positions of text (defined by POLYGON) with readed text based on language. The result is stored in _ocr field (see JSON RESULT below).
Parameters:
records: list of photos to predict the ocr for- must contain either of
_urlor_base64field - see section image data for details - maximum number of records in request is 10
- optionaly you can specify
langparam per record (if each image is from different language)
- must contain either of
lang: language settings for ocr model, currently: "en", "zh" (chinese), "ko" (korean), "ja" (japanese), "ru" (russian). Default: "en"
$ curl https://api.ximilar.com/ocr/v2/read -H "Content-Type: application/json" -H "Authorization: Token __API_TOKEN__" -d '{
"lang": "en",
"records": [
{ "_url": "https://images.ximilar.com/examples/cards/mew_pokemon.jpeg" }
]
}'
from ximilar.client import OCRClient
ocr_client = OCRClient(token="__API_TOKEN__")
result = ocr_client.read([{"_url": "__URL_PATH_TO_IMAGE__"}], lang="en")
print(result['records'][0]['_ocr'])
Returns:
- HTTP error code 2XX, if the method was OK and other HTTP error code, if the method failed.
- Body of the response is a JSON object (map) with the following fields:
status- a JSON map with a status of the method processing. It contains these subfields:code- a numeric code of the operation status; it follows the concept of HTTP status codes (2XX, 4XX). Specific codes are described for each type of answer (or operation) (see below).text- a text describing the status code
- Example of statuses that can be returned:
"status": {"code": 200, "text": "OK"}
"status": {"code": 402, "text": "aborted by error", error_description="..."}
"status": {"code": 500, "text": "unknown error", "error_description": "..."} statistics- a map of various statistics about the processing. The only statistic included every time isprocessing time- time of actual processing of the query [in seconds]
records- JSON array with the input records, each record enriched by field "_ocr"
CLICK TO SHOW JSON RESULT
{
"records": [
{
"_url": "https://images.ximilar.com/examples/cards/mew_pokemon.jpeg",
"_status": {
"code": 200,
"text": "OK",
"request_id": "f9cec41f-dd6e-4aa9-93b3-4f2510fd5eb6"
},
"_id": "4d903663-a348-4d9b-be76-4df38df7d66a",
"_width": 1140,
"_height": 1520,
"_ocr": {
"texts": [
{
"polygon": [
[
285.0,
221.0
],
[
384.0,
229.0
],
[
382.0,
256.0
],
[
283.0,
248.0
]
],
"text": "BASIC",
"prob": 0.9564334750175476
},
{
"polygon": [
[
404.0,
226.0
],
[
568.0,
246.0
],
[
562.0,
295.0
],
[
398.0,
275.0
]
],
"text": "MewV",
"prob": 0.9809828996658325
},
{
"polygon": [
[
818.0,
285.0
],
[
931.0,
296.0
],
[
926.0,
350.0
],
[
813.0,
339.0
]
],
"text": "180",
"prob": 0.9965084195137024
},
{
"polygon": [
[
851.0,
389.0
],
[
946.0,
396.0
],
[
944.0,
419.0
],
[
850.0,
411.0
]
],
"text": "ASTRIKE",
"prob": 0.8877888321876526
},
{
"polygon": [
[
504.0,
891.0
],
[
703.0,
891.0
],
[
703.0,
931.0
],
[
504.0,
931.0
]
],
"text": "EnergyMix",
"prob": 0.9763773679733276
},
{
"polygon": [
[
290.0,
939.0
],
[
947.0,
936.0
],
[
947.0,
969.0
],
[
290.0,
972.0
]
],
"text": "Search your deck foran Energy cardand attach itto.lof",
"prob": 0.9097585082054138
},
{
"polygon": [
[
902.0,
1026.0
],
[
946.0,
1026.0
],
[
946.0,
1061.0
],
[
902.0,
1061.0
]
],
"text": "70",
"prob": 0.9970272779464722
},
{
"polygon": [
[
495.0,
1036.0
],
[
702.0,
1030.0
],
[
703.0,
1068.0
],
[
496.0,
1074.0
]
],
"text": "Psychic Leap",
"prob": 0.9445925354957581
},
{
"polygon": [
[
285.0,
1088.0
],
[
945.0,
1065.0
],
[
946.0,
1102.0
],
[
286.0,
1125.0
]
],
"text": "You may shuffle this Pokemon and all attached cards into",
"prob": 0.9245634078979492
},
{
"polygon": [
[
281.0,
1128.0
],
[
419.0,
1117.0
],
[
422.0,
1150.0
],
[
284.0,
1161.0
]
],
"text": "your deck.",
"prob": 0.938432514667511
},
{
"polygon": [
[
437.0,
1181.0
],
[
656.0,
1170.0
],
[
658.0,
1198.0
],
[
438.0,
1210.0
]
],
"text": "resistance-30",
"prob": 0.9185628890991211
},
{
"polygon": [
[
720.0,
1172.0
],
[
777.0,
1166.0
],
[
780.0,
1189.0
],
[
722.0,
1194.0
]
],
"text": "retreat",
"prob": 0.9936887621879578
},
{
"polygon": [
[
257.0,
1193.0
],
[
444.0,
1182.0
],
[
446.0,
1209.0
],
[
259.0,
1220.0
]
],
"text": "weaknessx2",
"prob": 0.9480481147766113
},
{
"polygon": [
[
556.0,
1233.0
],
[
619.0,
1226.0
],
[
621.0,
1249.0
],
[
559.0,
1255.0
]
],
"text": "Vrule",
"prob": 0.9715956449508667
},
{
"polygon": [
[
685.0,
1226.0
],
[
950.0,
1211.0
],
[
951.0,
1233.0
],
[
686.0,
1248.0
]
],
"text": "When your Pokemon V Knocked Oet",
"prob": 0.8164744973182678
},
{
"polygon": [
[
270.0,
1255.0
],
[
386.0,
1246.0
],
[
388.0,
1269.0
],
[
272.0,
1278.0
]
],
"text": "Tus NackPSa",
"prob": 0.6147907376289368
},
{
"polygon": [
[
529.0,
1266.0
],
[
791.0,
1244.0
],
[
792.0,
1266.0
],
[
531.0,
1288.0
]
],
"text": "your oppoment takes 2 Prize cards.",
"prob": 0.8714807629585266
},
{
"polygon": [
[
267.0,
1278.0
],
[
458.0,
1273.0
],
[
459.0,
1311.0
],
[
268.0,
1316.0
]
],
"text": "2126",
"prob": 0.6876019239425659
}
],
"full_text": "BASIC MewV 180 ASTRIKE EnergyMix Search your deck foran Energy cardand attach itto.lof 70 Psychic Leap You may shuffle this Pokemon and all attached cards into your deck. resistance-30 retreat weaknessx2 Vrule When your Pokemon V Knocked Oet Tus NackPSa your oppoment takes 2 Prize cards. 2126 ",
"lang": "en",
"lang_name": "english"
}
}
],
"status": {
"code": 200,
"text": "OK",
"request_id": "f9cec41f-dd6e-4aa9-93b3-4f2510fd5eb6",
"proc_id": "1a088e14-714a-41bd-9a89-4d732becafcd"
},
"statistics": {
"processing time": 5.054640769958496
}
}
OCR reading and GPT analysis: /v2/read_gpt
Given a list of image records, this method returns analyzed readed text with ocr and analysed text with ChatGPT based on your input prompt. For each image it predict positions of text (defined by POLYGON) with readed text based on language. The result is stored in _ocr and _gpt fields. This endpoint requires at least a business plan activated via app.ximilar.com
Parameters:
records: list of photos to predict the ocr for- must contain either of
_urlor_base64field - see section image data for details - maximum number of records in request is 10
- must contain
promptfor calling gpt service. Maximum number of tokens to process is limited to 1000 per record/image. - optionaly you can specify
langparam per record (if each image is from different language)
- must contain either of
lang: language settings for ocr model, currently: "en", "zh" (chinese), "ko" (korean), "ja" (japanese), "ru" (russian). Default: "en"
$ curl --request POST \
--url https://api.ximilar.com/ocr/v2/read_gpt \
--header 'Authorization: Token __API_TOKEN__' \
--header 'content-type: application/json' \
--data '{
"records": [
{
"_url": "https://images.ximilar.com/examples/cards/mew_pokemon.jpeg",
"prompt": "based on the following result from ocr system what is the name and type of the card as json result ({'\''name'\'':'\'''\'', '\''type'\'': '\'''\''})"
}
],
"lang": "en"
}'
from ximilar.client import OCRClient
ocr_client = OCRClient(token="__API_TOKEN__")
result = ocr_client.read_gpt([{"_url": "__URL_PATH_TO_IMAGE__", "prompt": "__YOUR_GPT_PROMPT__"}], lang="en")
print(result['records'][0]['_ocr'])
print(result['records'][0]['_gpt'])
Returns:
- HTTP error code 2XX, if the method was OK and other HTTP error code, if the method failed.
- Body of the response is a JSON object (map) with the following fields:
status- a JSON map with a status of the method processing. It contains these subfields:code- a numeric code of the operation status; it follows the concept of HTTP status codes (2XX, 4XX). Specific codes are described for each type of answer (or operation) (see below).text- a text describing the status code
- Example of statuses that can be returned:
"status": {"code": 200, "text": "OK"}
"status": {"code": 402, "text": "aborted by error", error_description="..."}
"status": {"code": 500, "text": "unknown error", "error_description": "..."} statistics- a map of various statistics about the processing. The only statistic included every time isprocessing time- time of actual processing of the query [in seconds]
records- JSON array with the input records, each record enriched by field "_ocr" and "_gpt" fields
CLICK TO SHOW JSON RESULT
{
"records": [
{
"_url": "https://images.ximilar.com/examples/cards/mew_pokemon.jpeg",
"prompt": "based on the following result from ocr system what is the name and type of the card as json result ({'name':'', 'type': ''})",
"_status": {
"code": 200,
"text": "OK",
"request_id": "5d735c9b-71e7-425f-a4c7-4423230b3748"
},
"_id": "903ae611-2df4-4385-989e-146fa4d0fd7a",
"_width": 1140,
"_height": 1520,
"_ocr": {
"texts": [
{
"polygon": [
[
285.0,
221.0
],
[
384.0,
229.0
],
[
382.0,
256.0
],
[
283.0,
248.0
]
],
"text": "BASIC",
"prob": 0.9564334750175476
},
{
"polygon": [
[
404.0,
226.0
],
[
568.0,
246.0
],
[
562.0,
295.0
],
[
398.0,
275.0
]
],
"text": "MewV",
"prob": 0.9809828996658325
},
{
"polygon": [
[
818.0,
285.0
],
[
931.0,
296.0
],
[
926.0,
350.0
],
[
813.0,
339.0
]
],
"text": "180",
"prob": 0.9965084195137024
},
{
"polygon": [
[
851.0,
389.0
],
[
946.0,
396.0
],
[
944.0,
419.0
],
[
850.0,
411.0
]
],
"text": "ASTRIKE",
"prob": 0.8877888321876526
},
{
"polygon": [
[
504.0,
891.0
],
[
703.0,
891.0
],
[
703.0,
931.0
],
[
504.0,
931.0
]
],
"text": "EnergyMix",
"prob": 0.9763773679733276
},
{
"polygon": [
[
290.0,
939.0
],
[
947.0,
936.0
],
[
947.0,
969.0
],
[
290.0,
972.0
]
],
"text": "Search your deck foran Energy cardand attach itto.lof",
"prob": 0.9097585082054138
},
{
"polygon": [
[
902.0,
1026.0
],
[
946.0,
1026.0
],
[
946.0,
1061.0
],
[
902.0,
1061.0
]
],
"text": "70",
"prob": 0.9970272779464722
},
{
"polygon": [
[
495.0,
1036.0
],
[
702.0,
1030.0
],
[
703.0,
1068.0
],
[
496.0,
1074.0
]
],
"text": "Psychic Leap",
"prob": 0.9445925354957581
},
{
"polygon": [
[
285.0,
1088.0
],
[
945.0,
1065.0
],
[
946.0,
1102.0
],
[
286.0,
1125.0
]
],
"text": "You may shuffle this Pokemon and all attached cards into",
"prob": 0.9245634078979492
},
{
"polygon": [
[
281.0,
1128.0
],
[
419.0,
1117.0
],
[
422.0,
1150.0
],
[
284.0,
1161.0
]
],
"text": "your deck.",
"prob": 0.938432514667511
},
{
"polygon": [
[
437.0,
1181.0
],
[
656.0,
1170.0
],
[
658.0,
1198.0
],
[
438.0,
1210.0
]
],
"text": "resistance-30",
"prob": 0.9185628890991211
},
{
"polygon": [
[
720.0,
1172.0
],
[
777.0,
1166.0
],
[
780.0,
1189.0
],
[
722.0,
1194.0
]
],
"text": "retreat",
"prob": 0.9936887621879578
},
{
"polygon": [
[
257.0,
1193.0
],
[
444.0,
1182.0
],
[
446.0,
1209.0
],
[
259.0,
1220.0
]
],
"text": "weaknessx2",
"prob": 0.9480481147766113
},
{
"polygon": [
[
556.0,
1233.0
],
[
619.0,
1226.0
],
[
621.0,
1249.0
],
[
559.0,
1255.0
]
],
"text": "Vrule",
"prob": 0.9715956449508667
},
{
"polygon": [
[
685.0,
1226.0
],
[
950.0,
1211.0
],
[
951.0,
1233.0
],
[
686.0,
1248.0
]
],
"text": "When your Pokemon V Knocked Oet",
"prob": 0.8164744973182678
},
{
"polygon": [
[
270.0,
1255.0
],
[
386.0,
1246.0
],
[
388.0,
1269.0
],
[
272.0,
1278.0
]
],
"text": "Tus NackPSa",
"prob": 0.6147907376289368
},
{
"polygon": [
[
529.0,
1266.0
],
[
791.0,
1244.0
],
[
792.0,
1266.0
],
[
531.0,
1288.0
]
],
"text": "your oppoment takes 2 Prize cards.",
"prob": 0.8714807629585266
},
{
"polygon": [
[
267.0,
1278.0
],
[
458.0,
1273.0
],
[
459.0,
1311.0
],
[
268.0,
1316.0
]
],
"text": "2126",
"prob": 0.6876019239425659
}
],
"full_text": "BASIC MewV 180 ASTRIKE EnergyMix Search your deck foran Energy cardand attach itto.lof 70 Psychic Leap You may shuffle this Pokemon and all attached cards into your deck. resistance-30 retreat weaknessx2 Vrule When your Pokemon V Knocked Oet Tus NackPSa your oppoment takes 2 Prize cards. 2126 ",
"lang": "en",
"lang_name": "english"
},
"_gpt": {
"full_prompt": "Instruction: based on the following result from ocr system what is the name and type of the card as json result ({'name':'', 'type': ''}) \nInput: BASIC MewV 180 ASTRIKE EnergyMix Search your deck foran Energy cardand attach itto.lof 70 Psychic Leap You may shuffle this Pokemon and all attached cards into your deck. resistance-30 retreat weaknessx2 Vrule When your Pokemon V Knocked Oet Tus NackPSa your oppoment takes 2 Prize cards. 2126 \nResult:",
"result": "{'name': 'Mew V', 'type': 'Basic'}",
"history": [
{
"role": "user",
"content": "Instruction: based on the following result from ocr system what is the name and type of the card as json result ({'name':'', 'type': ''}) \nInput: BASIC MewV 180 ASTRIKE EnergyMix Search your deck foran Energy cardand attach itto.lof 70 Psychic Leap You may shuffle this Pokemon and all attached cards into your deck. resistance-30 retreat weaknessx2 Vrule When your Pokemon V Knocked Oet Tus NackPSa your oppoment takes 2 Prize cards. 2126 \nResult:"
},
{
"role": "assistant",
"content": "{'name': 'Mew V', 'type': 'Basic'}"
}
],
"json_result": {
"name": "Mew V",
"type": "Basic"
}
}
}
],
"lang": "en",
"status": {
"code": 200,
"text": "OK",
"request_id": "5d735c9b-71e7-425f-a4c7-4423230b3748",
"proc_id": "b366485a-8ed9-462d-9ade-cdb817cae03a"
},
"statistics": {
"processing time": 6.970479965209961
}
}