Azure Form Recognizer

Azure Form Recognizer is a cognitive service that uses machine learning technology to identify and extract text, key/value pairs, selection marks, tables, and structure from documents. We have the option of using prebuilt models (invoices, sales receipt and business cards) or adding custom templates to make the recognition process more accurate, even using complicated documents.

For this blog I am going to integrate the prebuilt receipt recognition model with APEX. The app will have the option to upload the receipt picture and detect the items, singles values and even the total value.

Getting Started

If you are new to Azure you can get started with a free subscription on the Azure Portal. When logged in to the portal, open the left menu and click on “Create a Resource”, search for “Form Recognizer” from the AI + Machine Leaning Category. The free tier allows us to perform up to 500 calls a month, which is more than enough to use on this demo.

Analyse Layout

To analyse the receipt, we make a REST call to the API . This will detect and extract data using optical character recognition (OCR). The input document must be of one of the supported content types - 'application/pdf', 'image/jpeg', 'image/png' or 'image/tiff'. Alternatively, we can use 'application/json' type to specify the URL location of the document file to be analysed.

In the following example, we upload the receipt picture using a file upload item called P1_RECEIPT_FILE, and we must store the response ID to use later in the second API call, for this I am using the P1_REQUEST_ID,

DECLARE

l_clob        CLOB;

l_body        CLOB;

l_receipt     BLOB;

l_request_id  VARCHAR2(4000);



BEGIN


apex_web_service.g_request_headers(1).name := 'Ocp-Apim-Subscription-Key';  


apex_web_service.g_request_headers(1).value := ' <add the API KEY here>   ';
apex_web_service.g_request_headers(2).name := 'Content-Type';  

apex_web_service.g_request_headers(2).value := 'application/octet-stream'; 




SELECT blob_content

INTO l_receipt

FROM apex_application_temp_files

WHERE NAME = :P1_RECEIPT_FILE;

l_clob := apex_web_service.make_rest_request(

p_url => 'https://northeurope.api.cognitive.microsoft.com/formreco
gnizer/v2.0/prebuilt/receipt/analyze',

p_http_method => 'POST',

p_body_blob => l_receipt);



:P1_REQUEST_ID := apex_web_service.g_headers(4).value;



exception

when others then



apex_error.add_error (

p_message => sqlerrm,

p_display_location => apex_error.c_inline_in_notification );



end;

With the response ID in hand, we can request the document information. Usually the analysis process is very fast, but it can take a few seconds depending on the complexity of the document. We will store the returned data into a APEX collection called COL_JSON_OBJECT.

Get Analyse Layout Result

DECLARE
l_clob CLOB;
l_body CLOB;
l_receipt BLOB;
l_request_id VARCHAR2(4000);

BEGIN
apex_web_service.g_request_headers(1).name := 'Ocp-Apim-Subscription-Key'; 
apex_web_service.g_request_headers(1).value := ' <add the API KEY here> ';
apex_web_service.g_request_headers(2).name := 'Content-Type'; 
apex_web_service.g_request_headers(2).value := 'application/octet-stream'; 


l_clob := apex_web_service.make_rest_request(
p_url =>  'https://northeurope.api.cognitive.microsoft.com/formreco
gnizer/v2.0/prebuilt/receipt/analyzeResults/'||
:P1_REQUEST_ID,
p_http_method => 'GET',
p_body => '{body}');

APEX_COLLECTION.CREATE_OR_TRUNCATE_COLLECTION(
p_collection_name => 'COL_JSON_OBJECT');

APEX_COLLECTION.ADD_MEMBER(
p_collection_name => 'COL_JSON_OBJECT',
P_C001 => 'receipt.json', 
p_CLOB001 => l_clob,
p_BLOB001 => clob2blob(l_clob)); 

exception
when others then

DBMS_OUTPUT.PUT_LINE(SQLERRM);

end;

The API returns the result in JSON format with a lot of additional information.

Using the apex_data_parser we can transform the JSON and show the data on an interactive report.

/* get the items/prices */

select prod_description.Col006 Product,
       prod_description.Col008 Price
from apex_collections f, 
       table( apex_data_parser.parse(
                  p_content => blob001,
                  p_add_headers_row => 'Y',
                  --
                  p_row_selector =>                      
'analyzeResult.documentResults.fields.Items.valueArray',
                  --
                  p_store_profile_to_collection => 'FILE_PARSER_COLLECTION',
                  p_file_name => c001 ) ) prod_description
where collection_name = 'COL_JSON_OBJECT'
and line_number is not null
order by line_number 

/* get the total */

select prod_description.Col002 total
from apex_collections f, 
table( apex_data_parser.parse(
           p_content => blob001,
           p_add_headers_row => 'Y',
           --
           p_row_selector => 'analyzeResult.documentResults.fields.Total',
           --
           p_store_profile_to_collection => 'FILE_PARSER_COLLECTION',
           p_file_name => c001 ) ) prod_description
where collection_name = 'COL_JSON_OBJECT'
and line_number is not null
order by line_number 

Now we can compare the returned information from the API and the actual receipt. Note that I filtered to only show the products and values, but information such as the store name and address are also returned inside the JSON.

The Microsoft Azure Form Recognizer is very powerful and a great call if you need to use machine learning to extract information from documents in an APEX app. It is easy to integrate and can also detect custom documents.

Author: Rodrigo Mesquita 

Job Title: Oracle APEX Development Consultant

Bio: Rodrigo is an Oracle ACE and APEX Developer Expert certified by Oracle and is an experienced software engineer with emphasis in analysis, design and development of bespoke Oracle applications utilising development tools such as PL/SQL, APEX and Forms for a variety of international businesses. Rodrigo speaks regularly at Oracle community events and enjoys creating APEX plugins and writing blogs.