Skip to content

Commit 4f1c4a7

Browse files
[formrecognizer] Adding custom forms perf test (#16969)
* Adding custom forms perf test * improve test to use labeled model * add README * update test name * use async training client * rename file * fix test check * update readme cmd instructions * update readme
1 parent 259934c commit 4f1c4a7

File tree

3 files changed

+118
-0
lines changed

3 files changed

+118
-0
lines changed
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Form Recognizer Performance Tests
2+
3+
In order to run the performance tests, the `azure-devtools` package must be installed. This is done as part of the `dev_requirements` install.
4+
Start by creating a new virtual environment for your perf tests. This will need to be a Python 3 environment, preferably >=3.7.
5+
6+
### Setup for test resources
7+
8+
The following environment variables will need to be set for the tests to access the live resources:
9+
10+
```
11+
FORMRECOGNIZER_TEST_ENDPOINT=<form recognizer service endpoint>
12+
FORMRECOGNIZER_TEST_API_KEY=<form recognizer API Key>
13+
FORMRECOGNIZER_TRAINING_DATA_CONTAINER_SAS_URL=<SAS url for container with training data>
14+
```
15+
16+
### Setup for perf test runs
17+
18+
```cmd
19+
(env) ~/azure-ai-formrecognizer> pip install -r dev_requirements.txt
20+
(env) ~/azure-ai-formrecognizer> pip install -e .
21+
```
22+
23+
## Test commands
24+
25+
Once `azure-devtools` is installed, you will have access to the `perfstress` command line tool, which will scan the current module for runable perf tests. Only a specific test can be run at a time (i.e. there is no "run all" feature).
26+
27+
```cmd
28+
(env) ~/azure-ai-formrecognizer> cd tests/perfstress_tests/
29+
(env) ~/azure-ai-formrecognizer/tests/perfstress_tests> perfstress
30+
```
31+
Using the `perfstress` command alone will list the available perf tests found.
32+
33+
### Common perf command line options
34+
These options are available for all perf tests:
35+
- `--duration=10` Number of seconds to run as many operations (the "run" function) as possible. Default is 10.
36+
- `--iterations=1` Number of test iterations to run. Default is 1.
37+
- `--parallel=1` Number of tests to run in parallel. Default is 1.
38+
- `--warm-up=5` Number of seconds to spend warming up the connection before measuring begins. Default is 5.
39+
- `--sync` Whether to run the tests in sync or async. Default is False (async). This flag must be used for Storage legacy tests, which do not support async.
40+
- `--no-cleanup` Whether to keep newly created resources after test run. Default is False (resources will be deleted).
41+
42+
## Example command
43+
```cmd
44+
(env) ~/azure-ai-formrecognizer/tests/perfstress_tests> perfstress RecognizeCustomForms
45+
```

sdk/formrecognizer/azure-ai-formrecognizer/tests/perfstress_tests/__init__.py

Whitespace-only changes.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# coding=utf-8
2+
# ------------------------------------
3+
# Copyright (c) Microsoft Corporation.
4+
# Licensed under the MIT License.
5+
# ------------------------------------
6+
7+
import os
8+
import pytest
9+
import functools
10+
from io import BytesIO
11+
from datetime import date, time
12+
from azure_devtools.perfstress_tests import PerfStressTest
13+
from azure.core.credentials import AzureKeyCredential
14+
from azure.ai.formrecognizer import FormRecognizerClient, FormContentType
15+
from azure.ai.formrecognizer.aio import FormRecognizerClient as AsyncFormRecognizerClient, FormTrainingClient as AsyncFormTrainingClient
16+
17+
class RecognizeCustomForms(PerfStressTest):
18+
19+
def __init__(self, arguments):
20+
super().__init__(arguments)
21+
22+
with open(os.path.abspath(os.path.join(os.path.abspath(__file__), "..", "./../sample_forms/forms/Form_1.jpg")), "rb") as fd:
23+
self.custom_form_jpg = fd.read()
24+
25+
# read test related env vars
26+
self.formrecognizer_storage_container_sas_url = os.environ["FORMRECOGNIZER_TRAINING_DATA_CONTAINER_SAS_URL"]
27+
formrecognizer_test_endpoint = os.environ["FORMRECOGNIZER_TEST_ENDPOINT"]
28+
form_recognizer_account_key = os.environ["FORMRECOGNIZER_TEST_API_KEY"]
29+
30+
# assign the clients that will be used in the perf tests
31+
self.service_client = FormRecognizerClient(formrecognizer_test_endpoint, AzureKeyCredential(form_recognizer_account_key))
32+
self.async_service_client = AsyncFormRecognizerClient(formrecognizer_test_endpoint, AzureKeyCredential(form_recognizer_account_key))
33+
34+
# training client will be used for model training in set up
35+
self.async_training_client = AsyncFormTrainingClient(formrecognizer_test_endpoint, AzureKeyCredential(form_recognizer_account_key))
36+
37+
async def global_setup(self):
38+
"""The global setup is run only once."""
39+
poller = await self.async_training_client.begin_training(
40+
self.formrecognizer_storage_container_sas_url,
41+
use_training_labels=True,
42+
model_name="labeled")
43+
model = await poller.result()
44+
self.model_id = model.model_id
45+
46+
async def global_cleanup(self):
47+
"""The global cleanup is run only once."""
48+
await self.async_training_client.delete_model(self.model_id)
49+
50+
async def close(self):
51+
"""This is run after cleanup."""
52+
await self.async_service_client.close()
53+
self.service_client.close()
54+
await self.async_training_client.close()
55+
await super().close()
56+
57+
def run_sync(self):
58+
"""The synchronous perf test."""
59+
poller = self.service_client.begin_recognize_custom_forms(
60+
self.model_id,
61+
self.custom_form_jpg,
62+
content_type=FormContentType.IMAGE_JPEG)
63+
result = poller.result()
64+
assert result
65+
66+
async def run_async(self):
67+
"""The asynchronous perf test."""
68+
poller = await self.async_service_client.begin_recognize_custom_forms(
69+
self.model_id,
70+
self.custom_form_jpg,
71+
content_type=FormContentType.IMAGE_JPEG)
72+
result = await poller.result()
73+
assert result

0 commit comments

Comments
 (0)