Demo calling Azure OpenAI with LangChain via Azure API Management (APIM).
Update: Now includes an example using the OpenAI SDK: apim-aoai-sdk.py
Your APIM will need to have Azure OpenAI REST API methods implemented (import OpenAPI spec from docs).
On your API, rename the subscription key header paramenter to api-key
. Create an APIM subscription with access to the APIs.
Note that "Open AI" endpoints are expected to end with /openai
after the endpoint base URL (AZURE_OPENAI_ENDPOINT
). When configuring your API in APIM Management, set the API URL Suffix
to end with /openai
, either just by setting it to openai
or something-else/openai
.
To run this sample, rename .env.sample
to .env
and populate the values for:
AZURE_OPENAI_API_KEY
: Your APIM subscription keyAZURE_OPENAI_ENDPOINT
: The URL to your APIM OpenAI API
Important
The AZURE_OPENAI_ENDPOINT
value should be the URL up to /openai
but not include the /openai
part of the URL.
Example: If your API URL is something like:
https://my-apim.azure-api.net/openai/deployments/...
... then your AZURE_OPENAPI_ENDPOINT
should be:
https://my-apim.azure-api.net
apim.py
: Calling the chat completions API from LangChainapim-aoai-sdk.py
: Calling using the chat completions API with the OpenAI SDKapim.http
: Calling the chat completions API directly with HTTP- Here you can see the
X-MS-Region
response header which indicates the Azure region used by Azure OpenAI. - Uses the VS Code REST Client extension.
- Here you can see the
Example of APIM policy to load balance AOAI across multiple regions/deployments: https://gist.github.com/ChrisRomp/f038cc233d99eaf578065723bade2a26
Load Balancing AOAI via APIM: https://github.com/ian-t-adams/azure-openai-api-m-retry -- This example walks through configuring APIM in Azure as well.
Active/Passive load balancing with APIM: https://techcommunity.microsoft.com/t5/fasttrack-for-azure/smart-load-balancing-for-openai-endpoints-and-azure-api/ba-p/3991616
The Python debugger should provide reasonable output to diagnose issues, however you may also wish to enable diagnostic logging in Azure API Management and configure it to emit Gateway logs to Azure Log Analytics or another option.
In Log Analytics, you can see APIM Gateway log messages including errors for bad requests using a query like:
ApiManagementGatewayLogs
| order by TimeGenerated desc
Or filter to failed requests with:
ApiManagementGatewayLogs
| where not(IsRequestSuccess)
| order by TimeGenerated desc