demo-langchain-apim

Demo calling Azure OpenAI with LangChain via Azure API Management (APIM).

Update: Now includes an example using the OpenAI SDK: apim-aoai-sdk.py

Azure Environment

Your APIM will need to have Azure OpenAI REST API methods implemented (import OpenAPI spec from docs).

On your API, rename the subscription key header paramenter to api-key. Create an APIM subscription with access to the APIs.

APIM API Configuration

Note that "Open AI" endpoints are expected to end with /openai after the endpoint base URL (AZURE_OPENAI_ENDPOINT). When configuring your API in APIM Management, set the API URL Suffix to end with /openai, either just by setting it to openai or something-else/openai.

Running the Sample

To run this sample, rename .env.sample to .env and populate the values for:

AZURE_OPENAI_API_KEY: Your APIM subscription key
AZURE_OPENAI_ENDPOINT: The URL to your APIM OpenAI API

Important

The AZURE_OPENAI_ENDPOINT value should be the URL up to /openai but not include the /openai part of the URL.

Example: If your API URL is something like: https://my-apim.azure-api.net/openai/deployments/...
... then your AZURE_OPENAPI_ENDPOINT should be: https://my-apim.azure-api.net

Samples Available

apim.py: Calling the chat completions API from LangChain
apim-aoai-sdk.py: Calling using the chat completions API with the OpenAI SDK
apim.http: Calling the chat completions API directly with HTTP
- Here you can see the X-MS-Region response header which indicates the Azure region used by Azure OpenAI.
- Uses the VS Code REST Client extension.

Other Resources

Example of APIM policy to load balance AOAI across multiple regions/deployments: https://gist.github.com/ChrisRomp/f038cc233d99eaf578065723bade2a26

Load Balancing AOAI via APIM: https://github.com/ian-t-adams/azure-openai-api-m-retry -- This example walks through configuring APIM in Azure as well.

Active/Passive load balancing with APIM: https://techcommunity.microsoft.com/t5/fasttrack-for-azure/smart-load-balancing-for-openai-endpoints-and-azure-api/ba-p/3991616

Troubleshooting

The Python debugger should provide reasonable output to diagnose issues, however you may also wish to enable diagnostic logging in Azure API Management and configure it to emit Gateway logs to Azure Log Analytics or another option.

In Log Analytics, you can see APIM Gateway log messages including errors for bad requests using a query like:

ApiManagementGatewayLogs
| order by TimeGenerated desc

Or filter to failed requests with:

ApiManagementGatewayLogs
| where not(IsRequestSuccess)
| order by TimeGenerated desc

chrisromp / demo-langchain-apim Goto Github PK