Code Monkey home page Code Monkey logo

gpt-azure-search-engine's Introduction

image

3 or 5 days POC VBD powered by: Azure AI Search + Azure OpenAI + Bot Framework + Langchain + Azure SQL + CosmosDB + Bing Search API + Document Intelligence SDK

Open in GitHub Codespaces Open in VS Code Dev Containers

Your organization requires a Multi-Channel Smart Chatbot and a search engine capable of comprehending diverse types of data scattered across various locations. Additionally, the conversational chatbot should be able to provide answers to inquiries, along with the source and an explanation of how and where the answer was obtained. In other words, you want private and secured ChatGPT for your organization that can interpret, comprehend, and answer questions about your business data.

The goal of the POC is to show/prove the value of a GPT Virtual Assistant built with Azure Services, with your own data in your own environment. The deliverables are:

  1. Backend Bot API built with Bot Framework and exposed to multiple channels (Web Chat, MS Teams, SMS, Email, Slack, etc)
  2. Frontend web application with a Search and a Bot UI.

The repo is made to teach you step-by-step on how to build a OpenAI-based Smart Search Engine. Each Notebook builds on top of each other and ends in building the two applications.

For Microsoft FTEs: This is a customer funded VBD, below the assets for the delivery.

Item Description Link
VBD SKU Info and Datasheet CSAM must dispatch it as "Customer Invested" against credits/hours of Unified Support Contract. Customer decides if 3 or 5 days. ESXP SKU page
VBD Accreditation for CSAs Links for CSAs to get the Accreditation needed to deliver the workshop Link 1 , Link 2
VBD 3-5 day POC Asset (IP) The MVP to be delivered (this GitHub repo) Azure-Cognitive-Search-Azure-OpenAI-Accelerator
VBD Workshop Deck The deck introducing and explaining the workshop Intro AOAI GPT Azure Smart Search Engine Accelerator.pptx
CSA Training Video 2 Hour Training for Microsoft CSA's POC VBD Training Recording (New video coming soon!)

Prerequisites Client 3-5 Days POC

  • Azure subscription
  • Accepted Application to Azure Open AI, including GPT-4. If customer does not have GPT-4 approved, Microsoft CSAs can lend theirs during the workshop
  • Microsoft members preferably to be added as Guests in clients Azure AD. If not possible, then customers can issue corporate IDs to Microsoft members
  • A Resource Group (RG) needs to be set for this Workshop POC, in the customer Azure tenant
  • The customer team and the Microsoft team must have Contributor permissions to this resource group so they can set everything up 2 weeks prior to the workshop
  • A storage account must be set in place in the RG.
  • Customer Data/Documents must be uploaded to the blob storage account, at least two weeks prior to the workshop date
  • A Multi-Tenant App Registration (Service Principal) must be created by the customer (save the Client Id and Secret Value).
  • Customer must provide the Microsoft Team , 10-20 questions (easy to hard) that they want the bot to respond correctly.
  • For IDE collaboration and standarization during workshop, AML compute instances with Jupyper Lab will be used, for this, Azure Machine Learning Workspace must be deployed in the RG
    • Note: Please ensure you have enough core compute quota in your Azure Machine Learning workspace

Architecture

Architecture

Flow

  1. The user asks a question.
  2. In the app, an OpenAI GPT-4 LLM uses a clever prompt to determine which source to use based on the user input
  3. Five types of sources are available:
    • 3a. Azure SQL Database - contains COVID-related statistics in the US.
    • 3b. API Endpoints - RESTful OpenAPI 3.0 API containing up-to-date statistics about Covid.
    • 3c. Azure Bing Search API - provides access to the internet allowing scenerios like: QnA on public websites .
    • 3d. Azure AI Search - contains AI-enriched documents from Blob Storage:
      • 10,000 Arxiv Computer Science PDFs
      • 90,000 Covid publication abstracts
      • 5 lenghty PDF books
    • 3f. CSV Tabular File - contains COVID-related statistics in the US.
    • 3g. Kraken broker API for currencies
  4. The app retrieves the result from the source and crafts the answer.
  5. The tuple (Question and Answer) is saved to CosmosDB as persistent memory and for further analysis.
  6. The answer is delivered to the user.

Demo

https://gptsmartsearchapp.azurewebsites.net/


🔧Features

  • 100% Python.
  • Uses Azure Cognitive Services to index and enrich unstructured documents: OCR over images, Chunking and automated vectorization.
  • Uses Hybrid Search Capabilities of Azure AI Search to provide the best semantic answer (Text and Vector search combined).
  • Uses LangChain as a wrapper for interacting with Azure OpenAI , vector stores, constructing prompts and creating agents.
  • Multi-Lingual (ingests, indexes and understand any language)
  • Multi-Index -> multiple search indexes
  • Tabular Data Q&A with CSV files and SQL flavor Databases
  • Uses Azure AI Document Intelligence SDK (former Form Recognizer) to parse complex/large PDF documents
  • Uses Bing Search API to power internet searches and Q&A over public websites.
  • Connects to API Data sources by converting natural language questions to API calls.
  • Uses CosmosDB as persistent memory to save user's conversations.
  • Uses Streamlit to build the Frontend web application in python.
  • Uses Bot Framework and Bot Service to Host the Bot API Backend and to expose it to multiple channels including MS Teams.
  • Uses also LangServe/FastAPI to deploy an alternative backend API with streaming capabilites

Steps to Run the POC/Accelerator

Note: (Pre-requisite) You need to have an Azure OpenAI service already created

  1. Fork this repo to your Github account.

  2. In Azure OpenAI studio, deploy these models (older models than the ones stated below won't work):

    • "gpt-35-turbo-1106 (or newer)"
    • "gpt-4-turbo-1106 (or newer)"
    • "text-embedding-ada-002 (or newer)"
  3. Create a Resource Group where all the assets of this accelerator are going to be. Azure OpenAI can be in different RG or a different Subscription.

  4. ClICK BELOW to create all the Azure Infrastructure needed to run the Notebooks (Azure AI Search, Cognitive Services, etc):

    Deploy To Azure

    Note: If you have never created a Azure AI Services Multi-Service account before, please create one manually in the azure portal to read and accept the Responsible AI terms. Once this is deployed, delete this and then use the above deployment button.

  5. Clone your Forked repo to your AML Compute Instance. If your repo is private, see below in Troubleshooting section how to clone a private repo.

  6. Make sure you run the notebooks on a Python 3.10 conda enviroment or newer

  7. Install the dependencies on your machine (make sure you do the below pip comand on the same conda environment that you are going to run the notebooks. For example, in AZML compute instance run:

    conda activate azureml_py310_sdkv2
    pip install -r ./common/requirements.txt
    

    You might get some pip dependancies errors, but that is ok, the libraries were installed correctly regardless of the error.

  8. Edit the file credentials.env with your own values from the services created in step 4.

    • For BLOB_SAS_TOKEN and BLOB_CONNECTION_STRING. Go to Storage Account>Security + networking>Shared access signature>Generate SAS
  9. Run the Notebooks in order. They build up on top of each other.


FAQs

FAQs

  1. Why use Azure AI Search engine to provide the context for the LLM and not fine tune the LLM instead?

A: Quoting the OpenAI documentation: "GPT-3 has been pre-trained on a vast amount of text from the open internet. When given a prompt with just a few examples, it can often intuit what task you are trying to perform and generate a plausible completion. This is often called "few-shot learning. Fine-tuning improves on few-shot learning by training on many more examples than can fit in the prompt, letting you achieve better results on a wide number of tasks. Once a model has been fine-tuned, you won't need to provide examples in the prompt anymore. This saves costs and enables lower-latency requests"

However, fine-tuning the model requires providing hundreds or thousands of Prompt and Completion tuples, which are essentially query-response samples. The purpose of fine-tuning is not to give the LLM knowledge of the company's data but to provide it with examples so it can perform tasks really well without requiring examples on every prompt.

There are cases where fine-tuning is necessary, such as when the examples contain proprietary data that should not be exposed in prompts or when the language used is highly specialized, as in healthcare, pharmacy, or other industries or use cases where the language used is not commonly found on the internet.

Troubleshooting

Troubleshooting

Steps to clone a private repo:

ssh-keygen -t ed25519 -C "[email protected]"
cat ~/.ssh/id_ed25519.pub
# Then select and copy the contents of the id_ed25519.pub file
# displayed in the terminal to your clipboard
  • On GitHub, go to Settings-> SSH and GPG Keys-> New SSH Key
  • In the "Title" field, add a descriptive label for the new key. "AML Compute". In the "Key" field, paste your public key.
  • Clone your private repo
git clone [email protected]:YOUR-USERNAME/YOUR-REPOSITORY.git

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

gpt-azure-search-engine's People

Contributors

amitmukh avatar dantelmomsft avatar erjadi avatar filipefumaux avatar giorgiosaez avatar github-actions[bot] avatar juliaashleyh8 avatar keayoub avatar lordaouy avatar maurominella avatar pablomarin avatar sethsteenken avatar tbunkley avatar tectonia avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

gpt-azure-search-engine's Issues

Missing credentials.py

On 01-Load-Data-ACogSearch.ypynb it mentions a credentials.py on the directory app:
"Create a data source
You should already have a blob container that contains the sample data, see app/credentials.py"
That file doesn't seem to be present anywhere.

credentials exposed

Great service, but looks like you're exposing your keys checking in credentials.py

Backend Webapp not able to spinup

While deploying earlier backend webapp was failing with container crash issue being exception: ModuleNotFoundError: No Module named ‘aiohttp’. When I redeployed again with python 3.10 this issue is resolved.

As of now container is not crashing and getting following warning:

File "/tmp/8db6d8a271372eb/antenv/lib/python3.10/site-packages/botbuilder/core/integration/aiohttp_channel_service_exception_middleware.py", line 20, in aiohttp_error_middleware
response = await handler(request)
File "/tmp/8db6d8a271372eb/antenv/lib/python3.10/site-packages/aiohttp/web_urldispatcher.py", line 931, in _handle
raise self._http_exception
aiohttp.web_exceptions.HTTPNotFound: Not Found
Task exception was never retrieved
future: exception=KeyError('access_token')>. These exceptions are coming from application. Any help in this regard would be much apprciated..Thanks:-)

Citations for sites indexed via CSV

Hi Pablo, can you provide any guidance around how you configured the web app to structure and citate the URL for the sites in the ncbi data set? Also, if you have any guidance on how you built the CSV from the ncbi website. Looking to do something similar for ServiceNow KB articles. Any input would be greatly appreciated! Awesome project.

Create Indexer REST API Call Fails with Data Source Auth Issue

Rest API call to create indexer in 01-Load-Data-ACogSearch.ipynb with the default arxivcs container with:

"error":{"code":"","message":"Error with data source: Credentials provided in the connection string are invalid or have expired.\r\nFor more information on troubleshooting connection issues to Azure Storage accounts, please see https://go.microsoft.com/fwlink/?linkid=2049388 Please adjust your data source definition in order to proceed."

Potentially a firewall issue but couldn't troubleshoot locally. SAS Token Auth also fails in Storage Explorer.

AttributeError: 'NoneType' object has no attribute 'get'

Hi All,

Running 01-Lod-Data-AcogSearch.ipynb

Change these below with your own services credentials

AZURE_SEARCH_ENDPOINT = "" <= URL of Search Service
AZURE_SEARCH_KEY = " " <= Keys Manage admin keys
COG_SERVICES_NAME = "" <= name of search service, e.g cog-search-xxxx
COG_SERVICES_KEY = "" <= cog-search-xxxx , this is manage query keys

I have added like the above and
400 false


AttributeError Traceback (most recent call last)
Cell In[27], line 6
4 # pprint(json.dumps(r.json(), indent=1))
5 print(r.status_code)
----> 6 print("Status:",r.json().get('lastResult').get('status'))
7 print("Items Processed:",r.json().get('lastResult').get('itemsProcessed'))
8 print(r.ok)

AttributeError: 'NoneType' object has no attribute 'get'. anyone seen this erro ?

HTTP status code NotFound at Backend Bot service

I'm trying to deploy my backend and fronted webiste to Azure App Services and Bot Service, however, the frontend website was successfully deployed but the backend doesn't. The bakend is not wokring with error There was an error sending this message to your bot: HTTP status code NotFound, it's both not working with Web Chat and Direct Line.
I've passed all the notebook files successfully but troubled while deploying to App Service, please help and thanks.
image

Can i run only note book 7 and deploy to check bing API

Can I run only note book 7 and deploy to check bing API.
What are the steps to do it.
As per ReadmMe it says it is built on running all the notebooks in order. Is it possible to deploy only notebook 7 features.
Please let me know what are the steps
I am able to run Notebook 7 on visual studio notebook.
But for testing purposes it need to be deployed right ? What is the procedure ?

Validation failed. View error details

Hi,

I tried to deploy with the custom template to create all the Azure Infrastructure needed to run the Notebooks (Azure Cognitive Search, Cognitive Services, SQL Database, CosmosDB, Bing Search API), but I got the following error message.

"InvalidTemplateDeployment: The template deployment 'MicrosoftTemplate' is not valid according to the validation procedure. The tracking id is '2d765e2c-4e79-4062-a78a-c26e02d13xxx. See inner errors for details.

Does anyone have the same problem before? Thanks

Bing search for the demo link

Excellent architecture and description on the work flow / Read me. Thanks for providing details and quick resolutions I have a quick question.
I am trying to do a bing search in the demo link provided.
I am getting the following information. Can you please let me know how to test the demo link for bing search
I am getting the following attached answers for bing search. Looks like it does Cognitive search alone on the internal data and hence does not provide the answer from the internet
Bing Search

Cannot hit the chatbot programmatically using Python

Hello,

I'm trying to develop the following Python program that would hit the created Bot Service programmatically

import requests
import time


direct_line_secret = "XXXXX"
base_url = "https://directline.botframework.com/v3/directline"

# Start a conversation
start_conversation_url = f"{base_url}/conversations"
headers = {
    "Authorization": f"Bearer {direct_line_secret}"
}

response = requests.post(start_conversation_url, headers=headers)
conversation_id = response.json()["conversationId"]
print('Converstion id:', conversation_id)

# Send a message to the bot
send_message_url = f"{base_url}/conversations/{conversation_id}/activities"
message = {
    "type": "message",
    "from": {"id": "user"},
    "text": "what is Reinforcement Learning?"
}

response = requests.post(send_message_url, headers=headers, json=message)
print(response.status_code)
print(response.text)

# Poll for the bot's response
get_activities_url = f"{base_url}/conversations/{conversation_id}/activities"

while True:
    response = requests.get(get_activities_url, headers=headers)
    activities = response.json()["activities"]
    print(activities)
    for activity in activities[::-1]:
        if activity["from"]["id"] == "bot_id":  # Replace "your_bot_id" with the actual bot's ID
            print("Bot's response:", activity["text"])
            exit()  # or break the loop if you only want the latest message

    # Wait for a while before polling again
    time.sleep(1)

However, it gives me Bot's response: To continue to run this bot, please fix the bot source code. but Test in Web Chat works just fine. Also checking the logs I saw the following:

  File "/tmp/8dbd09dbf228fa9/antenv/lib/python3.10/site-packages/botbuilder/core/middleware_set.py", line 79, in receive_activity_internal
    return await callback(context)
  File "/tmp/8dbd09dbf228fa9/antenv/lib/python3.10/site-packages/botbuilder/core/activity_handler.py", line 70, in on_turn
    await self.on_message_activity(turn_context)
  File "/tmp/8dbd09dbf228fa9/bot.py", line 74, in on_message_activity
    input_text_metadata["local_timestamp"] = turn_context.activity.local_timestamp.strftime("%I:%M:%S %p, %A, %B %d of %Y")
AttributeError: 'NoneType' object has no attribute 'strftime'

Is there anything I'm doing wrong in the code and I do not pass the parameters correctly or there is an issue in bot's code?

Can't deploy App Service

  • Trial 1: az webapp deployment, Git Actions, Local git
    ==> Got azure deploy deployment conflict (code: 409)

  • Trial 2: create template on portal
    ==> Got timeout

Test commit hash: 2c10f7b

Playing with the model, interesting outcome...

Playing with the Streamlit model, I was asking the recommended question "list the authors that talk about Boosting Algorithms" and I got (what I assume) is the expected response - "The authors that talk about Boosting Algorithms are Ninan Sajeeth Philip and K. Babu Joseph, and Schapire.". However, if I start asking about dates, it doesn't return anything... such as: "How do Ninan Sajeeth Philip papers relate to booting algorithms?" or "when did Ninan Sajeeth Philip write papers relating to booting algorithms?".

The model doesn't seem to know anything about dates. Does this have to do with how the data model was built?

KeError when submitting QuickAnswer or BEst Answer button

KeyError: 'value'
Traceback:
File "/tmp/8db30dc93843905/antenv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
exec(code, module.dict)
File "/tmp/8db30dc93843905/app/main.py", line 116, in
for result in search_results['value']:
image

AttributeError: 'str' object has no attribute 'items'

I tried to follow the guide, did the pip install based on the requirements file and ran the "streamlit Home.py" command. However, I am getting this error: Traceback (most recent call last):
File "/home/linuxbrew/.linuxbrew/bin/streamlit", line 8, in
sys.exit(main())
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/streamlit/web/cli.py", line 177, in main_run
bootstrap.load_config_options(flag_options=kwargs)
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/streamlit/web/bootstrap.py", line 342, in load_config_options
config.get_config_options(force_reparse=True, options_from_flags=options_from_flags)
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/streamlit/config.py", line 1144, in get_config_options
_update_config_with_toml(file_contents, filename)
File "/home/linuxbrew/.linuxbrew/opt/[email protected]/lib/python3.9/site-packages/streamlit/config.py", line 1025, in _update_config_with_toml
for name, value in options.items():
AttributeError: 'str' object has no attribute 'items'

SAS TOKEN Unable to find

Not data returned from Azure Search, check connection..

name 'content' is not defined

KeyError: 'chunk'
Traceback:
File "/tmp/8dbb8b7be883dca/antenv/lib/python3.10/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script
exec(code, module.dict)
File "/tmp/8dbb8b7be883dca/pages/1_Search.py", line 130, in
top_docs.append(Document(page_content=value["chunk"], metadata={"source": location+os.environ['BLOB_SAS_TOKEN']}))

Timeout after 15s, 502 Error GATEWAY TIMEOUT

Hello,

Tested all notebooks and ran fine, deployed the backend web app with no errors but when testing it i get 'To continue to run this bot, please fix the bot source code. ' WHen i look at the Bot channels, both direct line & web chat have the same message for healthy status(issues): 'There was an error sending this message to your bot: HTTP status code GatewayTimeout' & when i do inspect in browser i see the POST call https://directline.botframework.com/v3/directline/conversations/XYZ-eu/activities with response:
{
"error": {
"code": "BotError",
"message": "Failed to send activity: bot timed out"
},
"httpStatusCode": 504
}

The same bot works in teams, but not in the frontend app, i also looked at all the backend webapp & frontend webapp configuration and checked all the values, including the BOT_DIRECTLINE_SECRET_KEY and it`s correct

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.