Move credentials check and initialization to config, rather than randomly all over the code

Even when the .env is populated, the following error is raised when running 02_qa_generation.py locally

    from ingest_data.acs_ingest import generate_qna
  File "/Users/shanepeckham/sources/rag-experiment-accelerator/ingest_data/acs_ingest.py", line 9, in <module>
    from llm.prompt_execution import generate_response
  File "/Users/shanepeckham/sources/rag-experiment-accelerator/llm/prompt_execution.py", line 8, in <module>
    openai.api_key = os.environ['OPENAI_API_KEY']
  File "/Users/shanepeckham/opt/miniconda3/envs/rag-test/lib/python3.10/os.py", line 680, in __getitem__
    raise KeyError(key) from None
KeyError: 'OPENAI_API_KEY'

Readme:
Replace
python -m pip install -r requirements.txt
with
python -m pip install -f requirements.txt

Shall we also note that on fresh machine it'll trigger error:
"Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools"

Minimal selection:

Under the "Workloads" tab, check the box for "Desktop development with C++."
Under the "Individual components" tab, make sure the following components are selected:
"Windows 10 SDK"
"MSVC v142 - VS 2019 C++ x64/x86 build tools"
"C++ ATL for latest v142 build tools (x86 & x64)"
"C++ MFC for latest v142 build tools (x86 & x64)"

logger should show output to console as well

Files like generated_index_names and other artifacts created in runtime appear as a difference on commit.
Let's move them all to a separate folder and gitignore it.

Update the files that are downloading spacy and replace that with spacy download to be performed only once for the project.

          You could also check existence of models at this stage. It can be done like this

from azure.core.exceptions import ResourceNotFoundError
from azure.ai.openai import OpenAIApiClient

client = OpenAIApiClient()

try:
deployment = client.deployments.get("your-deployment-name")
print("Deployment exists.")
except ResourceNotFoundError:
print("Deployment does not exist.")

Originally posted by @WVadim in #58 (comment)

Currently OPENAI_EMBEDDING_DEPLOYED_MODEL is declared in .env file, while model used for chat completions is declared in config file. Anticipating more models in the future, it makes sense to move all model names into config under section "nlp_models" or something similar.

Migrate non-Open Source or non-External Collaboration repositories to GitHub inside Microsoft

In order to protect and secure Microsoft, private or internal repositories in GitHub for Open Source which are not related to open source projects or require collaboration with 3rd parties (customer, partners, etc.) must be migrated to GitHub inside Microsoft a.k.a GitHub Enterprise Cloud with Enterprise Managed User (GHEC EMU).

Action

✍️ Please RSVP to opt-in or opt-out of the migration to GitHub inside Microsoft.

❗Only users with admin permission in the repository are allowed to respond. Failure to provide a response will result to your repository getting automatically archived.🔒

Instructions

Reply with a comment on this issue containing one of the following optin or optout command options below.

✅ Opt-in to migrate

@gimsvc optin --date <target_migration_date in mm-dd-yyyy format>

Example: @gimsvc optin --date 03-15-2023

OR

❌ Opt-out of migration

@gimsvc optout --reason <staging|collaboration|delete|other>

Example: @gimsvc optout --reason staging

Options:

• staging : This repository will ship as Open Source or go public
• collaboration : Used for external or 3rd party collaboration with customers, partners, suppliers, etc.
• delete : This repository will be deleted because it is no longer needed.
• other : Other reasons not specified

Need more help? 🖐️

• Email [email protected]. ✉️
• Post your questions in GitHub inside Microsoft Team in Microsoft Teams. 🗨️

Update 04_evaluation.py to run with config class instead of reading from config file directly.
Refer to 03_qa_generation.py

Place load_dotenv() before the imports of the code that references the env variables

Possible data connectors:

• HTML web pages
• Azure Blob Storage
• Sharepoint
• md from repos

We should validate existence of all models and keys required for a particular step before executing the step. At the moment, in step 01_Index.py of model chat_model_name does not exist, we will build a bunch of embeddings only to fail at the last line of the script.
This logic should also be propagated to other scripts, in order to save user's resources and time

Add code to load markdown files

Should include prompt templates within the RAG Experiment Accelerator to allow the user to experiment.
Current prompt templates are located in llm\prompts.py

I encountered such error when running 01_Index.py file following the guidance in readme.md. After some debugging, I figured out it is because the data folder is empty.
Can you pls add some instructions in readme to let customers know they should add pdfs in data folder then run 01_index.py?

Traceback (most recent call last):
File "C:\Users\yijunzhang\Documents\code_repo\rag-experiment-accelerator\01_Index.py", line 72, in
upload_data(data_load,service_endpoint,index_name,key, dimension, chat_model_name, temperature)
File "C:\Users\yijunzhang\Documents\code_repo\rag-experiment-accelerator\ingest_data\acs_ingest.py", line 46, in upload_data
results = search_client.upload_documents(documents)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yijunzhang\Anaconda3\envs\fhl\Lib\site-packages\azure\search\documents_search_client.py", line 543, in upload_documents
results = self.index_documents(batch, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yijunzhang\Anaconda3\envs\fhl\Lib\site-packages\azure\core\tracing\decorator.py", line 76, in wrapper_use_tracer
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yijunzhang\Anaconda3\envs\fhl\Lib\site-packages\azure\search\documents_search_client.py", line 641, in index_documents
return self._index_documents_actions(actions=batch.actions, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yijunzhang\Anaconda3\envs\fhl\Lib\site-packages\azure\search\documents_search_client.py", line 649, in _index_documents_actions
batch_response = self._client.documents.index(batch=batch, error_map=error_map, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yijunzhang\Anaconda3\envs\fhl\Lib\site-packages\azure\core\tracing\decorator.py", line 76, in wrapper_use_tracer
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\yijunzhang\Anaconda3\envs\fhl\Lib\site-packages\azure\search\documents_generated\operations_documents_operations.py", line 1264, in index
raise HttpResponseError(response=response, model=error)
azure.core.exceptions.HttpResponseError: (MissingRequiredParameter) The request is invalid. Details: actions : No indexing actions found in the request. Please include between 1 and 32000 indexing actions in your request.
Code: MissingRequiredParameter
Message: The request is invalid. Details: actions : No indexing actions found in the request. Please include between 1 and 32000 indexing actions in your request.
Exception Details: (MissingIndexDocumentsActions) No indexing actions found in the request. Please include between 1 and 32000 indexing actions in your request. Parameters: actions
Code: MissingIndexDocumentsActions
Message: No indexing actions found in the request. Please include between 1 and 32000 indexing actions in your request. Parameters: actions

Support txt, md, HTML and PDF files for ingestion

(i.e. entitlement)

Introduce schema to name all the artifacts generated:

• case name
• parameters combination as a prefix

The Accelerator currently supports models from Azure, but we should allow access to models provided directly from OpenAI.

For openai, you dont have to set endpoint, just key and org. For azure openai, we have endpoint, etc.

Can you do some modifications like this to provide 2 .env examples, one for openai, one for azure openai.

Current implementation captures the "title" and "summary", will need to include keywords, keyphrases, and entities. Need to edit the prompts.py and acs_ingest.py files to request that the LLM generate keywords and entities.

Azure Cognitive Search has skillsets for language detection/processing. Exploration will be required to determine the best implementation. Should experiment with German, Italian, and English as languages to be tested, as these are currently being used in an active customer engagement.

No module named 'azure.core'

Need to standardize the process of loading and returning data ingested into the tool.

Prioritize MD files.
Next should be HTML

File formats to consider:

• PDF
• HTML
• MD
• docx

Interactive tool which helps user choose intent, query type, understand their data and possible label sets, focus on different relevant parts of the documents to generate different results to help user choose what makes most sense.

Can be summarized as content understanding and intent understanding:

• Understanding source of data, potential sources of data and type of data
• Leveraging CMS sources including metadata
• Questions to identify what content processing would work best
• Questions to identify intent - Comparison intent, search intent, opinion intent

Currently running the 02_qa_generation.py sometimes provides an invalid JSON output.
Update or fix the QA generation prompt in prompts.py to make sure that only a valid JSON is created as a result of running the 02_qa_generation.py script.

Add reading from HTML files and splitting them into parts.

Parallel jobs need to be enabled within the Accelerator to accept the ingestion and processing of multiple documents/artifacts.

Potential pattern to be applied would be via the implementation of Azure Machine Learning pipelines.

Currently we are expecting some version of python and fixed versions of libraries, but do not specify it anywhere except README.
What could be done, is to create environment.yaml for conda, to automatically create conda env with everything required, including preloaded models and versioned libraries as well as specify python version

For example, the eval_data.jsonl file has PII (emails, phone numbers, job titles, etc.) related to personnel who work for a customer. All files need to be scanned for PII and sanitized.

Create a config file associated with the generate_embeddings.py file that can provide the appropriate sizes and chunks for Azure OpenAI embeddings models.

The definition for embedding_dimension should be automatically mapped to the selected embeddings model.

Also update documentation with the relevant instructions for users.

Background:
Azure Open AI offers embeddings that can be used to do search and analyze complex documents. Some examples of complex documents are:

• Legal contracts
• Medical records
• Scientific articles

Embeddings are numerical representations of words and phrases that capture the meaning and context of the text. These embeddings can be used to build powerful search and analysis tools that can extract insights from large volumes of text data.

Everything should be linked to the search configuration.

https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/understand-embeddings

Documentation on "Task-specific knowledge enrichment" can assist in the implementation of embeddings models.

Add documentation to explain:

Need to evaluate the search response from Azure Cognitive Search to determine its information retrieval accuracy. A score for response relevancy should be included as part of the tooling for this section.

https://learn.microsoft.com/en-us/azure/search/search-pagination-page-layout

https://learn.microsoft.com/en-us/azure/search/index-similarity-and-scoring

Search metrics that should be included:

• precision@k
• recall@k
• map@k

Update the 03_querying.py to use a config driven approach like the 02_qa_generation.py currently does.

Update documentation explaining the config to the users.

The final evaluation of the RAG Pattern, there should be a recommendation on the strategies within each step of the RAG Experiment Accelerator that would provide the best results.