Since the main.py is getting long and affecting the overall readability and functionality of the project.I was thinking to make the code more modular into different files.

The objective of this issue is to develop a versatile SQL data retriever that allows users to interact with SQL databases using natural language prompts and queries. Langchain provides a SQLDatabaseChain , enabling the implementation of this functionality. The retriever is to be designed to work with various database management systems, including MySQL, PostgreSQL, SQLite3, and more, making it adaptable to different data storage solutions.

Add a function in builder.py to write all the logs to a file for inspection.

NeoGPT uses several CLI commands allowing users to customise its behaviour.

Make a .md file providing a brief documentation of the CLI and how to run and expected behaviour of NeoGPT. Making it easier for others to customise the project based on their needs.

Add the langchain MultiQueryRetriever is designed to provide a more comprehensive approach to information retrieval by generating multiple queries from different perspectives for a given user input query.

Add documentation for the project

In the load_model() add support for loading full models from huggingface directly. For hardware supported devices.

Add support for GPTQ models when loading a model in load_model()

Arg Parser

Issue Description

add an argument parser to set fields like device_type, specific model

ChromaStore

Update the main.py to fetch data from the ChromaStore and fetch based on query.

allow users to load XML dumps (Wikipedia,etc) and convert them into vectors in chroma db

Currently, the application doesn't run on Python < 3.10, and Python > 3.10 doesn't work due to incompatibilities with PyTorch.

We recognize the importance of comprehensive documentation for our project, and we are taking steps to improve it. This issue is dedicated to adding documentation using Docusaurus that will enhance the accessibility and quality of our project's documentation.

Currently there are no such documentation rather than just readme files for running and builder.md to run youtube URL's

Adding support for various types of document loader so convert them into vectors (txt) file

NeoGPT currently relies on Google Search for web-based retrieval. To provide more options and reduce reliance on external services, we should add support for additional search engines such as DuckDuckGo, Bing, or others.

Feature Request: We propose implementing a temporary memory buffer within NeoGPT to improve the continuity and coherence of conversations during a session. This feature will enhance user engagement by allowing NeoGPT to recall and reference previous parts of the conversation. Your contributions and ideas are welcome!

Add Model Specific Prompt Templates

Create PromptTemplates for various tasks that can be used to prompt the LLMs

Thank you for your contribution to Hacktoberfest 2023!

UI

Create an basic UI like a chatbot like interface to interact with neogpt

Chainlit UI

Create UI using chainlit

Please follow this template to create a new issue for the neokd/NeoGPT repository. Thank you for your contribution to Hacktoberfest 2023!

Unable to run the docker image properly. It gives error when installing the requirements

Multiple types of input documents to be supported for now it only supports PDF

If no input is received from the user for next 30 seconds or 1 minute then the loop should close automatically and the program should end.

add a template for full request and issue template in .github

To add a flag to main retriever as --show or --source which is to be false by default. If it is true then the bot should display the source documents in a well structured way in the terminal

NeoGPT allows user to specify a persona "default", "recruiter", "academician", "friend", "ml_engineer", "ceo", "researcher". Add a select box in the sidebar allowing users to select a persona according to their requirement.

Description

Add support for pinecone for storing the vectors and using with RAG

Create a section explaining how to run the model.
Explain that Google CSE and API Key are needed, which command use to run, etc.

Output parser for structured output

Add an output parser for the LLM to display an structured output in JSON or any other format.

Enhance the pinecone support allowing users to store the embeddings in pinecone

Issue Template for Hacktoberfest 2023 Contribution

Issue Description

There is a need of Github actions, mostly for now, for testing python packages by creating the docker image.

Expected Behavior

Stability of docker image.

Current Behavior

Not stable for the moment.

Steps to Reproduce (if applicable)

Creating an docker image with the dockerfile

Suggested Solution (if known)

Having a pipeline with Github Actions.

Web Research

Allowing the LLM to search online backed by a vector store such that it can search on web and answer to the user based on their query and not limiting it to the trained data or the vector store.

The embedding model loads twice in the vectorstore director. Decreasing the performance for the same.

The original issue template and PR template were for use of hacktoberfest. Change the template to a good template for ease of use.

name: Request for a ChromaStore Class for Embedding Documents
about: Request the addition of a ChromaStore class that allows users to embed documents.

Issue Description

A a ChromaStore class in the ChromaDB library that facilitates the embedding of documents. This class would simplify the process of embedding documents using a specified model, and optionally, storing them in an SQLite3 database or another format.

Proposed Implementation

I propose the addition of the ChromaStore class,. This class should have methods for embedding documents, associating them with IDs if provided, and optionally saving them to a database or directory.

Reason for Request

Embedding documents is a common task in natural language processing and information retrieval applications. Providing a convenient and standardized way to perform document embedding within ChromaDB would benefit users and encourage adoption of the library.

Write a guide or tutorial aimed at helping beginners get started with the project. Share your knowledge and make it easy for newcomers to dive in.

Issue Template for Hacktoberfest 2023 Contribution

Issue Description

Provide a brief description of the issue you'd like to address. This can be a bug report, feature request, or any other improvement.

The current README.md doesn't define the procedure to work with Dockerfile and Containers to run NeoGPT.

Expected Behavior

Explain what you expect to happen when this issue is resolved.
Not Applicable

Current Behavior

Describe the current behavior of the project, which is affected by this issue.
Not Applicable

Steps to Reproduce (if applicable)

If this is a bug report, provide a step-by-step guide to reproduce the issue.
Not Applicable

Suggested Solution (if known)

Providing helpful note on using Dockerfile to create images and run containers.

Your Environment (if applicable)

• Operating System:
• Browser (if relevant):
• Any other relevant information:

Additional Context

Add any other context or information about the issue here.

Your Contribution (For Pull Requests)

If you plan to contribute to this issue with a pull request, please outline your proposed changes and mention any related issues or pull requests.

Acknowledgment (For Contributors)

By creating this issue or pull request, you acknowledge that you have read and agree to follow the Code of Conduct for this project.

Please follow this template to create a new issue for the neokd/NeoGPT repository. Thank you for your contribution to Hacktoberfest 2023!

Develop a shell script (e.g., a Bash script) that simplifies the process of running NeoGPT by executing the necessary commands and options. This script should allow users to quickly start NeoGPT without needing to remember complex command-line arguments.

Implement Docker support to make it easier for users to deploy and run NeoGPT in various environments. Dockerization ensures consistent and isolated runtime environments, simplifying the setup process and enhancing portability.

Tasks:

1. Create a Dockerfile for NeoGPT.
2. Define the base image and required dependencies.
3. Configure the container to run NeoGPT with specific command-line arguments.
4. Provide clear instructions for building and running NeoGPT using Docker.

Restructure the project directory to bring the project under one folder and renaming files to have much cleaner repo.

Add support for models from Ollama to run them locally.

User has to wait for model to query the prompt and then give the output, instead of that ui will show the output as same as what is shown in the terminal it shows live response from the LLM rather than making the user wait for output.

Add whatsapp chat loader to builder.py for loading whatsapp chat and allowing user to chat with it.

Issue Description

Add support for GPT4ALL models to load and use

The bot doesn't show proper response in the UI and there is also no streamming support. Add one in UI.py

Add Multi threading to build multiple files at once in builder.py

Issue Description

Make an function to load multiple files and run simultaneously to convert them into vectors

Expected Behavior

Reduce the run time when multiple files are imported

Modify the builder in such a way to support coding files like .py allowing users to chat with them.

Add steps to setup the project via Ananconda in the readme

Change in README

Issue Description

Provide a brief description about the project and its features with installation steps.

In builder.py add tqdm loader showing progress. Allowing users to see progress of documents getting ingested into the vectorstore when loading large files

To add self-querying retriever in neogpt.

Self-querying retriever.

A self-querying retriever is one that, as the name suggests, has the ability to query itself. Specifically, given any natural language query, the retriever uses a query-constructing LLM chain to write a structured query and then applies that structured query to its underlying VectorStore. This allows the retriever to not only use the user-input query for semantic similarity comparison with the contents of stored documents but to also extract filters from the user query on the metadata of stored documents and to execute those filters.

You can use self query retriever from langchain