I have noticed that in the project, the action timeout duration is determined by the action_exec_time in SceneAgent. However, an agent might have multiple actions with differing timeout requirements. Under the current setup, this could lead to issues. To address this, I propose a solution: it would be more appropriate to place action_exec_time within ActionDefinition.

Below is my pull request regarding this solution. If you find it suitable, please consider merging it.

#40

Overview

This is a demonstration of the roadmap to v1.0.0.

The core objective of this project is to deliver a meticulously designed and sufficiently flexible framework, accompanied by a set of tools to assist developers in rapidly implementing simulation scenarios where multiple LLM agents can interact or compete to fulfill specific needs or tasks with minimal code. Simultaneously, the project incorporates pre-built, diverse simulation scenarios to enable developers to directly test the specific performance of their LLM agents within corresponding contexts, and to compare with other agents implemented by their own or the community.

By the time of the v1.0.0 release, this project will encompass the following features:

• A highly abstract core framework with standardized protocols to creat scene projects.
• A web service that is stable enough to concurrently running multiple scenario simulation tasks.
• Develop scene projects as many as possible.
• Implement popular LLM reasoning methods as many as possible.
• Support popular LLM backends as many as possible.
• Support popular prompting frameworks as many as possible.

Table of Contents

1. Core Framework Implementation
2. Web Service Implementation
3. Scene Projects Development
4. LLM Reasoning Methods Implementations
5. LLM Backends Supporting
6. Prompting Frameworks Supporting

Core framework implementatioin

Implement a meticulously designed, highly abstract core framework where defines all the elements necessary for creating a scene project, providing standardized protocols that accurately identify all scene projects' components in accordance with the specified configurations.

(todo list here)

Web service implementatioin

Implement a stable, high-concurrency web service that offers a range of APIs that facilitate seamless interaction with leaf-playground-webui. It will operate in a containerized manner, concurrently executing multiple scenario simulation tasks.

(todo list here)

Scene projects development

Develop a multitude of scene projects that combine entertainment value and application value to meet various evaluation needs of community users. The results of simulation tasks generated by each scene project should effectively quantify the specific application skills and general abilities of LLM agents.

(todo list here)

LLM reasoning methods implementations

See #9 for more details.

LLM Backends supporting

Support a selection of mainstream LLM backends, and define communication protocols when necessary.

(todo list here)

Prompting frameworks supporting

See #9 for more details.

There is a recently published paper A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future.

In addition to supporting mainstream LLM backends, the project should also encompass support for some popular prompting frameworks such as langchain, semantic kernel, prompt flow, textai, LlamaIndex, etc. The fundamental requirement is to offer straightforward wrappers for these frameworks, ensuring developers can seamlessly integrate them into our project. Additionally, efforts should be made to combine and encapsulate commonly used functionalities from these frameworks, minimizing redundancy in developers' work.

Simultaneously, the project should extensively implement useful LLM reasoning strategies and encapsulate them into functional modules. Ideally, these strategies should be packaged into individual tools, enabling LLMs to autonomously select and apply the appropriate strategy as needed.

(todo list here)