We introduced a new model designed for the Code generation task. Its test accuracy on the HumanEval base dataset surpasses that of GPT-4 Turbo (April 2024) and GPT-4o.
Home Page: https://arxiv.org/abs/2405.14906
License: Apache License 2.0
This is great, is there any plans to support debugging / dataset creation from any repository?
I imagine coding models for each repository would be beneficial however the "code interpretor" or the engine that runs the code must support more than just python. And from my perspective may be difficult and only support languages that implement jupyter notebooks? Is that a limitation you forsee for being able to gen a compiler/compute backed dataset for any arbitrary gh repo?
For example
Hi, thanks for the nice work. Do you have any plan in releasing the source code of the data collection framework? It would greatly help.
由于之前测试过 OpenCodeInterpreter 项目，本项目的 Python 环境要求，与 OpenCodeInterpreter 项目一致，所以部署测试很快。
本人环境：Windows 11 x64，CPU 64GB，GPU 24GB。Python 3.10.13、CUDA 11.8、Pytorch 2.1.0 等。
下载 AutoCoder_S_6.7B 模型，在原 OpenCodeInterpreter 的 Python 环境跑测试：
python chatbot-me.py --path checkpoints/AutoCoder_S_6.7B
模型加载后，显存就占到了 23690MiB / 24576MiB，一对话，显存立马爆了。在 code_interpreter/AutoCoderInterpreter.py 脚本中，添加 torch_dtype=torch.float16 模型加载参数后，模型加载占用 13649MiB / 24576MiB，基本正常，可对话。但是：
我之前测试过 OpenCodeInterpreter 项目，知道 Windows 平台因为用户权限和目录结构等问题，直接进行 Code Interpreter 很多时候是不行的，如果在 Linux 环境 root 用户下，就可以进行更多的操作。
不知以上理解是否正确，请大佬指导，谢谢！
Thank you for the great work and sharing this repo! One question: What information is left out during inference from the output of model.generate()?
For context, after we run model.generate, we get the final answer for the code generation.
However, from the fine-tuning dataset proposed in the AIEV-Instruct paper, AutoCoder is fine-tuned on a multi-turn dialogue dataset. Therefore, we expect the output of model.generate to be a multi-turn dialogue as well, but it isn't -- it appears to be a single-turn answer.
What information is left out? Does model.generate implicitly go through the process of 1) generating an initial answer, 2) having a code executor run the test cases, and then 3) giving feedback and posing a question if it didn't pass the test case? And if so, how many rounds of this cycle is employed during model.generate?
Thank you again for the great work. :)
Good job. Do you consider establishing a commnunity group for discussion? A discord group or wechat group.
很好的工作。可不可以建立一个社区交流群呢？微信群或者 Discord 群聊都可以，这样的话可以及时接收大家的反馈和意见，大家也可以一起看看有没有能帮忙的地方。
Hey thanks for the hard work. Unfortunately we couldn't test AutoCoder, cause the conda VENV complains with lots of errors when trying to install the dependencies.
We kept on adjusting requirements and versions to get CUDA, torch, etc. installed on our Windows 10 with RTX 4080. Usually we don't have any problems with such a little dependency forest.
Ultimately we tried to bump to 3.12 but had no luck there either (we tried 3.10, 3.11, 3.12) - could be just randomly intersecting package issues.
miniconda3\python.exe -m pip install --upgrade pip setuptools wheel
AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
Building wheel for numpy (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for numpy (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [40 lines of output]
<string>:54: RuntimeWarning: NumPy 1.25.0 may not yet support Python 3.12.
miniconda3\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
exec(code, locals())
File "<string>", line 567, in <module>
File "<string>", line 546, in setup_package
File "<string>", line 328, in generate_cython
OSError: Cython needs to be installed in Python as a module
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for numpy
Failed to build numpy
We had a million more issues never seen before.
Please kindly provide some additional setup instructions.
[2024-05-30 13:22:09,739] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:03<00:00, 1.06s/it]
Running on local URL: http://0.0.0.0:10518
To create a public link, set share=True in launch().
INFO:root:allocating new uuid 0ef99ba0-4085-4963-85ca-954097161a80 for conversation...
INFO:root:user message:
Write a script in Python and draw a heart-shaped pattern and verify it
INFO:root:inputs:
tensor([[ 39, 8384, 25, 17437, 245, 4744, 279, 13003, 285, 4293,
245, 2986, 12, 25269, 7949, 285, 12909, 359, 185, 5618,
15481, 25, 207]])
INFO:root:kwargs:
{'input_ids': tensor([[ 39, 8384, 25, 17437, 245, 4744, 279, 13003, 285, 4293,
245, 2986, 12, 25269, 7949, 285, 12909, 359, 185, 5618,
15481, 25, 207]], device='cuda:0'), 'streamer': <transformers.generation.streamers.TextIteratorStreamer object at 0x7fc6ba5b4400>, 'max_new_tokens': 1024, 'do_sample': True, 'top_k': 50, 'top_p': 0.95, 'eos_token_id': 32021, 'pad_token_id': 32021}
generated_text Sure, here is a basic script that uses the Turtle Graphics library to draw a heart-shaped pattern. We will use a simple square and circle function to draw this shape.
import turtle
# function to draw square
def draw_square(side_length, color):
turtle.color(color)
turtle.begin_fill()
for _ in range(4):
turtle.forward(side_length)
turtle.right(90)
turtle.end_fill()
# function to draw circle
def draw_circle(radius, color):
turtle.color(color)
turtle.begin_fill()
turtle.circle(radius)
turtle.end_fill()
# function to draw heart shape
def draw_heart():
# Set the fill color
turtle.fillcolor("red")
# Draw square
draw_square(150, "red")
# Move to start of heart
turtle.penup()
turtle.goto(-75, 125)
turtle.pendown()
# Draw circle
draw_circle(75, "red")
# Move to the center of the heart
turtle.penup()
turtle.goto(0, 0)
turtle.pendown()
# Draw circle
draw_circle(75, "red")
# Move to the end of the heart
turtle.penup()
turtle.goto(75, 125)
turtle.pendown()
# Draw circle
draw_circle(75, "red")
# Draw the heart
draw_heart()
# To keep the window open
turtle.mainloop()When you run this script, it will open a window and display a red heart-shaped pattern. It can be tested to verify its accuracy.<|EOT|>
INFO:root:saving current dialog to file 0ef99ba0-4085-4963-85ca-954097161a80.json...
INFO:root:current dialog: [{'role': 'user', 'content': 'Write a script in Python and draw a heart-shaped pattern and verify it'}, {'role': 'assistant', 'content': 'Sure, here is a basic script that uses the Turtle Graphics library to draw a heart-shaped pattern. We will use a simple square and circle function to draw this shape.\n\npython\nimport turtle\n\n# function to draw square\ndef draw_square(side_length, color):\n turtle.color(color)\n turtle.begin_fill()\n for _ in range(4):\n turtle.forward(side_length)\n turtle.right(90)\n turtle.end_fill()\n\n# function to draw circle\ndef draw_circle(radius, color):\n turtle.color(color)\n turtle.begin_fill()\n turtle.circle(radius)\n turtle.end_fill()\n\n# function to draw heart shape\ndef draw_heart():\n # Set the fill color\n turtle.fillcolor("red")\n\n # Draw square\n draw_square(150, "red")\n\n # Move to start of heart\n turtle.penup()\n turtle.goto(-75, 125)\n turtle.pendown()\n\n # Draw circle\n draw_circle(75, "red")\n\n # Move to the center of the heart\n turtle.penup()\n turtle.goto(0, 0)\n turtle.pendown()\n\n # Draw circle\n draw_circle(75, "red")\n\n # Move to the end of the heart\n turtle.penup()\n turtle.goto(75, 125)\n turtle.pendown()\n\n # Draw circle\n draw_circle(75, "red")\n\n# Draw the heart\ndraw_heart()\n\n# To keep the window open\nturtle.mainloop()\n\nWhen you run this script, it will open a window and display a red heart-shaped pattern. It can be tested to verify its accuracy.'}]
ed8490a3da0b2c2f04c49154379b53a357ac00c28e22a771730ef13615eecad9
HAS_Code:False
INFO:root:user message:
please run the script to verify
INFO:root:inputs:
tensor([[ 39, 8384, 25, 17437, 245, 4744, 279, 13003, 285, 4293,
........
12909, 185, 5618, 15481, 25, 207]])
INFO:root:kwargs:
{'input_ids': tensor([[ 39, 8384, 25, 17437, 245, 4744, 279, 13003, 285, 4293,
........
32014, 185, 39, 8384, 25, 4058, 1404, 254, 4744, 276,
12909, 185, 5618, 15481, 25, 207]], device='cuda:0'), 'streamer': <transformers.generation.streamers.TextIteratorStreamer object at 0x7fc6ba5b56f0>, 'max_new_tokens': 1024, 'do_sample': True, 'top_k': 50, 'top_p': 0.95, 'eos_token_id': 32021, 'pad_token_id': 32021}
generated_text
We need to execute the code to see if it's working properly.
Our initial task should be to install the related packages.
<API_RUN_START>```sh
pip install numpy
We can proceed to execute the code:
<API_RUN_START>```python
import numpy as np
def generate_data_and_save(file_path, size):
# Generate random 2D data
data = np.random.rand(size, size)
# Save data to a file
np.save(file_path, data)
print(f"Data saved to: {file_path}")
def load_data_and_process(file_path):
# Load the data
data = np.load(file_path)
# Apply some processing (e.g., element-wise multiplication)
processed_data = data * 2
return processed_data
def main():
# Define the path where the data will be saved
data_file_path = "random_data.npy"
# Generate and save data
generate_data_and_save(data_file_path, 1000)
# Load and process data
processed_data = load_data_and_process(data_file_path)
# Optionally, check the output
print(processed_data)
print("Processed data successfully loaded.")
if __name__ == "__main__":
main()
```<API_RUN_STOP><|EOT|>
INFO:root:saving current dialog to file 0ef99ba0-4085-4963-85ca-954097161a80.json...
INFO:root:current dialog: [{'role': 'user', 'content': 'Write a script in Python and draw a heart-shaped pattern and verify it'}, {'role': 'assistant', 'content': 'Sure, here is a basic script that uses the Turtle Graphics library to draw a heart-shaped pattern. We will use a simple square and circle function to draw this shape.\n\n```python\nimport turtle\n\n# function to draw square\ndef draw_square(side_length, color):\n turtle.color(color)\n turtle.begin_fill()\n for _ in range(4):\n turtle.forward(side_length)\n turtle.right(90)\n turtle.end_fill()\n\n# function to draw circle\ndef draw_circle(radius, color):\n turtle.color(color)\n turtle.begin_fill()\n turtle.circle(radius)\n turtle.end_fill()\n\n# function to draw heart shape\ndef draw_heart():\n # Set the fill color\n turtle.fillcolor("red")\n\n # Draw square\n draw_square(150, "red")\n\n # Move to start of heart\n turtle.penup()\n turtle.goto(-75, 125)\n turtle.pendown()\n\n # Draw circle\n draw_circle(75, "red")\n\n # Move to the center of the heart\n turtle.penup()\n turtle.goto(0, 0)\n turtle.pendown()\n\n # Draw circle\n draw_circle(75, "red")\n\n # Move to the end of the heart\n turtle.penup()\n turtle.goto(75, 125)\n turtle.pendown()\n\n # Draw circle\n draw_circle(75, "red")\n\n# Draw the heart\ndraw_heart()\n\n# To keep the window open\nturtle.mainloop()\n```\nWhen you run this script, it will open a window and display a red heart-shaped pattern. It can be tested to verify its accuracy.'}, {'role': 'user', 'content': 'please run the script to verify'}, {'role': 'assistant', 'content': '\nWe need to execute the code to see if it\'s working properly.\nOur initial task should be to install the related packages.\n<API_RUN_START>```sh\npip install numpy\n```<API_RUN_STOP>\nWe can proceed to execute the code:\n<API_RUN_START>```python\nimport numpy as np\n\ndef generate_data_and_save(file_path, size):\n # Generate random 2D data\n data = np.random.rand(size, size)\n # Save data to a file\n np.save(file_path, data)\n print(f"Data saved to: {file_path}")\n\ndef load_data_and_process(file_path):\n # Load the data\n data = np.load(file_path)\n # Apply some processing (e.g., element-wise multiplication)\n processed_data = data * 2\n return processed_data\n\ndef main():\n # Define the path where the data will be saved\n data_file_path = "random_data.npy"\n # Generate and save data\n generate_data_and_save(data_file_path, 1000)\n \n # Load and process data\n processed_data = load_data_and_process(data_file_path)\n \n # Optionally, check the output\n print(processed_data)\n print("Processed data successfully loaded.")\n\nif __name__ == "__main__":\n main()\n```<API_RUN_STOP>'}]
docker: Error response from daemon: Conflict. The container name "/container_python_0" is already in use by container "ed8490a3da0b2c2f04c49154379b53a357ac00c28e22a771730ef13615eecad9". You have to remove (or rename) that container to be able to reuse that name.
See 'docker run --help'.
Traceback (most recent call last):
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/gradio/queueing.py", line 407, in call_prediction
output = await route_utils.call_process_api(
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/gradio/route_utils.py", line 226, in call_process_api
output = await app.get_blocks().process_api(
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/gradio/blocks.py", line 1550, in process_api
result = await self.call_function(
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/gradio/blocks.py", line 1199, in call_function
prediction = await utils.async_iteration(iterator)
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/gradio/utils.py", line 519, in async_iteration
return await iterator.__anext__()
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/gradio/utils.py", line 512, in __anext__
return await anyio.to_thread.run_sync(
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/anyio/to_thread.py", line 56, in run_sync
return await get_async_backend().run_sync_in_worker_thread(
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
return await future
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 859, in run
result = context.run(func, *args)
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/gradio/utils.py", line 495, in run_sync_iterator_async
return next(iterator)
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/site-packages/gradio/utils.py", line 649, in gen_wrapper
yield from f(*args, **kwargs)
File "/home/tkadmin/AutoCoder/chatbot.py", line 247, in bot
subprocess.run(["docker", "run", "-d", "-v",
File "/home/tkadmin/miniconda3/envs/code/lib/python3.10/subprocess.py", line 526, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['docker', 'run', '-d', '-v', '/home/tkadmin/AutoCoder/sandbox/python_0:/app', '--name', 'container_python_0', 'python-sandbox_0', 'tail', '-f', '/dev/null']' returned non-zero exit status 125.
^CKeyboard interruption in main thread... closing server.
container_python_0
Container named container_python_0 has been removed.
从上面日志可以看出，容器 ed8490a3da0b2c2f04c49154379b53a357ac00c28e22a771730ef13615eecad9 是对话后新创建的。但为什么会提示已被占用？
tkadmin@ubu:~$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
python-sandbox_0 latest c4abb2fec3ac 10 minutes ago 1.02GB
tkadmin@ubu:~$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ed8490a3da0b python-sandbox_0 "tail -f /dev/null" 3 minutes ago Up 3 minutes container_python_0
从 docker 命令查询，可以看到，image 先创建，然后 7 分钟后，容器才被启动。
请大佬帮忙分析分析，为什么容器使用有冲突了，谢谢！
Web 界面如下：
Hi! Thanks for your great work! I was wondering if you'd be interested in sharing the AEIV-Instruct dataset so that this method could be improved or applied to other base models. Typically, people would share the dataset on HuggingFace and the training code to apply the dataset to the model on GitHub.
Is it possible to share the host conda environments with the containers, and have them execute under a given environment?
rt
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.