Code Monkey home page Code Monkey logo

princeton-nlp / swe-agent Goto Github PK

View Code? Open in Web Editor NEW
11.8K 87.0 1.2K 7.1 MB

SWE-agent takes a GitHub issue and tries to automatically fix it, using GPT-4, or your LM of choice. It solves 12.47% of bugs in the SWE-bench evaluation set and takes just 1 minute to run.

Home Page: https://princeton-nlp.github.io/SWE-agent/

License: MIT License

Shell 10.30% Python 74.63% Dockerfile 0.51% JavaScript 10.06% HTML 0.73% CSS 3.76%
agent ai developer-tools llm agent-based-model lms

swe-agent's Introduction

swe-agent.com

Website & Demo  |   Documentation  |   Discord  |   Preprint

SWE-agent turns LMs (e.g. GPT-4) into software engineering agents that can resolve issues in real GitHub repositories.

On SWE-bench, SWE-agent resolves 12.47% of issues, achieving the state-of-the-art performance on the full test set.

We accomplish our results by designing simple LM-centric commands and feedback formats to make it easier for the LM to browse the repository, view, edit and execute code files. We call this an Agent-Computer Interface (ACI). Read more about it in our paper!

SWE-agent is built and maintained by researchers from Princeton University.

My Movie 3

You can use SWE-agent either through a web interface (shown above) or through the command line.

🚀 Get started!

👉 Try SWE-agent in your browser: Open in GitHub Codespaces (more information)

Read our documentation to learn more:

💫 Contributions

  • If you'd like to ask questions, learn about upcoming features, and participate in future development, join our Discord community!
  • If you'd like to contribute to the codebase, we welcome issues and pull requests!

Contact person: John Yang and Carlos E. Jimenez (Email: {jy1682, carlosej}@princeton.edu).

📝 Citation

If you found this work helpful, please consider citing it using the following:

@misc{yang2024sweagent,
      title={SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering},
      author={John Yang and Carlos E. Jimenez and Alexander Wettig and Kilian Lieret and Shunyu Yao and Karthik Narasimhan and Ofir Press},
      year={2024},
      eprint={2405.15793},
      archivePrefix={arXiv},
      primaryClass={cs.SE}
}

🪪 License

MIT. Check LICENSE.

Pytest Test build containers Release to dockerhub (nightly) Release to dockerhub (release) build-docs codecov pre-commit.ci status Markdown links

swe-agent's People

Contributors

borda avatar burnettk avatar bvandorf avatar carlosejimenez avatar cpendery avatar eltociear avatar foragerr avatar jgalego avatar john-b-yang avatar khangich avatar klieret avatar kwight avatar mikanfactory avatar milaiwi avatar montaguem avatar moresearch avatar mspronesti avatar nfedyashev avatar nims11 avatar ofirpress avatar ollmer avatar pre-commit-ci[bot] avatar rainrat avatar raymyers avatar tam-ng0905 avatar tcoyze avatar yeonwoosung avatar ysymyth avatar zgrannan avatar zhipengzuo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

swe-agent's Issues

Azure ecosystem support

It would be great if the agent supported the Azure ecosystem both through supporting Azure OpenAI & through pulling Azure Devops stories/tasks instead of Github issues.

Some errors

Currently after executing the setup.sh I've tried to execute the usage part but it failed because a lot of dependencies were missing. I installed all of them and it worked, but I don't know why that manual step, maybe I failed something in the setup.

By the other hand, with the latest commit, now it works, but it finish without creating the PR, how should it be executed to be able to create the PR?

Great job!

Run on a local directory (instead of GitHub issue)

It would be great if it could run on a local directory and suggest a new git branch so I could compare the changes + request a change without needing github and a repo.

PS. Right now I don't know how to actually make changes from run.py it outputted some terminal logs but I can't reasonably use them for anything.

Support agent working on codebases outside of python

It would be great to have the source code language be configurable beyond just python. It would involve tweaking some of the prompts

Note however that you cannot use any interactive session commands (e.g. python, vim) in this environment, but you can write scripts and run them. E.g. you can write a python script and then run it with `python <script_name>.py`.

and also changing the shell scripts to customize the linter that is run for different languages like golangci-lint for Go, but I'm not sure limitations there might be.

lint_output=$(flake8 --select=F821,F822,F831,E111,E112,E113,E999,E902 "$CURRENT_FILE" 2>&1)

Using LiteLLM to support more models

This project is pretty great BUT we need more options to use different LLM's.You don't have to worry about creating a solution which supports 100+ LLM easily as LiteLLM is another foss project which is capable of doing this task for you.

Project LiteLLM link - https://github.com/BerriAI/litellm

Adding LiteLLM will be big win for the project as many will be easily able to use many more LLM easily which everyone wants and project will require 3 major parameters from user like base url,model name,api key that's all and with open ai api general structure it can query and give back result for the query.Many big projects have started adding support for this project in there project to make things advanced in easier way so study it and after that if you have any query you can ask them they are pretty responsive plus if u want to know more about my personal experience of using it with other great projects like flowise then I can tell you that too .

Inference - AttributeError: module 'signal' has no attribute 'SIGALRM'

From usage guide on inference in readme

python run.py --model_name gpt4 --data_path pvlib/pvlib-python#1603 --config_file config/default_from_url.yaml

Parsing command file: config/commands/defaults.sh
Parsing command file: config/commands/search.sh
Parsing command file: config/commands/edit_linting.sh
Parsing command file: config/commands/_split_string.py
Parsing command file: config/commands/defaults.sh
Parsing command file: config/commands/search.sh
Parsing command file: config/commands/edit_linting.sh
Parsing command file: config/commands/_split_string.py
INFO 📙 Arguments: agent:
...
INFO 💽 Loaded dataset from pvlib/pvlib-python#1603

Traceback (most recent call last):
line 140, in enter
signal.signal(signal.SIGALRM, self.handle_timeout)
AttributeError: module 'signal' has no attribute 'SIGALRM'

Maybe incompatible version of Python on Docker image?

python --version
Python 3.9.19

Ollama vision models?

Can I use an Ollama vision model for this? Is there any use or benefit to that? I'm not super familiar with this project yet but I'd like to try to use it for a MATLAB project, which requires a payed account so I was hoping it could use the MATLAB app on my computer or something and also see graphical results with an Ollama vision model. Is any of this possible to any degree? Thanks for the help.

Claude 3 benchmarks

It would be very useful to have some Claude 3 benchmarks using SWE-Agent.

IMG_1200

Since Claude 3 Opus performs better than GPT-4 using RAG, there’s a fair chance that will also be the case when using SWE-Agents, right?

Also really curious how Claude 3 Sonnet and Haiku perform, since they not that far behind Opus (and way cheaper).

No such file of instance.traj, only its parent directory exists.

issue

No such file of instance.traj, only its parent directory exists. Could you please provide the specific file of instance.traj for the demonstrations section within the config/default.yaml file?

This is log:

INFO DEMONSTRATION:
trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__default__t-0.20__p-0.95__c-2.00__install-1___install_from_source/marshmallow-code__marshmallow-186
7.traj
Traceback (most recent call last):
File "/home/manna/SWE-agent/run.py", line 109, in main
info = agent.run(
File "/home/manna/SWE-agent/sweagent/agent/agents.py", line 638, in run
self.setup(setup_args, init_model_stats)
File "/home/manna/SWE-agent/sweagent/agent/agents.py", line 218, in setup
demo_history = json.load(open(demonstration_path, "r"))["history"]
FileNotFoundError: [Errno 2] No such file or directory: 'trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__default__t-0.20__p-0.95__c-2.00__install-1___install_from_source/marshmallow-code__marshmallow-1867.traj'
WARNING ❌ Failed on pvlib__pvlib-python-i1603: [Errno 2] No such file or directory:
'trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__default__t-0.20__p-0.95__c-2.00__install-1___install_from_source/marshmallow-code__marshmallow-18
67.traj'
INFO Beginning environment shutdown...
INFO Agent container stopped
INFO 🌱 Environment Initialized

Incompatible with docker-desktop

Installed (on arm64) successfully, then running:

>>python run.py --model_name gpt4   --data_path https://github.com/pymc-devs/pymc/issues/7223 --config_file config/default_from_url.yaml
Parsing command file: config/commands/defaults.sh
Parsing command file: config/commands/search.sh
Parsing command file: config/commands/edit_linting.sh
Parsing command file: config/commands/_split_string.py
Parsing command file: config/commands/defaults.sh
Parsing command file: config/commands/search.sh
Parsing command file: config/commands/edit_linting.sh
Parsing command file: config/commands/_split_string.py
INFO     📙 Arguments: agent:
           config:
             _commands:
             - arguments:
                 line_number:
                   description: the line number to move the window to (if not provided, the
                     window will start at the top of the file)
                   required: false
                   type: integer
                 path:
                   description: the path to the file to open
                   required: true
                   type: string
               code: 'open() {    if [ -z "$1" ]    then        echo "Usage: open <file>"        return    fi    #
                 Check if the second argument is provided    if [ -n "$2" ]; then        #
                 Check if the provided argument is a valid number        if ! [[ $2 =~ ^[0-9]+$
                 ]]; then            echo "Usage: open <file> [<line_number>]"            echo
                 "Error: <line_number> must be a number"            return  # Exit if the line
                 number is not valid        fi        local max_line=$(awk ''END {print NR}''
                 $1)        if [ $2 -gt $max_line ]; then            echo "Warning: <line_number>
                 ($2) is greater than the number of lines in the file ($max_line)"            echo
                 "Warning: Setting <line_number> to $max_line"            local line_number=$(jq
                 -n "$max_line")  # Set line number to max if greater than max        elif
                 [ $2 -lt 1 ]; then            echo "Warning: <line_number> ($2) is less than
                 1"            echo "Warning: Setting <line_number> to 1"            local
                 line_number=$(jq -n "1")  # Set line number to 1 if less than 1        else            local
                 OFFSET=$(jq -n "$WINDOW/6" | jq ''floor'')            local line_number=$(jq
                 -n "[$2 + $WINDOW/2 - $OFFSET, 1] | max | floor")        fi    else        local
                 line_number=$(jq -n "$WINDOW/2")  # Set default line number if not provided    fi    if
                 [ -f "$1" ]; then        export CURRENT_FILE=$(realpath $1)        export
                 CURRENT_LINE=$line_number        _constrain_line        _print    elif [ -d
                 "$1" ]; then        echo "Error: $1 is a directory. You can only open files.
                 Use cd or ls to navigate directories."    else        echo "File $1 not found"    fi}'
               docstring: opens the file at the given path in the editor. If line_number is
                 provided, the window will be move to include that line
               end_name: null
               name: open
               signature: open <path> [<line_number>]
             - arguments:
                 line_number:
                   description: the line number to move the window to
                   required: true
                   type: integer
               code: 'goto() {    if [ $# -gt 1 ]; then        echo "goto allows only one line
                 number at a time."        return    fi    if [ -z "$CURRENT_FILE" ]    then        echo
                 "No file open. Use the open command first."        return    fi    if [ -z
                 "$1" ]    then        echo "Usage: goto <line>"        return    fi    if
                 ! [[ $1 =~ ^[0-9]+$ ]]    then        echo "Usage: goto <line>"        echo
                 "Error: <line> must be a number"        return    fi    local max_line=$(awk
                 ''END {print NR}'' $CURRENT_FILE)    if [ $1 -gt $max_line ]    then        echo
                 "Error: <line> must be less than or equal to $max_line"        return    fi    local
                 OFFSET=$(jq -n "$WINDOW/6" | jq ''floor'')    export CURRENT_LINE=$(jq -n
                 "[$1 + $WINDOW/2 - $OFFSET, 1] | max | floor")    _constrain_line    _print}'
               docstring: moves the window to show <line_number>
               end_name: null
               name: goto
               signature: goto <line_number>
             - arguments: null
               code: scroll_down() {    if [ -z "$CURRENT_FILE" ]    then        echo "No file
                 open. Use the open command first."        return    fi    export CURRENT_LINE=$(jq
                 -n "$CURRENT_LINE + $WINDOW - $OVERLAP")    _constrain_line    _print}
               docstring: moves the window down {WINDOW} lines
               end_name: null
               name: scroll_down
               signature: scroll_down
             - arguments: null
               code: scroll_up() {    if [ -z "$CURRENT_FILE" ]    then        echo "No file
                 open. Use the open command first."        return    fi    export CURRENT_LINE=$(jq
                 -n "$CURRENT_LINE - $WINDOW + $OVERLAP")    _constrain_line    _print}
               docstring: moves the window down {WINDOW} lines
               end_name: null
               name: scroll_up
               signature: scroll_down
             - arguments:
                 filename:
                   description: the name of the file to create
                   required: true
                   type: string
               code: "create() {    if [ -z \"$1\" ]; then        echo \"Usage: create <filename>\"\
                 \        return    fi    # Check if the file already exists    if [ -e \"\
                 $1\" ]; then        echo \"Error: File '$1' already exists.\"\t\topen \"$1\"\
                 \        return    fi    # Create the file an empty new line    printf \"\\\
                 n\" > \"$1\"    # Use the existing open command to open the created file \
                 \   open \"$1\"}"
               docstring: creates and opens a new file with the given name
               end_name: null
               name: create
               signature: create <filename>
             - arguments: null
               code: 'submit() {    cd $ROOT    # Check if the patch file exists and is non-empty    if
                 [ -s "/root/test.patch" ]; then        # Apply the patch in reverse        git
                 apply -R < "/root/test.patch"    fi    git add -A    git diff --cached > model.patch    echo
                 "<<SUBMISSION||"    cat model.patch    echo "||SUBMISSION>>"}'
               docstring: submits your current code and terminates the session
               end_name: null
               name: submit
               signature: submit
             - arguments:
                 dir:
                   description: the directory to search in (if not provided, searches in the
                     current directory)
                   required: false
                   type: string
                 search_term:
                   description: the term to search for
                   required: true
                   type: string
               code: 'search_dir() {    if [ $# -eq 1 ]; then        local search_term="$1"        local
                 dir="./"    elif [ $# -eq 2 ]; then        local search_term="$1"        if
                 [ -d "$2" ]; then            local dir="$2"        else            echo "Directory
                 $2 not found"            return        fi    else        echo "Usage: search_dir
                 <search_term> [<dir>]"        return    fi    dir=$(realpath "$dir")    local
                 matches=$(find "$dir" -type f ! -path ''*/.*'' -exec grep -nIH "$search_term"
                 {} + | cut -d: -f1 | sort | uniq -c)    # if no matches, return    if [ -z
                 "$matches" ]; then        echo "No matches found for \"$search_term\" in $dir"        return    fi    #
                 Calculate total number of matches    local num_matches=$(echo "$matches" |
                 awk ''{sum+=$1} END {print sum}'')    # calculate total number of files matched    local
                 num_files=$(echo "$matches" | wc -l | awk ''{$1=$1; print $0}'')    # if num_files
                 is > 100, print an error    if [ $num_files -gt 100 ]; then        echo "More
                 than $num_files files matched for \"$search_term\" in $dir. Please narrow
                 your search."        return    fi        echo "Found $num_matches matches
                 for \"$search_term\" in $dir:"    echo "$matches" | awk ''{$2=$2; gsub(/^\.+\/+/,
                 "./", $2); print $2 " ("$1" matches)"}''    echo "End of matches for \"$search_term\"
                 in $dir"}'
               docstring: searches for search_term in all files in dir. If dir is not provided,
                 searches in the current directory
               end_name: null
               name: search_dir
               signature: search_dir <search_term> [<dir>]
             - arguments:
                 file:
                   description: the file to search in (if not provided, searches in the current
                     open file)
                   required: false
                   type: string
                 search_term:
                   description: the term to search for
                   required: true
                   type: string
               code: 'search_file() {    # Check if the first argument is provided    if [
                 -z "$1" ]; then        echo "Usage: search_file <search_term> [<file>]"        return    fi    #
                 Check if the second argument is provided    if [ -n "$2" ]; then        #
                 Check if the provided argument is a valid file        if [ -f "$2" ]; then            local
                 file="$2"  # Set file if valid        else            echo "Usage: search_file
                 <search_term> [<file>]"            echo "Error: File name $2 not found. Please
                 provide a valid file name."            return  # Exit if the file is not valid        fi    else        #
                 Check if a file is open        if [ -z "$CURRENT_FILE" ]; then            echo
                 "No file open. Use the open command first."            return  # Exit if no
                 file is open        fi        local file="$CURRENT_FILE"  # Set file to the
                 current open file    fi    local search_term="$1"    file=$(realpath "$file")    #
                 Use grep to directly get the desired formatted output    local matches=$(grep
                 -nH "$search_term" "$file")    # Check if no matches were found    if [ -z
                 "$matches" ]; then        echo "No matches found for \"$search_term\" in $file"        return    fi    #
                 Calculate total number of matches    local num_matches=$(echo "$matches" |
                 wc -l | awk ''{$1=$1; print $0}'')        # calculate total number of lines
                 matched    local num_lines=$(echo "$matches" | cut -d: -f1 | sort | uniq |
                 wc -l | awk ''{$1=$1; print $0}'')    # if num_lines is > 100, print an error    if
                 [ $num_lines -gt 100 ]; then        echo "More than $num_lines lines matched
                 for \"$search_term\" in $file. Please narrow your search."        return    fi    #
                 Print the total number of matches and the matches themselves    echo "Found
                 $num_matches matches for \"$search_term\" in $file:"    echo "$matches" |
                 cut -d: -f1-2 | sort -u -t: -k2,2n | while IFS=: read -r filename line_number;
                 do        echo "Line $line_number:$(sed -n "${line_number}p" "$file")"    done    echo
                 "End of matches for \"$search_term\" in $file"}'
               docstring: searches for search_term in file. If file is not provided, searches
                 in the current open file
               end_name: null
               name: search_file
               signature: search_file <search_term> [<file>]
             - arguments:
                 dir:
                   description: the directory to search in (if not provided, searches in the
                     current directory)
                   required: false
                   type: string
                 file_name:
                   description: the name of the file to search for
                   required: true
                   type: string
               code: 'find_file() {    if [ $# -eq 1 ]; then        local file_name="$1"        local
                 dir="./"    elif [ $# -eq 2 ]; then        local file_name="$1"        if
                 [ -d "$2" ]; then            local dir="$2"        else            echo "Directory
                 $2 not found"            return        fi    else        echo "Usage: find_file
                 <file_name> [<dir>]"        return    fi    dir=$(realpath "$dir")    local
                 matches=$(find "$dir" -type f -name "$file_name")    # if no matches, return    if
                 [ -z "$matches" ]; then        echo "No matches found for \"$file_name\" in
                 $dir"        return    fi    # Calculate total number of matches    local
                 num_matches=$(echo "$matches" | wc -l | awk ''{$1=$1; print $0}'')    echo
                 "Found $num_matches matches for \"$file_name\" in $dir:"    echo "$matches"
                 | awk ''{print $0}''}'
               docstring: finds all files with the given name in dir. If dir is not provided,
                 searches in the current directory
               end_name: null
               name: find_file
               signature: find_file <file_name> [<dir>]
             - arguments:
                 end_line:
                   description: the line number to end the edit at (inclusive)
                   required: true
                   type: integer
                 replacement_text:
                   description: the text to replace the current selection with
                   required: true
                   type: string
                 start_line:
                   description: the line number to start the edit at
                   required: true
                   type: integer
               code: 'edit() {    if [ -z "$CURRENT_FILE" ]    then        echo ''No file open.
                 Use the `open` command first.''        return    fi    local start_line="$(echo
                 $1: | cut -d: -f1)"    local end_line="$(echo $1: | cut -d: -f2)"    if [
                 -z "$start_line" ] || [ -z "$end_line" ]    then        echo "Usage: edit
                 <start_line>:<end_line>"        return    fi    local re=''^[0-9]+$''    if
                 ! [[ $start_line =~ $re ]]; then        echo "Usage: edit <start_line>:<end_line>"        echo
                 "Error: start_line must be a number"        return    fi    if ! [[ $end_line
                 =~ $re ]]; then        echo "Usage: edit <start_line>:<end_line>"        echo
                 "Error: end_line must be a number"        return    fi    # Bash array starts
                 at 0, so let''s adjust    local start_line=$((start_line - 1))    local end_line=$((end_line))    local
                 line_count=0    local replacement=()    while IFS= read -r line    do        replacement+=("$line")        ((line_count++))    done    #
                 Create a backup of the current file    cp "$CURRENT_FILE" "/root/$(basename
                 "$CURRENT_FILE")_backup"    # Read the file line by line into an array    mapfile
                 -t lines < "$CURRENT_FILE"    local new_lines=("${lines[@]:0:$start_line}"
                 "${replacement[@]}" "${lines[@]:$((end_line))}")    # Write the new stuff
                 directly back into the original file    printf "%s\n" "${new_lines[@]}" >|
                 "$CURRENT_FILE"        # Run linter    if [[ $CURRENT_FILE == *.py ]]; then        lint_output=$(flake8
                 --select=F821,F822,F831,E111,E112,E113,E999,E902 "$CURRENT_FILE" 2>&1)    else        #
                 do nothing        lint_output=""    fi    # if there is no output, then the
                 file is good    if [ -z "$lint_output" ]; then        export CURRENT_LINE=$start_line        _constrain_line        _print        echo
                 "File updated. Please review the changes and make sure they are correct (correct
                 indentation, no duplicate lines, etc). Edit the file again if necessary."    else        echo
                 "Your proposed edit has introduced new syntax error(s). Please understand
                 the fixes and retry your edit commmand."        echo ""        echo "ERRORS:"        _split_string
                 "$lint_output"        echo ""        # Save original values        original_current_line=$CURRENT_LINE        original_window=$WINDOW        #
                 Update values        export CURRENT_LINE=$(( (line_count / 2) + start_line
                 )) # Set to "center" of edit        export WINDOW=$((line_count + 10)) # Show
                 +/- 5 lines around edit        echo "This is how your edit would have looked
                 if applied"        echo "-------------------------------------------------"        _constrain_line        _print        echo
                 "-------------------------------------------------"        echo ""        #
                 Restoring CURRENT_FILE to original contents.        cp "/root/$(basename "$CURRENT_FILE")_backup"
                 "$CURRENT_FILE"        export CURRENT_LINE=$(( ((end_line - start_line + 1)
                 / 2) + start_line ))        export WINDOW=$((end_line - start_line + 10))        echo
                 "This is the original code before your edit"        echo "-------------------------------------------------"        _constrain_line        _print        echo
                 "-------------------------------------------------"        # Restore original
                 values        export CURRENT_LINE=$original_current_line        export WINDOW=$original_window        echo
                 "Your changes have NOT been applied. Please fix your edit command and try
                 again."        echo "You either need to 1) Specify the correct start/end line
                 arguments or 2) Correct your edit code."        echo "DO NOT re-run the same
                 failed edit command. Running it again will lead to the same error."    fi    #
                 Remove backup file    rm -f "/root/$(basename "$CURRENT_FILE")_backup"}'
               docstring: replaces lines <start_line> through <end_line> (inclusive) with the
                 given text in the open file. The replacement text is terminated by a line
                 with only end_of_edit on it. All of the <replacement text> will be entered,
                 so make sure your indentation is formatted properly. Python files will be
                 checked for syntax errors after the edit. If the system detects a syntax error,
                 the edit will not be executed. Simply try to edit the file again, but make
                 sure to read the error message and modify the edit command you issue accordingly.
                 Issuing the same command a second time will just lead to the same error message
                 again.
               end_name: end_of_edit
               name: edit
               signature: |-
                 edit <start_line>:<end_line>
                 <replacement_text>
                 end_of_edit
             _subroutines: {}
             blocklist:
             - vim
             - vi
             - emacs
             - nano
             - nohup
             - git
             blocklist_error_template: Interactive operation '{name}' is not supported by this
               environment
             blocklist_standalone:
             - python
             - python3
             - ipython
             - bash
             - sh
             - exit
             - /bin/bash
             - /bin/sh
             - nohup
             - vi
             - vim
             - emacs
             - nano
             command_docs: |+
               open:
                 docstring: opens the file at the given path in the editor. If line_number is provided, the window will be move to include that line
                 signature: open <path> [<line_number>]
                 arguments:
                   - path (string) [required]: the path to the file to open
                   - line_number (integer) [optional]: the line number to move the window to (if not provided, the window will start at the top of the file)

               goto:
                 docstring: moves the window to show <line_number>
                 signature: goto <line_number>
                 arguments:
                   - line_number (integer) [required]: the line number to move the window to

               scroll_down:
                 docstring: moves the window down {WINDOW} lines
                 signature: scroll_down

               scroll_up:
                 docstring: moves the window down {WINDOW} lines
                 signature: scroll_down

               create:
                 docstring: creates and opens a new file with the given name
                 signature: create <filename>
                 arguments:
                   - filename (string) [required]: the name of the file to create

               submit:
                 docstring: submits your current code and terminates the session
                 signature: submit

               search_dir:
                 docstring: searches for search_term in all files in dir. If dir is not provided, searches in the current directory
                 signature: search_dir <search_term> [<dir>]
                 arguments:
                   - search_term (string) [required]: the term to search for
                   - dir (string) [optional]: the directory to search in (if not provided, searches in the current directory)

               search_file:
                 docstring: searches for search_term in file. If file is not provided, searches in the current open file
                 signature: search_file <search_term> [<file>]
                 arguments:
                   - search_term (string) [required]: the term to search for
                   - file (string) [optional]: the file to search in (if not provided, searches in the current open file)

               find_file:
                 docstring: finds all files with the given name in dir. If dir is not provided, searches in the current directory
                 signature: find_file <file_name> [<dir>]
                 arguments:
                   - file_name (string) [required]: the name of the file to search for
                   - dir (string) [optional]: the directory to search in (if not provided, searches in the current directory)

               edit:
                 docstring: replaces lines <start_line> through <end_line> (inclusive) with the given text in the open file. The replacement text is terminated by a line with only end_of_edit on it. All
         of the <replacement text> will be entered, so make sure your indentation is formatted properly. Python files will be checked for syntax errors after the edit. If the system detects a syntax
         error, the edit will not be executed. Simply try to edit the file again, but make sure to read the error message and modify the edit command you issue accordingly. Issuing the same command a
         second time will just lead to the same error message again.
                 signature: edit <start_line>:<end_line>
               <replacement_text>
               end_of_edit
                 arguments:
                   - start_line (integer) [required]: the line number to start the edit at
                   - end_line (integer) [required]: the line number to end the edit at (inclusive)
                   - replacement_text (string) [required]: the text to replace the current selection with

             command_files:
             - config/commands/defaults.sh
             - config/commands/search.sh
             - config/commands/edit_linting.sh
             - config/commands/_split_string.py
             demonstration_template: |
               Here is a demonstration of how to correctly accomplish this task.
               It is included to show you how to correctly use the interface.
               You do not need to follow exactly what is done in the demonstration.
               --- DEMONSTRATION ---
               {demonstration}
               --- END OF DEMONSTRATION ---
             demonstrations:
             - trajectories/demonstrations/replay__marshmallow-code__marshmallow-1867__default__t-0.20__p-0.95__c-2.00__install-1___install_from_source/marshmallow-code__marshmallow-1867.traj
             env_variables:
               CURRENT_FILE: ''
               CURRENT_LINE: '0'
               OVERLAP: '2'
               SEARCH_FILES: ()
               SEARCH_INDEX: '0'
               SEARCH_RESULTS: ()
               WINDOW: '100'
             format_error_template: |
               Your output was not formatted correctly. You must always include one discussion and one command as part of your response. Make sure you do not have multiple discussion/command tags.
               Please make sure your output precisely matches the following format:
               DISCUSSION
               Discuss here with yourself about what your planning and what you're going to do in this step.

               ```
               command(s) that you're going to run
               ```
             history_processor: {}
             history_processor_args: {}
             instance_template: "We're currently solving the following issue within our repository.\
               \ Here's the issue text:\nISSUE:\n{issue}\n\nINSTRUCTIONS:\nNow, you're going\
               \ to solve this issue on your own. Your terminal session has started and you're\
               \ in the repository's root directory. You can use any bash commands or the special\
               \ interface to help you. Edit all the files you need to and run any checks or\
               \ tests that you want. \nRemember, YOU CAN ONLY ENTER ONE COMMAND AT A TIME.\
               \ You should always wait for feedback after every command. \nWhen you're satisfied\
               \ with all of the changes you've made, you can submit your changes to the code\
               \ base by simply running the submit command.\nNote however that you cannot use\
               \ any interactive session commands (e.g. python, vim) in this environment, but\
               \ you can write scripts and run them. E.g. you can write a python script and\
               \ then run it with `python <script_name>.py`.\n\nNOTE ABOUT THE EDIT COMMAND:\
               \ Indentation really matters! When editing a file, make sure to insert appropriate\
               \ indentation before each line! \n\nIMPORTANT TIPS:\n1. Always start by trying\
               \ to replicate the bug that the issues discusses. \n   If the issue includes\
               \ code for reproducing the bug, we recommend that you re-implement that in your\
               \ environment, and run it to make sure you can reproduce the bug.\n   Then start\
               \ trying to fix it.\n   When you think you've fixed the bug, re-run the bug\
               \ reproduction script to make sure that the bug has indeed been fixed.\n   \n\
               \   If the bug reproduction script does not print anything when it succesfully\
               \ runs, we recommend adding a print(\"Script completed successfully, no errors.\"\
               ) command at the end of the file,\n   so that you can be sure that the script\
               \ indeed ran fine all the way through. \n\n2. If you run a command and it doesn't\
               \ work, try running a different command. A command that did not work once will\
               \ not work the second time unless you modify it!\n\n3. If you open a file and\
               \ need to get to an area around a specific line that is not in the first 100\
               \ lines, say line 583, don't just use the scroll_down command multiple times.\
               \ Instead, use the goto 583 command. It's much quicker. \n   \n4. If the bug\
               \ reproduction script requires inputting/reading a specific file, such as buggy-input.png,\
               \ and you'd like to understand how to input that file, conduct a search in the\
               \ existing repo code, to see whether someone else has already done that. Do\
               \ this by running the command: find_file \"buggy-input.png\" If that doensn't\
               \ work, use the linux 'find' command. \n\n5. Always make sure to look at the\
               \ currently open file and the current working directory (which appears right\
               \ after the currently open file). The currently open file might be in a different\
               \ directory than the working directory! Note that some commands, such as 'create',\
               \ open files, so they might change the current  open file.\n\n6. When editing\
               \ files, it is easy to accidentally specify a wrong line number or to write\
               \ code with incorrect indentation. Always check the code after you issue an\
               \ edit to make sure that it reflects what you wanted to accomplish. If it didn't,\
               \ issue another command to fix it.\n\n7. It may be necessary to install the\
               \ repository from source before you can run code. Please think about how to\
               \ install the environment from the repository directory if you need to do so.\n\
               \   \n\n(Open file: {open_file})\n(Current directory: {working_dir})\nbash-$"
             next_step_no_output_template: |-
               Your command ran successfully and did not produce any output.
               (Open file: {open_file})
               (Current directory: {working_dir})
               bash-$
             next_step_template: |-
               {observation}
               (Open file: {open_file})
               (Current directory: {working_dir})
               bash-$
             parse_command: {}
             parse_function: {}
             put_demos_in_history: false
             state_command:
               arguments: null
               code: |
                 state() {
                   local working_dir="$PWD";
                   if [ -z $CURRENT_FILE ]; then
                       echo '{"open_file": "n/a", "working_dir": "'$working_dir'"}';
                   else
                       echo '{"open_file": "'$(realpath $CURRENT_FILE)'", "working_dir": "'$working_dir'"}';
                   fi
                 };
               docstring: null
               end_name: null
               name: state
               signature: null
             strategy_template: null
             submit_command: submit
             subroutine_types: []
             system_template: "SETTING: You are an autonomous programmer, and you're working\
               \ directly in the command line with a special interface.\n\nThe special interface\
               \ consists of a file editor that shows you {WINDOW} lines of a file at a time.\n\
               In addition to typical bash commands, you can also use the following commands\
               \ to help you navigate and edit files.\n\nCOMMANDS:\n{command_docs}\n\nPlease\
               \ note that THE EDIT COMMAND REQUIRES PROPER INDENTATION. \nIf you'd like to\
               \ add the line '        print(x)' you must fully write that out, with all those\
               \ spaces before the code! Indentation is important and code that is not indented\
               \ correctly will fail and require fixing before it can be run.\n\nRESPONSE FORMAT:\n\
               Your shell prompt is formatted as follows:\n(Open file: <path>) <cwd> $\n\n\
               You need to format your output using two fields; discussion and command.\nYour\
               \ output should always include _one_ discussion and _one_ command field EXACTLY\
               \ as in the following example:\nDISCUSSION\nFirst I'll start by using ls to\
               \ see what files are in the current directory. Then maybe we can look at some\
               \ relevant files to see what they look like.\n```\nls -a\n```\n\nYou should\
               \ only include a *SINGLE* command in the command section and then wait for a\
               \ response from the shell before continuing with more discussion and commands.\
               \ Everything you include in the DISCUSSION section will be saved for future\
               \ reference.\nIf you'd like to issue two commands at once, PLEASE DO NOT DO\
               \ THAT! Please instead first submit just the first command, and then after receiving\
               \ a response you'll be able to issue the second command. \nYou're free to use\
               \ any other bash commands you want (e.g. find, grep, cat, ls, cd) in addition\
               \ to the special commands listed above.\nHowever, the environment does NOT support\
               \ interactive session commands (e.g. python, vim), so please do not invoke them."
             util_functions:
             - arguments: null
               code: '_print() {    local total_lines=$(awk ''END {print NR}'' $CURRENT_FILE)    echo
                 "[File: $(realpath $CURRENT_FILE) ($total_lines lines total)]"    lines_above=$(jq
                 -n "$CURRENT_LINE - $WINDOW/2" | jq ''[0, .] | max | floor'')    lines_below=$(jq
                 -n "$total_lines - $CURRENT_LINE - $WINDOW/2" | jq ''[0, .] | max | round'')    if
                 [ $lines_above -gt 0 ]; then        echo "($lines_above more lines above)"    fi    cat
                 $CURRENT_FILE | grep -n $ | head -n $(jq -n "[$CURRENT_LINE + $WINDOW/2, $WINDOW/2]
                 | max | floor") | tail -n $(jq -n "$WINDOW")    if [ $lines_below -gt 0 ];
                 then        echo "($lines_below more lines below)"    fi}'
               docstring: null
               end_name: null
               name: _print
               signature: _print
             - arguments: null
               code: _constrain_line() {    if [ -z "$CURRENT_FILE" ]    then        echo "No
                 file open. Use the open command first."        return    fi    local max_line=$(awk
                 'END {print NR}' $CURRENT_FILE)    local half_window=$(jq -n "$WINDOW/2" |
                 jq 'floor')    export CURRENT_LINE=$(jq -n "[$CURRENT_LINE, $max_line - $half_window]
                 | min")    export CURRENT_LINE=$(jq -n "[$CURRENT_LINE, $half_window] | max")}
               docstring: null
               end_name: null
               name: _constrain_line
               signature: _constrain_line
           config_file: config/default_from_url.yaml
           model:
             host_url: localhost:11434
             model_name: gpt4
             per_instance_cost_limit: 2.0
             replay_path: null
             temperature: 0.2
             top_p: 0.95
             total_cost_limit: 0.0
         environment:
           base_commit: null
           container_name: null
           data_path: https://github.com/pymc-devs/pymc/issues/7223
           image_name: swe-agent
           install_environment: true
           no_mirror: false
           split: dev
           timeout: 35
           verbose: true
         instance_filter: .*
         skip_existing: true
         suffix: ''

INFO     💽 Loaded dataset from https://github.com/pymc-devs/pymc/issues/7223
Traceback (most recent call last):
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/util/retry.py", line 470, in increment
    raise reraise(type(error), error, _stacktrace)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/util/util.py", line 38, in reraise
    raise value.with_traceback(tb)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/connectionpool.py", line 496, in _make_request
    conn.request(
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/urllib3/connection.py", line 400, in request
    self.endheaders()
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/api/client.py", line 213, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/utils/decorators.py", line 44, in inner
    return f(self, *args, **kwargs)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/api/client.py", line 236, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/twiecki/projects/SWE-agent/run.py", line 223, in <module>
    main(args)
  File "/Users/twiecki/projects/SWE-agent/run.py", line 66, in main
    env = SWEEnv(args.environment)
  File "/Users/twiecki/projects/SWE-agent/sweagent/environment/swe_env.py", line 101, in __init__
    self._reset_container()
  File "/Users/twiecki/projects/SWE-agent/sweagent/environment/swe_env.py", line 349, in _reset_container
    self._init_container()
  File "/Users/twiecki/projects/SWE-agent/sweagent/environment/swe_env.py", line 371, in _init_container
    client = docker.from_env()
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/client.py", line 94, in from_env
    return cls(
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/Users/twiecki/micromamba/envs/swe-agent/lib/python3.9/site-packages/docker/api/client.py", line 220, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

It also didn't seem to find keys.cfg, I had to set GITHUB_TOKEN as an env variable.

Inspector view of patch file after successful `submit` does not allow the user to copy the last newline of the patch file.

When using the inspector and viewing a trajectory with a successful submit, users are able to retrieve the patch file.

However, when a user goes to copy the patch file in the localhost:8000 UI, the last newline of the patch file is unselectable (see screenshot)
Screenshot 2024-04-04 at 12 08 39 PM

I assume this is universal on all browsers and is a function of the

 tag (https://stackoverflow.com/questions/638648/getting-linebreaks-in-pre-tags), but just in case, I am using Firefox 124.0.1 (64-bit).

Workaround: After copying what is selectable from the patch file, add a newline to the patch file manually.

Logs from SWE-Agent running on SWE-Bench

First off, I'd like to say thank you so much for publishing SWE-bench and SWE-agent. I was wonder is their anywhere that has the logs from running SWE-Bench/SWE-ENG evaluation are posted? I am working on some langchain scripts to categorize and groups bugs/features that are being used to evaluate the models/agents and I'd like to dig into what issues fail or succeeded.

I noticed on a previous issue @carlosejimenez provide links to generated results from Claude and GPTs. Is there something similar available but for the logs resulting from evaluation? Thanks so much in advance.

[Feature request] Support adding features

Instead of only fixing issues, consider also adding new features (process issues describing feature requests).

This will accelerate/leverage the development of SWE-Agent itself and provide new real test cases.

Consider taking a look at GPT-Pilot and Meta-GPT.

Feature request: Create a PR on the repo

I gave it a github api token that has access to my playground repo: https://github.com/catid/swe_agent_playground

Token is in the keys.cfg but it doesn't seem to read that so I also manually exported the key envvars.

It completed the task but did not create a PR or branch and does not have any useful errors explaining why:

         Let me know if you would like me to explain or modify anything! Otherwise, I think we can go
         ahead and submit this as the solution.

INFO     🎬 ACTION (primary)
         submit

INFO     Found submission:

INFO     Saved trajectory to
         trajectories/catid/claude-opus__catid__swe_agent_playground__default_from_url__t-0.20__p-0.95__c-
         2.00__install-1/catid__swe_agent_playground-i1.traj
INFO     Saved predictions to
         trajectories/catid/claude-opus__catid__swe_agent_playground__default_from_url__t-0.20__p-0.95__c-
         2.00__install-1/all_preds.jsonl
(swe-agent) ➜  SWE-agent git:(main)

Also no useful logs explaining why it did not submit anything in the trajectories folder.

Also noted that this project fails to start if the repo is empty. I'd prefer to have it work on empty repos too.

Issue with parsing keys.cfg

I run this command to check inference of an issue in a private GitHub repo-

(swe-agent) rasdaman@mohinem-TUF-Gaming-FX505DT-FX505DT:~/SWE-agent$ python run.py --model_name gpt4 --data_path https://github.com/Mohinem/

and get a TokenizerError with the following console log-

Traceback (most recent call last):
  File "/home/rasdaman/SWE-agent/run.py", line 223, in <module>
    main(args)
  File "/home/rasdaman/SWE-agent/run.py", line 64, in main
    agent = Agent("primary", args.agent)
  File "/home/rasdaman/SWE-agent/sweagent/agent/agents.py", line 186, in __init__
    self.model = get_model(args.model, args.config._commands + args.config.subroutine_types)
  File "/home/rasdaman/SWE-agent/sweagent/agent/models.py", line 684, in get_model
    return OpenAIModel(args, commands)
  File "/home/rasdaman/SWE-agent/sweagent/agent/models.py", line 222, in __init__
    cfg = config.Config(os.path.join(os.getcwd(), "keys.cfg"))
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/__init__.py", line 707, in __init__
    self.load_file(stream_or_path, kwargs.get('encoding'))
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/__init__.py", line 816, in load_file
    self.load(f)
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/__init__.py", line 803, in load
    items = p.container()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 292, in container
    result = self.mapping_body()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 183, in mapping_body
    value = self.expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 496, in or_expr
    result = self.and_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 488, in and_expr
    result = self.not_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 481, in not_expr
    result = self.comparison()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 472, in comparison
    result = self.bitor_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 450, in bitor_expr
    result = self.bitxor_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 442, in bitxor_expr
    result = self.bitand_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 434, in bitand_expr
    result = self.shift_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 425, in shift_expr
    result = self.add_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 416, in add_expr
    result = self.mul_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 407, in mul_expr
    result = self.u_expr()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 400, in u_expr
    result = self.power()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 390, in power
    result = self.primary()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 383, in primary
    result = self.atom()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 309, in atom
    result = self.value()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 274, in value
    self.advance()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/parser.py", line 130, in advance
    self.token = self.tokenizer.get_token()
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/tokens.py", line 1154, in get_token
    tt, token, value, endline, endcol = get_number(token)
  File "/home/rasdaman/miniconda3/envs/swe-agent/lib/python3.9/site-packages/config/tokens.py", line 923, in get_number
    raise e
config.tokens.TokenizerError: Invalid character in number: Q

I do not understand this log, but chatgpt4 tells me that it's caused because of parsing errors from keys.cfg configuration file. Here is my config-

OPENAI_API_KEY: sk-691QbGhoMYXTn9nyXXXXT4BlbHFdl2eYcxxxxB5jKpJ9jJpo
GITHUB_TOKEN: ghp_GTiVuqGTUxHGXXXXJTnAp1Jb70gOH0GXXXX

(i have changed the keys of course)

  1. Do I need to add quotes to the API keys ?
  2. Otherwise, is there anything I could do so that the parsing is correct ?

Web interface

Happy to contribute here, I'm sure others are already thinking here but figured I'd write some notes about what would've made trying this out easier:

  • A locally served onboarding and output UI
  • git clone, run make spins up and opens localhost:3000
  • "Playground":
    • Paste GH issue (or file selector->a directory & a textbox for issue description)
    • Config chooser
  • Tail output
  • Choose to apply the diff locally or generate the PR back up to GH
  • Making this available locally vs. hosting the UI and provisioning compute for the agent (e.g. inside a Docker-layer lambda)?

Running SWE-agent with SWE-bench Dataset

SWE-agent Team!

Great work, congrats.

I am attempting to run the SWE-agent on the SWE-bench dataset and have encountered some challenges in understanding the process. I see some scripts in the repository at this link, which seem relevant to what I am trying to achieve.

However, the documentation does not outline the steps for executing these scripts, collecting patches, running them against the test suites, and then gathering the results. Could you please provide help me?

  • Executing Scripts: how to run the scripts located in the scripts directory.
  • Collecting Patches: how to collect patches generated by SWE-agent.
  • Running Test Suites: run these patches against test suites within the SWE-bench dataset.
  • Gathering Results: how to best collect and interpret the results from the test runs.

Thanks!

Ollama support

Would it be possible that you guys add Ollama support? or is it already in progress?

ModuleNotFoundError: No module named 'simple_parsing'

npall@Nathan MINGW64 ~
$ cd documents

npall@Nathan MINGW64 ~/documents
$ cd swe-agent

npall@Nathan MINGW64 ~/documents/swe-agent (main)
$ ./setup.sh
Setting up docker image for swe-agent...
Building the x86 Docker image
2024/04/06 10:26:11 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
[+] Building 1.9s (18/18) FINISHED                                                                       docker:default
 => [internal] load build definition from swe.Dockerfile                                                           0.0s
 => => transferring dockerfile: 1.15kB                                                                             0.0s
 => [internal] load metadata for docker.io/library/ubuntu:jammy                                                    1.8s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                      0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [ 1/13] FROM docker.io/library/ubuntu:jammy@sha256:77906da86b60585ce12215807090eb327e7386c8fafb5402369e421f44  0.0s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 105B                                                                                  0.0s
 => CACHED [ 2/13] RUN apt-get update &&     apt-get install -y bash gcc git jq wget g++ make &&     apt-get clea  0.0s
 => CACHED [ 3/13] RUN git config --global user.email "[email protected]"                                          0.0s
 => CACHED [ 4/13] RUN git config --global user.name "sweagent"                                                    0.0s
 => CACHED [ 5/13] RUN prompt() { echo " > "; };                                                                   0.0s
 => CACHED [ 6/13] RUN touch /root/files_to_edit.txt                                                               0.0s
 => CACHED [ 7/13] RUN touch /root/test.patch                                                                      0.0s
 => CACHED [ 8/13] RUN echo "alias ls='ls -F'" >> /root/.bashrc                                                    0.0s
 => CACHED [ 9/13] RUN wget https://repo.anaconda.com/miniconda/Miniconda3-py39_23.11.0-1-Linux-x86_64.sh -O mini  0.0s
 => CACHED [10/13] RUN conda --version     && conda init bash     && conda config --append channels conda-forge    0.0s
 => CACHED [11/13] COPY docker/requirements.txt /root/requirements.txt                                             0.0s
 => CACHED [12/13] RUN pip install -r /root/requirements.txt                                                       0.0s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:f3e95f3f313c45afd4a59889c3e51ab32ba71db8dd1adf70acb9b4b13d1501c7                       0.0s
 => => naming to docker.io/library/swe-agent                                                                       0.0s

View build details: docker-desktop://dashboard/build/default/default/dhv0k3g83hgdgde12kpz2gjwn

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
Setting up docker image for evaluation...
2024/04/06 10:26:14 http2: server: error reading preface from client //./pipe/docker_engine: file has already been closed
[+] Building 0.1s (9/9) FINISHED                                                                         docker:default
 => [internal] load build definition from eval.Dockerfile                                                          0.0s
 => => transferring dockerfile: 238B                                                                               0.0s
 => [internal] load metadata for docker.io/library/swe-agent:latest                                                0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 10.28kB                                                                               0.0s
 => [1/4] FROM docker.io/library/swe-agent:latest                                                                  0.0s
 => CACHED [2/4] COPY ../evaluation/evaluation.py /evaluation.py                                                   0.0s
 => CACHED [3/4] RUN pip install git+https://github.com/princeton-nlp/SWE-bench.git                                0.0s
 => CACHED [4/4] RUN pip install unidiff                                                                           0.0s
 => exporting to image                                                                                             0.0s
 => => exporting layers                                                                                            0.0s
 => => writing image sha256:1ed08877bde203c7997e59f977bc7d39604dbf017dd6b414f918c4b7c8250de7                       0.0s
 => => naming to docker.io/library/swe-eval                                                                        0.0s

View build details: docker-desktop://dashboard/build/default/default/0800pnq5thztnswk8qhps4o3y

What's Next?
  View a summary of image vulnerabilities and recommendations → docker scout quickview
Done with setup!
npall@Nathan MINGW64 ~/documents/swe-agent (main)
$ python run.py --model_name gpt4 \
  --data_path https://github.com/pdovhomilja/nextcrm-app/issues/30 \
  --config_file config/default_from_url.yaml
Traceback (most recent call last):
  File "C:\Users\npall\documents\swe-agent\run.py", line 13, in <module>
    from simple_parsing import parse
ModuleNotFoundError: No module named 'simple_parsing'

npall@Nathan MINGW64 ~/documents/swe-agent (main)
$ python run.py --model_name gpt4   --data_path https://github.com/pdovhomilja/nextcrm-app/issues/30   --config_file config/default_from_url.yaml
Traceback (most recent call last):
  File "C:\Users\npall\documents\swe-agent\run.py", line 13, in <module>
    from simple_parsing import parse
ModuleNotFoundError: No module named 'simple_parsing'

npall@Nathan MINGW64 ~/documents/swe-agent (main)

Windows 11
Anaconda
Python 3.10
Docker Desktop

Run on local repositories

E.g.:

  • via cli string flag
  • via cli flag pointing to filename

Why:

  • will enable non-github workflows on top

Explore whether 'smart code search' (AST, stack graphs, scope graphs) would improve performance beyond the current find/grep based approach.

Looking at the current 'search' command that the agent has access to, it seems it just uses grep for searches within a file, and find for searches by filename:

# Use grep to directly get the desired formatted output
local matches=$(grep -nH "$search_term" "$file")
# Check if no matches were found
if [ -z "$matches" ]; then
echo "No matches found for \"$search_term\" in $file"
return
fi

local matches=$(find "$dir" -type f -name "$file_name")
# if no matches, return
if [ -z "$matches" ]; then
echo "No matches found for \"$file_name\" in $dir"
return
fi

Then the are the other tools that allow files to be opened/viewed/edited/etc:

https://github.com/princeton-nlp/SWE-agent/blob/main/config/commands/defaults.sh

https://github.com/princeton-nlp/SWE-agent/blob/main/config/commands/edit_linting.sh#L49-L68

It would be interesting to see whether giving some more powerful AST / 'smart' code based search functionality would improve the performance of the agent, particularly on larger/more complex codebases; or tasks that require more complex implementations.

One direction that might be interesting to explore with regards to this is 'stack graphs' / 'scope graphs'. I've included a bunch of references/resources I was exploring today below:

Stack Graphs (an evolution of Scope Graphs) sound like they could be really interesting/useful with regards to code navigation, symbol mapping, etc. Perhaps we could use them for module identification, or variable/function identifier naming stabilisation or similar?

  • https://github.blog/changelog/2024-03-14-precise-code-navigation-for-typescript-projects/
    • Precise code navigation is now available for all TypeScript repositories.
      Precise code navigation gives more accurate results by only considering the set of classes, functions, and imported definitions that are visible at a given point in your code.

      Precise code navigation is powered by the stack graphs framework.
      You can read about how we use stack graphs for code navigation and visit the stack graphs definition for TypeScript to learn more.

      • https://github.blog/2021-12-09-introducing-stack-graphs/
        • Introducing stack graphs

        • Precise code navigation is powered by stack graphs, a new open source framework we’ve created that lets you define the name binding rules for a programming language using a declarative, domain-specific language (DSL). With stack graphs, we can generate code navigation data for a repository without requiring any configuration from the repository owner, and without tapping into a build process or other CI job.

        • LOTS of interesting stuff in this post..
        • As part of developing stack graphs, we’ve added a new graph construction language to Tree-sitter, which lets you construct arbitrary graph structures (including but not limited to stack graphs) from parsed CSTs. You use stanzas to define the gadget of graph nodes and edges that should be created for each occurrence of a Tree-sitter query, and how the newly created nodes and edges should connect to graph content that you’ve already created elsewhere.

        • Why aren’t we using the Language Server Protocol (LSP) or Language Server Index Format (LSIF)?

          To dig even deeper and learn more, I encourage you to check out my Strange Loop talk and the stack-graphs crate: our open source Rust implementation of these ideas.

  • https://docs.github.com/en/repositories/working-with-files/using-files/navigating-code-on-github
    • GitHub has developed two code navigation approaches based on the open source tree-sitter and stack-graphs library:

      • Search-based - searches all definitions and references across a repository to find entities with a given name
      • Precise - resolves definitions and references based on the set of classes, functions, and imported definitions at a given point in your code

      To learn more about these approaches, see "Precise and search-based navigation."

      • https://docs.github.com/en/repositories/working-with-files/using-files/navigating-code-on-github#precise-and-search-based-navigation
        • Precise and search-based navigation
          Certain languages supported by GitHub have access to precise code navigation, which uses an algorithm (based on the open source stack-graphs library) that resolves definitions and references based on the set of classes, functions, and imported definitions that are visible at any given point in your code. Other languages use search-based code navigation, which searches all definitions and references across a repository to find entities with a given name. Both strategies are effective at finding results and both make sure to avoid inappropriate results such as comments, but precise code navigation can give more accurate results, especially when a repository contains multiple methods or functions with the same name.

  • https://pl.ewi.tudelft.nl/research/projects/scope-graphs/
    • Scope Graphs | A Theory of Name Resolution

    • Scope graphs provide a new approach to defining the name binding rules of programming languages. A scope graph represents the name binding facts of a program using the basic concepts of declarations and reference associated with scopes that are connected by edges. Name resolution is defined by searching for paths from references to declarations in a scope graph. Scope graph diagrams provide an illuminating visual notation for explaining the bindings in programs.

Originally posted by @0xdevalias in 0xdevalias/chatgpt-source-watch#11

How to use local Ollama models?

Hello, I'm trying to figure out how should I provide the model_name for Ollama models which are in my local machine. I am already running Ollama and have mistral installed.

python run.py --model_name ollama/mistral --data_path <github issue url>

SWE-agent as a github action or bot

Automatically process SWE-agent repo issues.
Investing effort in this more complex feature will accelerate/leverage future development of SWE-agent itself.

Incorrect error handling in run_replay.py when missing config_file

python run_replay.py --traj_path=trajectories/fuchur/gpt4__klieret__swe-agent-test-repo__default_from_url__t-0.20__p-0.95__c-2.00__install-1/klieret__swe-agent-test-repo-i1.traj --data_path=trajectories/fuchur/gpt4__klieret__swe-agent-test-repo__default_from_url__t-0.20__p-0.95__c-2.00__install-1/all_preds.jsonl
Traceback (most recent call last):
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/run_replay.py", line 142, in <module>
    main(**vars(args))
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/run_replay.py", line 125, in main
    process_single_traj(traj_path, config_file, data_path, suffix)
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/run_replay.py", line 109, in process_single_traj
    subprocess.run(command)
  File "/Users/fuchur/micromamba/envs/swe-agent/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Users/fuchur/micromamba/envs/swe-agent/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/fuchur/micromamba/envs/swe-agent/lib/python3.9/subprocess.py", line 1770, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
TypeError: expected str, bytes or os.PathLike object, not NoneType

Existing args.yaml with different arguments

When calling run.py more than one time with the same command it complains that args.yaml already exists and quits. So I have to manually delete the relevant trajectories folder.

Since I couldn't get any run going in #16 due to context length limits, I assume that the code stops at this try catch block

sweagent/agent/model.py

try:
            # Perform OpenAI API call
            print(self.history_to_messages(history)) 
            response = self.client.chat.completions.create(
                messages=self.history_to_messages(history),
                model=self.api_model,
                temperature=self.args.temperature,
                top_p=self.args.top_p,
            )
        except BadRequestError as e:
            raise CostLimitExceededError(f"Context window ({self.model_metadata['max_context']} tokens) exceeded")

Altough I dont get the Context window ({self.model_metadata['max_context']} tokens) exceeded message in the console 🤔

run 1

...
2024-04-05 16:43:55,156 - httpx - INFO - HTTP Request: POST <REDACTED>/chat/completions?api-version=<REDACTED> "HTTP/1.1 400 model_error"
WARNING  Cost limit exceeded                                                                                                                                                                                                                                                                                                                                                                                             
INFO     💭 THOUGHT (primary)                                                                                                                                                                                                                                                                                                                                                                                            
       Exit due to cost limit                                                                                                                                                                                                                                                                                                                                                                                          
INFO     🎬 ACTION (primary)                                                                                                                                                                                                                                                                                                                                                                                             
       exit_cost                                                                                                                                                                                                                                                                                                                                                                                                       
INFO     Saved trajectory to trajectories/<REDACTED>/<REDACTED>__kim-borgen__swe-agent-test__testtt__t-0.20__p-0.95__c-2.00__install-1/kim-borgen__swe-agent-test-i1.traj                                                                                                                                                                                                                                                 
INFO     Saved predictions to trajectories/<REDACTED>/<REDACTED>__kim-borgen__swe-agent-test__testtt__t-0.20__p-0.95__c-2.00__install-1/all_preds.jsonl   

run 2

...
INFO     💽 Loaded dataset from https://github.com/kim-borgen/swe-agent-test/issues/1                                                                                                                                                                                                                                                                                                                                    
INFO     🌱 Environment Initialized                                                                                                                                                                                                                                                                                                                                                                                      
WARNING  **************************************************                                                                                                                                                                                                                                                                                                                                                              
WARNING  Found existing args.yaml with different arguments!                                                                                                                                                                                                                                                                                                                                                              
WARNING  **************************************************                                                                                                                                                                                                                                                                                                                                                              
INFO     ⏭️ Skipping existing trajectory: trajectories/<REDACTED>/<REDACTED>__kim-borgen__swe-agent-test__testtt__t-0.20__p-0.95__c-2.00__install-1/kim-borgen__swe-agent-test-i1.traj 

run_replay.py cannot deal with github urls and gets stuck in infinite loop

python run_replay.py --traj_path=trajectories/fuchur/gpt4__klieret__swe-agent-test-repo__default_from_url__t-0.20__p-0.95__c-2.00__install-1/klieret__swe-agent-test-repo-i1.traj --data_path=trajectories/fuchur/gpt4__klieret__swe-agent-test-repo__default_from_url__t-0.20__p-0.95__c-2.00__install-1/all_preds.jsonl --config_file=config/default_from_url.yaml   
INFO     💽 Loaded dataset from klieret__swe-agent-test-repo-i1.jsonl
INFO     🌱 Environment Initialized
INFO     ▶️  Beginning task 0
Traceback (most recent call last):
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/run.py", line 85, in main
    observation, info = env.reset(index)
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/sweagent/environment/swe_env.py", line 135, in reset
    self.base_commit = self.record["base_commit"]
KeyError: 'base_commit'
WARNING  ❌ Failed on klieret__swe-agent-test-repo-i1: 'base_commit'
INFO     Beginning environment shutdown...
INFO     Agent container stopped
INFO     🌱 Environment Initialized
INFO     ▶️  Beginning task 1
Traceback (most recent call last):
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/run.py", line 85, in main
    observation, info = env.reset(index)
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/sweagent/environment/swe_env.py", line 135, in reset
    self.base_commit = self.record["base_commit"]
KeyError: 'base_commit'
WARNING  ❌ Failed on klieret__swe-agent-test-repo-i1: 'base_commit'
INFO     Beginning environment shutdown...
INFO     Agent container stopped
INFO     🌱 Environment Initialized
INFO     ▶️  Beginning task 2
Traceback (most recent call last):
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/run.py", line 85, in main
    observation, info = env.reset(index)
  File "/Users/fuchur/Documents/24/git_sync/SWE-agent/sweagent/environment/swe_env.py", line 135, in reset
    self.base_commit = self.record["base_commit"]
KeyError: 'base_commit'

Streamlining the installation and usage process

(posting at @ofirpress's request)

The current installation process requires a number of steps, and requires a non-trivial investment for non-contributors. For example, someone from discord recently mentioned that this process took them 2 days.

Ideally swe-agent would have a single CLI command that anyone could run from any machine, regardless of whether they have cloned the repo or downloaded any dependencies:

$ swe-agent \
  --github-issue=https://github.com/princeton-nlp/SWE-agent/issues/57 \
  --model=gpt4 \
  --openai_api_key=... \
  --cost-limit=2.00

Design possibilities

Note that none of these solutions are mutually exclusive.

Docker container

Usage: docker run princeton-nlp/swe-agent ...

In this solution, we would publish a Docker container with CMD set to python run.py.

Considerations:

  • Would require the user to have Docker installed before running the command.
  • Would involve docker-in-docker, which is do-able but brings its own subtleties.

Nix flake

Usage: nix run github:princeton-nlp/SWE-agent

In this solution, we would package swe-agent as a nix flake. This has the advantage of increasing reproducibility and obviating the need for the current setup.py/pyproject.toml situation.

Considerations:

  • Would require the user to have Nix, and possibly Docker, installed before running the command. While nix has packaged the Docker CLI, the Docker daemon will likely require a native installation on non-NixOS operating systems.
  • Pro: would not require docker-in-docker
  • Pro: dependency pulling and caching would be handled automatically by Nix

curl-to-shell

Usage: curl https://github.com/princeton-nlp/SWE-agent/blob/main/run.sh | sh

Here we would be following in the footsteps of rustup, the Nix installer, and friends.

Considerations

  • Pro: would involve minimal changes to the existing packaging of SWE-agent
  • Pro: would require effectively no pre-requisites
  • Con: would require prompting for sudo to install Docker on Linux, and may not be possible on macOS
  • Con: would require implementing logic to query, download, and cache this repo and its dependencies on the user's local machine. This would effectively be a reimplementation of package management logic that already exists in Nix/Docker.
  • Con: curl-to-sh is commonly chided for its security implications. Some people are reluctant to run curl-to-sh commands.

Broken links in README

The swe-agent/agent/ and swe-agent/environment/ links (as highlighted in the below screenshot) in the Inference section of the readme are stale and return 404 pages. They were likely rendered stale when the top level agent and environment directories were moved under the sweagent directory.

Current path (incorrect): princeton-nlp/SWE-agent/blob/main/agent

Correct path: princeton-nlp/SWE-agent/tree/main/sweagent/agent

Screenshot 2024-04-03 at 12 26 38 PM

Decouple linter from Python to enable other language workflows

The following four files rely on flake8 to produce either a file or snippit lint output:

Flake8 is a python linter -> to run this project on other languages, it will need to either:

  • continue along the existing lines: introspect the language being generated and run a pre-assigned linter for that language
  • introspect the existing linting configuration installed in the project (e.g. make lint, npm run lint etc)
  • another approach?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.