joshpxyne / gpt-migrate Goto Github PK

View Code? Open in Web Editor NEW

6.7K 57.0 475.0 171 KB

Easily migrate your codebase from one framework or language to another.

Home Page: https://gpt-migrate.com

License: MIT License

Python 100.00%

gpt-migrate's Introduction

◐ GPT-Migrate ◑

Easily migrate your codebase from one framework or language to another.

If you've ever faced the pain of migrating a codebase to a new framework or language, this project is for you.

GPT-Migrate.mp4

Migration is a costly, tedious, and non-trivial problem. Do not trust the current version blindly and please use responsibly. Please also be aware that costs can add up quickly as GPT-Migrate is designed to write (and potentially re-write) the entirety of a codebase.

However, with the collective brilliance of the OSS community and the current state of LLMs, it is also a very tractable problem.

⚡️ Usage

Install Docker and ensure that it's running. It's also recommended that you use at least GPT-4, preferably GPT-4-32k.

📦 Installation using Poetry

Install Poetry by following the instructions on the official Poetry website.
Once Poetry is installed, navigate to the project directory and install the project dependencies using the following command:

poetry install

This will create a virtual environment and install all the necessary dependencies in that environment.

Set your OpenRouter API key (default) and/or your OpenAI API key (to use the OpenAI API directly...in this case, set --model to gpt-4-32k or your desired model) and install the python requirements:

export OPENROUTER_API_KEY=<your key> export OPENAI_API_KEY=<your key> pip install -r requirements.txt

Run the main script with the target language you want to migrate to:

python main.py --targetlang nodejs

(Optional) If you'd like GPT-Migrate to validate the unit tests it creates against your app before it tests the migrated app with them, please have your existing app exposed and use the --sourceport flag. For executing this against the benchmark, open a separate terminal, navigate to the benchmarks/language-pair/source directory, and run python app.py after installing the requirements. It will expose on port 5000. Use this with the --sourceport flag.

By default, this script will execute the flask-nodejs benchmark. You can specify the language, source directory, and many other things using the options guide below.

💡 Options

You can customize the behavior of GPT-Migrate by passing the following options to the main.py script:

--model: The Large Language Model to be used. Default is "gpt-4-32k".
--temperature: Temperature setting for the AI model. Default is 0.
--sourcedir: Source directory containing the code to be migrated. Default is "../benchmarks/flask-nodejs/source".
--sourcelang: Source language or framework of the code to be migrated. No default value.
--sourceentry: Entrypoint filename relative to the source directory. For instance, this could be an app.py or main.py file for Python. Default is "app.py".
--targetdir: Directory where the migrated code will live. Default is "../benchmarks/flask-nodejs/target".
--targetlang: Target language or framework for migration. Default is "nodejs".
--operating_system: Operating system for the Dockerfile. Common options are 'linux' or 'windows'. Default is 'linux'.
--testfiles: Comma-separated list of files that have functions to be tested. For instance, this could be an app.py or main.py file for a Python app where your REST endpoints are. Include the full relative path. Default is "app.py".
--sourceport: (Optional) Port for testing the unit tests file against the original app. No default value. If not included, GPT-Migrate will not attempt to test the unit tests against your original app.
--targetport: Port for testing the unit tests file against the migrated app. Default is 8080.
--guidelines: Stylistic or small functional guidelines that you'd like to be followed during the migration. For instance, "Use tabs, not spaces". Default is an empty string.
--step: Step to run. Options are 'setup', 'migrate', 'test', 'all'. Default is 'all'.

For example, to migrate a Python codebase to Node.js, you might run:

python main.py --sourcedir /path/to/my-python-app --sourceentry app.py --targetdir /path/to/my-nodejs-app --targetlang nodejs

This will take the Python code in ./my-python-app, migrate it to Node.js, and write the resulting code to ./my-nodejs-app.

GPT-assisted debugging

GPT-Migrate-debugging.mp4

🤖 How it Works

For migrating a repo from --sourcelang to --targetlang...

GPT-Migrate first creates a Docker environment for --targetlang, which is either passed in or assessed automatically by GPT-Migrate.
It evaluates your existing code recursively to identify 3rd-party --sourcelang dependencies and selects corresponding --targetlang dependencies.
It recursively rebuilds new --targetlang code from your existing code starting from your designated --sourceentry file. This step can be started from with the --step migrate option.
It spins up the Docker environment with the new codebase, exposing it on --targetport and iteratively debugging as needed.
It develops unit tests using Python's unittest framework, and optionally tests these against your existing app if it's running and exposed on --sourceport, iteratively debugging as needed. This step can be started from with the --step test option.
It tests the new code on --targetport against these unit tests.
It iteratively debugs the code for for you with context from logs, error messages, relevant files, and directory structure. It does so by choosing one or more actions (move, create, or edit files) then executing them. If it wants to execute any sort of shell script (moving files around), it will first ask for clearance. Finally, if at any point it gets stuck or the user ends the debugging loop, it will output directions for the user to follow to move to the next step of the migration.
The new codebase is completed and exists in --targetdir.

📝 Prompt Design

Subprompts are organized in the following fashion:

HIERARCHY: this defines the notion of preferences. There are 4 levels of preference, and each level prioritized more highly than the previous one.
p1: Preference Level 1. These are the most general prompts, and consist of broad guidelines.
p2: Preference Level 2. These are more specific prompts, and consist of guidelines for certain types of actions (e.g., best practices and philosophies for writing code).
p3: Preference Level 3. These are even more specific prompts, and consist of directions for specific actions (e.g., creating a certain file, debugging, writing tests).
p4: Preference Level 4. These are the most specific prompts, and consist of formatting for output.

Prompts are a combination of subprompts. This concept of tagging and composability can be extended to other properties as well to make prompts even more robust. This is an area we're highly interested in actively exploring.

In this repo, the prompt_constructor() function takes in one or more subprompts and yields a string which may be formatted with variables, for example with GUIDELINES being a p1, WRITE_CODE being a p2 etc:

prompt = prompt_constructor(HIERARCHY, GUIDELINES, WRITE_CODE, DEBUG_TESTFILE, SINGLEFILE).format(targetlang=targetlang,buggyfile=buggyfile)

📈 Performance

GPT-Migrate is currently in development alpha and is not yet ready for production use. For instance, on the relatively simple benchmarks, it gets through "easy" languages like python or javascript without a hitch ~50% of the time, and cannot get through more complex languages like C++ or Rust without some human assistance.

✅ Benchmarks

We're actively looking to build up a robust benchmark repository. If you have a codebase that you'd like to contribute, please open a PR! The current benchmarks were built from scratch: REST API apps which have a few endpoints and dependency files.

🧗 Roadmap

Below are improvements on the to-do list. If you'd like to knock any of these or others out, please submit a PR :)

High urgency

Add logic for model input size limiting based on the window size. See issue #2.

Med urgency

Add unit tests to the entire project for better reliability and CI/CD
Add more benchmark examples, especially larger repos
Add functionality to let the LLM request access to dependency functions in other files as it debugs
Add support for other LLMs

Low urgency

Enable internet search requests as the model debugs
Identify and compile language-specific issues + solve for them

📣 Call to Action

We're looking for talented contributors. Whether you have a particular passion about a specific language or framework, want to help in creating a more robust test suite, or generally have interesting ideas on how to make this better, we'd love to have you!

🛠 Expert-Assisted Migration

Due to the inflow of requests, we've decided to create a standardized process for helping people with their migrations. If you're a company that needs help with a big migration or an expert that is willing to help with them, please visit the following website: https://gpt-migrate.com/

Join the conversation on Twitter!

gpt-migrate's People

Contributors

Stargazers

Watchers

Forkers

hellyeah alumny mevengue iuzn morgante d3287t328 cwalk19 ai-awe commerceless justmuteai alieismy youminxue evdcush donutloop mrcodechef inayet ctr26 xadhrit dan-wood lrochetta ronhuafeng sheryaarbutt pynchmeister bluntworks redbird96 pelly liamdgray berkeleynerd hbcbh1999 apollohuang1 seshakiran devdoshi dfa1234 ai-jie01 jeremylcarter nusu77 jivane rac-sri amaranthlis canburaks raja-vcomply jorgehuck ssahgal rx350h tonywhite11 mmizutani kumar045 code-sourabh georasaq wodole jansystemic mfkiwl techthiyanes kerlic hanvv3 gelove qqq-tech sorokinvld uijnn amkev101 jknkh ontilitbi fdsfsww ornfelt andyoulovexy sdi82 tuilahoangne maleficent-magik bayuapriansah liuzhen153 lephyltone kmlllb monssegcompyu damonclifford 00mjk fabianhornn 9acoxexga tuckerweb3 shaofengz035 8tianaclinn bangenlanbai mebap162 denisovagap tocisxhosn 0maslowneni neivayang 8crabalpitzo shineovo drnitr pirrone574s 0thephasergo alexischartrand mrnnmaclean mostafaashraf97 debuviosu farmer322 8uk0w5k1 gmh5225 mkuuwaujinga malcoolboy7

gpt-migrate's Issues

Vector embeddings for different languages?

Hello! I am trying to learn a little more about gpt-migrate. Does gpt-migrate access a vector database with documentation for various languages, or is this pure gpt capability?

Thank you for this cool tool!

ValueError: No valid completion model args passed in

I'm trying to migrate a project from Fortran95 (yes, I know it's old, that's why it needs to be migrated... 😁) to Dart, but I get the following error:

Traceback (most recent call last):

  File "/var/home/agardh/Development/gpt-migrate/gpt_migrate/main.py", line 127, in <module>
    app()

  File "/var/home/agardh/Development/gpt-migrate/gpt_migrate/main.py", line 87, in main
    create_environment(globals)

  File "/var/home/agardh/Development/gpt-migrate/gpt_migrate/steps/setup.py", line 15, in create_environment
    llm_write_file(prompt,

  File "/var/home/agardh/Development/gpt-migrate/gpt_migrate/utils.py", line 52, in llm_write_file
    file_name,language,file_content = globals.ai.write_code(prompt)[0]
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/var/home/agardh/Development/gpt-migrate/gpt_migrate/ai.py", line 23, in write_code
    response = completion(
               ^^^^^^^^^^^

  File "/var/home/agardh/.local/lib/python3.11/site-packages/litellm/utils.py", line 98, in wrapper
    raise e

  File "/var/home/agardh/.local/lib/python3.11/site-packages/litellm/utils.py", line 89, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/var/home/agardh/.local/lib/python3.11/site-packages/litellm/timeout.py", line 44, in wrapper
    result = future.result(timeout=local_timeout_duration)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^

  File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception

  File "/var/home/agardh/.local/lib/python3.11/site-packages/litellm/timeout.py", line 35, in async_func
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "/var/home/agardh/.local/lib/python3.11/site-packages/litellm/main.py", line 248, in completion
    raise exception_type(model=model, original_exception=e)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/var/home/agardh/.local/lib/python3.11/site-packages/litellm/utils.py", line 273, in exception_type
    raise original_exception # base case - return the original exception
    ^^^^^^^^^^^^^^^^^^^^^^^^

  File "/var/home/agardh/.local/lib/python3.11/site-packages/litellm/main.py", line 242, in completion
    raise ValueError(f"No valid completion model args passed in - {args}")

It also prints all the parameters passed to the model, if you want I can give you those too.

The command I run is:

python3 main.py --sourcedir "../my/source" --sourceentry md_3PG.f95 --sourcelang fortran95 --targetlang dart --targetdir "../my/target"

I couldn't find anyone else had encountered the same problem, so I thought I might open an Issue.

I have tried a few different models, but that doesn't seem to make a difference; all of them throw the same error. Except when using the model "gpt-3.5-turbo", then it insteads complains that the model's maximum context length has been reached.

Benchmarks fail for the gpt-3.5-turbo model family

Hi Josh,

thanks for this awesome project 🎉.

Some observations from running python main.py --sourcedir ../benchmarks/flask-nodejs/source --targetlang nodejs --model gpt-3.5-turbo-16k --temperature 0:

The Dockerfile is already not generated correctly. It just contains one line named CODE. This is probably due to the instructions in p4_output_formats/single_file.
Internal dependency resolve of benchmarks/flask-nodejs/source/db.py is stuck in an infinite loop cause the output of the LLM is [db.py] again.

Perhaps this can be fixed with a lot of prompt engineering but as of now I'd probably communicate gpt-3.5-turbo is not supported. Wdyt?
Also, you could easily check that file x isn't in the list of internal dependencies of file x to fix getting stuck in an infinite loop. Happy to add a PR for this.

Lack of Internal Dependency Files from python source project

Hello,

During the project setup, I noticed that a source project does not have any internal dependency files.

I propose the following solution to generate a list of dependencies for this project:

Solution Steps:

Use Pipdeptree to generate a list of dependencies.

Install pipdeptree:

$ pip install pipdeptree

As an example, let's say we want to generate a dependency tree for a package named 'requests'. Execute the following command:

$ pipdeptree -p requests

This should provide an output similar to the following:

requests==2.23.0
  - certifi [required: >=2017.4.17, installed: 2020.4.5.1]
  - chardet [required: >=3.0.2,<4, installed: 3.0.4]
  - idna [required: >=2.5,<3, installed: 2.9]
  - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.9]

Next, copy these dependencies and their version information into a requirements.txt file, like so:

certifi>=2017.4.17
chardet>=3.0.2,<4
idna>=2.5,<3
urllib3>=1.21.1,<1.26,!=1.25.1,!=1.25.0

Lastly, you can download the dependencies to the current directory without installing them by using the following command:

(current directory) $ pip download -r requirements.txt

This method will allow us make the project set-up smoother for python source projects. Thanks.

original link: https://www.activestate.com/resources/quick-reads/how-to-download-python-dependencies/

Issue with GPT-3.5 for Python to PHP

Created Docker environment for php project in directory '/home/genealogia/projects/php-dna'.
✅ Identifying external dependencies for init.py...
✅ Identifying internal dependencies for init.py...
✅ Identifying external dependencies for io/reader.py...
✅ Identifying internal dependencies for io/reader.py...
✅ Identifying external dependencies for io/init.py...
✅ Identifying internal dependencies for io/init.py...
✅ Creating migration file for io/init.py...
Created file_name.ext at /home/genealogia/projects/php-dna
Traceback (most recent call last):

File "/home/genealogia/projects/gpt-migrate/gpt_migrate/main.py", line 126, in
app()

File "/home/genealogia/projects/gpt-migrate/gpt_migrate/main.py", line 99, in main
migrate(sourceentry, globals)

File "/home/genealogia/projects/gpt-migrate/gpt_migrate/main.py", line 96, in migrate
migrate(dependency, globals)

File "/home/genealogia/projects/gpt-migrate/gpt_migrate/main.py", line 97, in migrate
write_migration(sourcefile, external_deps_list, globals)

File "/home/genealogia/projects/gpt-migrate/gpt_migrate/steps/migrate.py", line 69, in write_migration
llm_write_file(prompt,

File "/home/genealogia/projects/gpt-migrate/gpt_migrate/utils.py", line 51, in llm_write_file
file_name,language,file_content = globals.ai.write_code(prompt)[0]

IndexError: list index out of range

Target repository doesn't make use of internal functions

Saw this using the benchmark flask-nodejs:

app.js is not importing db.js but instead uses an external library node-json-db: const db = new JsonDB(new Config("storage/items", true, false, '/'));.

What about adding the internal function signatures, which the source file depends on, to the migration prompt for the target file? This should probably fix it :) Related to #5.

Add distributed inference

Hi,

I've been looking to write something like this project myself. I've been researching how I would do source conversion, but distributed. It seems like you've done most of the heavy lifting for the general conversion, so it seems like it would be a good idea for me to contribute to this project instead of starting from scratch.

I saw that this project had some integration with litellm, so the distributed part can be done using that. I've also written some code to split source files up intelligently which could be used with this.

Is this project still active? If so, I would love to discuss more about contributing.

Add azure support <Feature request>

Please add azure openai support that will receive a different openai endpoint.

TypeError: can only concatenate str (not "NoneType") to str

I'm getting this on pretty much any command. Even when going from python to python to try having it debug, test, etc or convert to different libraries with the --guidelines.

Here's the full console output (note this is an example with an obscure language, but getting the same on py to py):

 ➜ /workspaces/ODB/gpt-migrate-main/gpt_migrate (main) $ python main.py --sourcelang EasyLanguage --sourcedir /workspaces/ODB/Python/ELtoPy/ELSrc --sourceentry OD_23.02-Strategy.el --guidelines "You specialize in trading algorithms, you are an expert in converting TradeStation EasyLanguage code to Python using the Lumibot library" --targetdir /workspaces/ODB/Python/ELtoPy/PyTgt --targetlang python 

◐ Reading EasyLanguage project from directory '/workspaces/ODB/Python/ELtoPy/ELSrc', with entrypoint 'OD_23.02-Strategy.el'.
◑ Outputting python project to directory '/workspaces/ODB/Python/ELtoPy/PyTgt'.
Source directory structure: 

        ├── OD Strat Mashup.el
        ├── ELtoPySrc/
            │   └── OD_23.02-Strategy_GH-CoPilot1.py
        ├── OD_23.02-Strategy.el
        ├── OD_23.02-Strategy copy.el
        └── OD_23.02-Strategy_GH-CoPilot1.py

✅  Creating your environment...
Created Docker environment for python project in directory '/workspaces/ODB/Python/ELtoPy/PyTgt'.
Traceback (most recent call last):

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/main.py", line 127, in <module>
    app()

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/main.py", line 100, in main
    migrate(sourceentry, globals)

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/main.py", line 94, in migrate
    internal_deps_list, external_deps_list = get_dependencies(sourcefile=sourcefile,globals=globals)

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/steps/migrate.py", line 58, in get_dependencies
    external_dependencies = llm_run(prompt,

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/utils.py", line 39, in llm_run
    output = globals.ai.run(prompt)

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/ai.py", line 49, in run
    chat += msg

TypeError: can only concatenate str (not "NoneType") to str

Another example:

➜ /workspaces/ODB/gpt-migrate-main/gpt_migrate (main) $ python main.py --sourcelang python --sourcedir /workspaces/ODB/Python/ELtoPy/ELSrc/ELtoPySrc --sourceentry OD_23.02-Strategy_GH-CoPilot1.py --targetdir /workspaces/ODB/Python/ELtoPy/PyTgt --targetlang python 
◐ Reading python project from directory '/workspaces/ODB/Python/ELtoPy/ELSrc/ELtoPySrc', with entrypoint 'OD_23.02-Strategy_GH-CoPilot1.py'.
◑ Outputting python project to directory '/workspaces/ODB/Python/ELtoPy/PyTgt'.
Source directory structure: 

        └── OD_23.02-Strategy_GH-CoPilot1.py

✅  Creating your environment...
Created Docker environment for python project in directory '/workspaces/ODB/Python/ELtoPy/PyTgt'.
Traceback (most recent call last):

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/main.py", line 127, in <module>
    app()

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/main.py", line 100, in main
    migrate(sourceentry, globals)

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/main.py", line 94, in migrate
    internal_deps_list, external_deps_list = get_dependencies(sourcefile=sourcefile,globals=globals)

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/steps/migrate.py", line 58, in get_dependencies
    external_dependencies = llm_run(prompt,

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/utils.py", line 39, in llm_run
    output = globals.ai.run(prompt)

  File "/workspaces/ODB/gpt-migrate-main/gpt_migrate/ai.py", line 49, in run
    chat += msg

TypeError: can only concatenate str (not "NoneType") to str

Support for libraries

Currently, gpt-migrate only works with code repositories that run as applications with a main file as entry point. However, many code bases are libraries. Libraries are different in that they:

Don't have a dedicated entry point
Often come with a test suite

The migration problem is equally important, so I think it'd be great if gpt-migrate could support this. For that, we'd have to allow for:

Several entry points (or no entry point and a separate logic to determine all sources of the dependency DAG)
The possibility to specify your own test suite (and migrate it to the new target repo if possible)

LiteLLM Improvements

Hi @joshpxyne,

Thanks for merging our PR for LiteLLM.

I'm one of the co-maintainers of the package. How can we make it better for you?

Demanding 320k tokens [ISSUE STILL NOT RESOLVED BUT CLOSED]

When I gave the command it thought for a long long time and then finally when it gave a response it said 32k tokens was not enough! It demands 320k tokens for my small app which is only 440 MB! Sucks

Sweep Github App

Hi! I’m one of the developers of Sweep, a github app that solves issues by writing pull requests. I see some good issues for Sweep https://github.com/sweepai/sweep to try. We have onboarding instructions here, I’m also happy to help you onboard directly :)

I wrote an initial PR wwzeng1#3 solving issue #2, I don't know if it's quite what you want though.

Openai error "ModuleNotFoundError: No module named 'openai.error'"

When running python main.py --targetlang nodejs I go this errror

Traceback (most recent call last):
  File "/Users/mac/Desktop/dev/abel/gpt-migrate/gpt_migrate/main.py", line 7, in <module>
    from ai import AI
  File "/Users/mac/Desktop/dev/abel/gpt-migrate/gpt_migrate/ai.py", line 6, in <module>
    from litellm import completion
  File "/Users/mac/Desktop/dev/abel/gpt-migrate/gpt_migrate/env/lib/python3.11/site-packages/litellm/__init__.py", line 28, in <module>
    from .timeout import timeout
  File "/Users/mac/Desktop/dev/abel/gpt-migrate/gpt_migrate/env/lib/python3.11/site-packages/litellm/timeout.py", line 11, in <module>
    from openai.error import Timeout
ModuleNotFoundError: No module named 'openai.error'

[FEATURE REQUEST]: Using this for more than migration.

I'm unsure about the exact title of this project, but I envision it going beyond a simple migration to a different language. I'm wondering if it could be utilized to make code changes as well.

Let's say you have a massive project and you want to replace all the int values with long values. Doing this manually would be challenging and time-consuming, especially considering other related conversions like IntegerToFloat. However, with GPT, we could automate these tedious tasks effortlessly.

I hope I explained my idea clearly.

Demos

GPT-Migrate.mp4

GPT-Migrate-debugging.mp4

Small todos

logic for model input size limiting based on window size
~~add feature flag for style (eg, "I want the output to use spaces instead of tabs")~~
Let the LLM select a "search google" option in the debug step
add natural language files in the source for the LLM to integrate into proper code, compatible with the rest of the project

Langchain / OpenAI Dependencies

I'm getting:

/home/teamcoltra/.local/lib/python3.10/site-packages/langchain/chat_models/__init__.py:31: LangChainDeprecationWarning: Importing chat models from langchain is deprecated. Importing from langchain will no longer be supported as of langchain==0.2.0. Please import from langchain-community instead:

`from langchain_community.chat_models import ChatOpenAI`.

To install langchain-community run `pip install -U langchain-community`.
  warnings.warn(
Traceback (most recent call last):
  File "/home/teamcoltra/gpt-migrate-main/gpt_migrate/main.py", line 7, in <module>
    from ai import AI
  File "/home/teamcoltra/gpt-migrate-main/gpt_migrate/ai.py", line 6, in <module>
    from litellm import completion
  File "/home/teamcoltra/.local/lib/python3.10/site-packages/litellm/__init__.py", line 28, in <module>
    from .timeout import timeout
  File "/home/teamcoltra/.local/lib/python3.10/site-packages/litellm/timeout.py", line 11, in <module>
    from openai.error import Timeout
ModuleNotFoundError: No module named 'openai.error'

I don't know if you want to update the code to be up-to-date with the modern langchain version or set a version number in requirements.

Use poetry for installation

Happy to help write this using Sweep https://github.com/sweepai/sweep

zespri

Support for OpenRouter

It would be great to use other GPT-like API endpoints like OpenRouter. For example, this would allow anyone to use gpt-4-32k even if they have no access from their OpenAI accounts, since OpenAI is no longer giving access to gpt-4-32k for the time being, and this model is basically a requisite to use gpt-migrate.

Rate limit error on Python 3 to PHP

openai.error.RateLimitError: Rate limit reached for default-gpt-3.5-turbo-16k in organization org-xxxxxxxxxxxxxxx on tokens per min. Limit: 180000 / min. Current: 178656 / min. Contact us through our help center at help.openai.com if you continue to have issues.

I don't have gpt-4 trial. It has a lot of lines appearing for same python file. Is it stuck in a loop? I'm trying to port https://github.com/apriha/lineage to PHP 8.2 from Python 3.

Sorry I should of described this in my previous issue that was closed.

Error

Invalid response object from API: 'Request Entity Too Large' (HTTP response code was 413) with openrouter gpt-4-32k

I am encountering an error when using the openrouter gpt-4-32k API. The error message I'm receiving is as follows:

Invalid response object from API: 'Request Entity Too Large\n\nFUNCTION_PAYLOAD_TOO_LARGE\n' (HTTP response code was 413)

The HTTP response code associated with this error is 413. It appears that the issue is related to the size of the prompt being sent to the openrouter API, as it is too large.

The prompt that is created by gpt-migrate is too large for openrouters api. Has anyone ran into this issue and fixed it?

`json.decoder.JSONDecodeError`

✅  Parsing function signatures for <file>...
Traceback (most recent call last):

  File "<string>", line 1, in <module>

  File ".../src/gpt_migrate/main.py", line 195, in main
    migrate(sourceentry, globals)

  File ".../src/gpt_migrate/main.py", line 186, in migrate
    migrate(dependency, globals, parent_file=sourcefile)

  File ".../src/gpt_migrate/main.py", line 187, in migrate
    file_name = write_migration(
                ^^^^^^^^^^^^^^^^

  File ".../src/gpt_migrate/steps/migrate.py", line 161, in write_migration
    sigs = get_function_signatures(deps_per_file, globals) if deps_per_file else []
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File ".../src/gpt_migrate/steps/migrate.py", line 62, in get_function_signatures
    sigs = json.loads(
           ^^^^^^^^^^^

  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/opt/homebrew/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Can this be used to convert a js project to typecript?

OPENAI_API_KEY for GPT-4

Hi - this looks like such a great project.

You say "It's also recommended that you use at least GPT-4, preferably GPT-4-32k.

Set your OpenAI API key and install the python requirements:

export OPENAI_API_KEY="

My key only permits GPT-3.5 at present, am on the waiting list for GPT-4 - I presume I won't be able to use your appliction in the meantime?

Can I do this for frameworks?

e.g. pytorch -> jax

OSError: [Errno 63] File name too long

vue.js 1 to 3

$ python main.py --sourcelang "vue.js 1"  --targetlang "vue.js 3" --sourcedir ../benchmarks/vue1-vue3 --sourceentry "index.html" --model "gpt-3.5-turbo-16k"
◐ Reading vue.js 1 project from directory '/Users/user/gpt-migrate/benchmarks/vue1-vue3', with entrypoint 'index.html'.
◑ Outputting vue.js 3 project to directory '/Users/user/gpt-migrate/benchmarks/flask-nodejs/target'.
Source directory structure:

        ├── favicon-16x16.png
        ├── safari-pinned-tab.svg
        ├── favicon.ico
        ├── index.html
        ├── android-chrome-192x192.png
        ├── apple-touch-icon.png
        ├── renovate.json
        ├── css/
            │   └── all.css
        ├── js/
            │   └── all.js
        ├── 404.html
        ├── README.md
        ├── img/
            │   ├── walnut-logo.svg
            │   ├── walnut-logo-white-background.png
            │   ├── spin.svg
            │   ├── share.svg
            │   └── walnut-logo-white-background view.svg
        ├── channels.js
        ├── android-chrome-512x512.png
        ├── site.webmanifest
        ├── package-lock.json
        ├── package.json
        ├── scripts/
            │   └── check-channels.js
        ├── mstile-150x150.png
        ├── browserconfig.xml
        └── favicon-32x32.png

✅  Creating your environment...
Created Docker environment for vue.js 3 project in directory '/Users/user/gpt-migrate/benchmarks/flask-nodejs/target'.
✅  Identifying external dependencies for index.html...
✅  Identifying internal dependencies for index.html...
✅  Creating migration file for index.html...
Created file_name.ext at /Users/user/gpt-migrate/benchmarks/flask-nodejs/target
Copied renovate.json from /Users/user/gpt-migrate/benchmarks/vue1-vue3 to /Users/gianpaj/tmp/gpt-migrate/benchmarks/flask-nodejs/target
✅  Creating dependencies file required for the Docker environment...
Traceback (most recent call last):

  File "/Users/user//gpt-migrate/gpt_migrate/main.py", line 100, in main
    add_env_files(globals)

  File "/Users/user/gpt-migrate/gpt_migrate/steps/migrate.py", line 91, in add_env_files
    external_deps_name, _, external_deps_content = llm_write_file(prompt,
                                                   ^^^^^^^^^^^^^^^^^^^^^^

  File "/Users/user/gpt-migrate/gpt_migrate/utils.py", line 61, in llm_write_file
    with open(os.path.join(globals.targetdir, file_name), 'w') as file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

OSError: [Errno 63] File name too long: "/Users/user/gpt-migrate/benchmarks/flask-nodejs/target/PREFERENCE LEVEL 1\n\nHere are the guidelines for this prompt:\n\n1. Follow the output instructions precisely and do not make any assumptions. Your output will not be read by a human; it will be directly input into a computer for literal processing. Adding anything else or deviating from the instructions w

Use my own OpenAI keys instead of Open Router

I have a paid account with Openai apis, which I would like to use. Due to token limit, I can not use open-router, and it doesn't make sense for me to make another purchase.

Is it possible to use openai apis directly instead of open-router ?

Local models

Is it going to support local ai models in the future?

Some languages not working for benchmark

C++, Rust need some work

Add milestones, roadmap or a TODO list in readme.md

As it stands right now, this is a great idea, but progress needs some parameters to allow more and better contributions:

A roadmap with the desired features, maybe classified by difficulty or urgency
Some kind of benchmark or success report for different projects and language conversion settings:
- This is more common on emulator repos, where the compatibility can be measured by testing the emulated software, but if you pick some popular repos or projects that have stable releases on source and target languages, you can compare what works and what doesn't.
Some decisions on how certain features will be added, for example: what kind of preprocessors for code should be allowed or ignored? should specific prompt hints be added based on certain functions or libraries to make the AI answer less prone to errors or hallucination?

These additions would funnel contributor efforts to specific areas and tasks, rather than anyone expecting for their own unique needs that broadly overlap with a more structured feature project.

Allow request for dependency function

The LLM should be allowed to "request" to see a dependency function it cannot see (in another file etc). This can apply to either source or target.

Decomposing functions/endpoints

Currently the source function ingests the entire file at once which will not work for larger API files. We'll need to go endpoint by endpoint and isolate dependency functions either in-file or in other files. With these, we should keep track of functions already writtend (file:function_name) in the LLM memory in order to avoid redundancy.

Confused about dependency handling

Why is both poetry and pip used? Shouldn't dependencies be handled by poetry, then after poetry install run with poetry run python main.py?

When I do try that, I get the error of no litellm, but when I try to add it, I get:

Because no versions of litellm match >1.7.12,<2.0.0
 and litellm (1.7.12) depends on openai (>=1.0.0), litellm (>=1.7.12,<2.0.0) requires openai (>=1.0.0).
So, because gpt-migrate depends on both openai (^0.27.8) and litellm (^1.7.12), version solving failed.

When I do use pip for installing the dependencies, I get the following error:

Traceback (most recent call last):
  File "/Users/koeng/py/src/github.com/joshpxyne/gpt-migrate/gpt_migrate/main.py", line 7, in <module>
    from ai import AI
  File "/Users/koeng/py/src/github.com/joshpxyne/gpt-migrate/gpt_migrate/ai.py", line 6, in <module>
    from litellm import completion
  File "/opt/homebrew/lib/python3.11/site-packages/litellm/__init__.py", line 28, in <module>
    from .timeout import timeout
  File "/opt/homebrew/lib/python3.11/site-packages/litellm/timeout.py", line 11, in <module>
    from openai.error import Timeout
ModuleNotFoundError: No module named 'openai.error'

Which I solved with pip install --upgrade litellm==1.0.0. This is a little suspicious since it is one of the only frozen dependencies... requirements.txt:

typer
langchain
yaspin
openai
tree_sitter
litellm==0.1.213
pydantic==1.10.8

So, what's up with mixing poetry and pip?