Docker build is broken about gpt-engineer HOT 30 CLOSED

The issue is with cmake missing in the container, not the host system.

Seems the Dockerfile needs:

RUN apk add --update build-base or similar.

from gpt-engineer.

Perfect! We are now looking into testing this on Linux, Mac, and Windows and we'll see how to fix it for everyone. I'll leave this issue open for now just as a reminder to @zigabrencic and me to follow-up on this for other platforms.

Glad it worked for you @t6tv8e! Thanks, @zigabrencic and @ErikBjare for helping out!

from gpt-engineer.

Okay, thanks for your time already in any case

from gpt-engineer.

@zigabrencic @viborc

That worked 👌

from gpt-engineer.

Here are the files. In the pyproject.toml I only added langchain-community ="0.2.0" and in Dockerfile - COPY --from=builder /usr/bin /usr/bin

```bash
# Stage 1: Builder stage
FROM python:3.11-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
  tk \
  tcl \
  curl \
  git \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /usr/bin /usr/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

[tool.poetry]
name = "gpt-engineer"
version = "0.3.0"
description = "Specify what you want it to build, the AI asks for clarification, and then builds it."
authors = ["Anton Osika <[email protected]>"]
license = "MIT"
readme = "README.md"
homepage = "https://github.com/gpt-engineer-org/gpt-engineer"
repository = "https://github.com/gpt-engineer-org/gpt-engineer"
documentation = "https://gpt-engineer.readthedocs.io/en/latest/"
classifiers = [
  "Development Status :: 4 - Beta",
  "Topic :: Scientific/Engineering :: Artificial Intelligence",
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = ">=3.10,<3.13"
openai = "^1.0"
termcolor = "2.3.0"
typer = ">=0.3.2"
rudder-sdk-python = ">=2.0.2"
dataclasses-json = "0.5.7"
tiktoken = ">=0.0.4"
tabulate = "0.9.0"
python-dotenv = ">=0.21.0"
langchain = ">=0.1.2"
langchain_openai = "*"
toml = ">=0.10.2"
tomlkit = "^0.12.4"
pyperclip = "^1.8.2"
langchain-anthropic = "^0.1.1"
regex = "^2023.12.25"
pillow = "^10.2.0"
datasets = "^2.17.1"
black = "23.3.0"
langchain-community ="0.2.0"

[tool.poetry.group.dev.dependencies]
pytest = ">=7.3.1"
pytest-cov = "^4.1.0"
mypy = "1.3.0"
ruff = ">=0.0.272"
pre-commit = "3.3.3"
tox = ">=3.0.0"

[tool.poetry.group.docs.dependencies]
autodoc_pydantic = ">=1.8.0"
myst_parser = ">=0.18.1"
nbsphinx = ">=0.8.9"
sphinx = ">=5.0.0"
sphinx-autobuild = ">=2021.3.14"
sphinx_book_theme = ">=0.3.3"
sphinx_rtd_theme = ">=1.0.0"
sphinx-typlog-theme = ">=0.8.0"
toml = ">=0.10.2"
myst-nb = ">=0.17.1"
linkchecker = ">=10.2.1"
sphinx-copybutton = ">=0.5.1"
markdown-include = ">=0.6.0"
sphinx_copybutton = ">=0.5.2"

[tool.poetry.scripts]
gpt-engineer = 'gpt_engineer.applications.cli.main:app'
ge = 'gpt_engineer.applications.cli.main:app'
gpte = 'gpt_engineer.applications.cli.main:app'
bench = 'gpt_engineer.benchmark.__main__:app'
gpte_test_application = 'tests.caching_main:app'

[tool.poetry.extras]
test = ["pytest", "pytest-cov"]
doc = ["autodoc_pydantic", "myst_parser", "nbsphinx", "sphinx", "sphinx-autobuild", "sphinx_book_theme", "sphinx_rtd_theme", "sphinx-typlog-theme", "myst-nb", "linkchecker", "sphinx-copybutton", "markdown-include", "sphinx_copybutton"]

[tool.ruff]
select = ["F", "E", "W", "I001"]
show-fixes = false
target-version = "py310"
task-tags = ["TODO", "FIXME"]
extend-ignore = ["E501", "E722"]

[tool.black]
target-version = ["py310"]

[tool.ruff.isort]
known-first-party = []
known-third-party = []
section-order = [
    "future",
    "standard-library",
    "third-party",
    "first-party",
    "local-folder",
]
combine-as-imports = true
split-on-trailing-comma = false
lines-between-types = 1

[tool.pytest.ini_options]
markers = [
    "requires_key: marks tests as requiring access to a valid OPENAI_API_KEY (deselect with '-m \"not requires_key\"')",
]

from gpt-engineer.

command 'cmake' failed: No such file or directory

from gpt-engineer.

Hi @t6tv8e - I think the issue is related to the fact that cmake utility is not installed or not available in the system's PATH where you are trying to install the pyarrow package. You might want to check their docs.

Or, depending on your OS, install it along these lines (this should work for Ubuntu or Debian-based stuff:

sudo apt-get update
sudo apt-get install cmake

from gpt-engineer.

I'm on macos and I have cmake on my machine

from gpt-engineer.

@t6tv8e, can you try running this command in your Terminal and share the output/screenshot here:

cmake --version

from gpt-engineer.

from gpt-engineer.

Let's check if it's configured properly.

mkdir ~/cmake-test
cd ~/cmake-test
echo "cmake_minimum_required(VERSION 3.10)" > CMakeLists.txt
echo "project(HelloCMake)" >> CMakeLists.txt
cmake .

This should check if it is functioning correctly by creating a temporary project directory and attempting to configure a simple project.

If that fails, you might want to consider reinstalling it using homebrew or whatever you usually use like this:

brew install cmake

from gpt-engineer.

from gpt-engineer.

from gpt-engineer.

Hm... what's the output of your PATH?

echo $PATH

from gpt-engineer.

That is:
/Users/t6t/.pyenv/shims:/Users/t6t/.nvm/versions/node/v20.12.1/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/share/dotnet:~/.dotnet/tools:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Users/t6t/.dotnet/tools

from gpt-engineer.

Hm... can you do this please:

which cmake

from gpt-engineer.

/opt/homebrew/bin/cmake

from gpt-engineer.

Okay, I'll need to check this later today with the team. I'm unsure what the issue might be. Maybe you can try reinstalling the cmake and see if that makes any difference.

from gpt-engineer.

Of course, we are always trying to help!

from gpt-engineer.

Yeah, I've been tinkering with the Dockerfile.

FROM python:3.10-alpine AS builder

RUN apk update && apk add --no-cache \
    tk \
    tcl \
    curl \
    cmake \
    gcc \
    g++ \
    python3-dev \
    musl-dev \
    make \
    libc-dev \
    linux-headers \
    build-base

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.10-alpine

RUN apk update && apk add --no-cache \
    tk \
    tcl \
    curl \
    cmake \
    gcc \
    g++ \
    python3-dev \
    musl-dev \
    make \
    libc-dev \
    linux-headers \ 
    build-base

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["sh", "/app/entrypoint.sh"]

from gpt-engineer.

Didn't work

from gpt-engineer.

Identical error?

from gpt-engineer.

Yes it's the same error...

from gpt-engineer.

Hey @t6tv8e, we just had our technical planning meeting and discussed this. One of our colleagues tried adding cmake to a Docker image, but that didn't solve the problem. We have a couple of other options we'll try out, but we'll keep you updated!

@ErikBjare, thanks for helping with this; I'm super thankful for checking on this, especially while you are away! 🙏

from gpt-engineer.

Hey.

So as I said in the meeting I tried to build the container. Adding cmake at install doesn't fix the issue.

I think the no-cache part in this commit broke this.

I ran the previous version of docker file which builds properly.

@t6tv8e can you try this as a docker file instead:

FROM python:3.11-slim

RUN apt-get update
RUN apt-get install -y sudo tk tcl gcc curl

WORKDIR /app

COPY . .
COPY docker/entrypoint.sh ./entrypoint.sh

RUN sudo pip install -e .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

And let us know if this fixes the issue.

If it does @viborc I propose we revert to this. Sure the docker image will be a bit bigger, but since we are dealing with LLM's few MB's is nothing 😅

from gpt-engineer.

That sounds good. Let's see what @t6tv8e says, and we can then figure out the next steps. Thanks for a quick turnaround, @zigabrencic! 💪

from gpt-engineer.

Hey.

So as I said in the meeting I tried to build the container. Adding cmake at install doesn't fix the issue.

I think the no-cache part in this commit broke this.

I ran the previous version of docker file which builds properly.

@t6tv8e can you try this as a docker file instead:

FROM python:3.11-slim

RUN apt-get update
RUN apt-get install -y sudo tk tcl gcc curl

WORKDIR /app

COPY . .
COPY docker/entrypoint.sh ./entrypoint.sh

RUN sudo pip install -e .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

And let us know if this fixes the issue.

If it does @viborc I propose we revert to this. Sure the docker image will be a bit bigger, but since we are dealing with LLM's few MB's is nothing 😅

@zigabrencic The no-cache flag is not the source of the issue. While pyarrow is supported on Alpine, it lacks a wheel. In other words, you need to build it from source.

I notice that the Dockerfile in #1144 will be reverted; use this one instead:

# Stage 1: Builder stage
FROM python:3.11-slim AS builder

RUN apt-get update && apt-get install -y --no-install-recommends \
  tk \
  tcl \
  curl \
  git \
  && rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY . .

RUN pip install --no-cache-dir -e .

# Stage 2: Final stage
FROM python:3.11-slim

WORKDIR /app

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
COPY --from=builder /app .

COPY docker/entrypoint.sh .

ENTRYPOINT ["bash", "/app/entrypoint.sh"]

The image size would still be reduced

And on the other hand, Docker containers are designed to run with an unprivileged user by default, and running commands with sudo, for reference RUN sudo pip install -e . , or as the root user can lead to potential security issues and unexpected behavior.

from gpt-engineer.

Hey

Thank you for your suggestions.

Yes you are correct @k1lgor the cmake dependency broke this. I misstated this before.

Tried your proposal and:

• 1.) Image on Mac OS is still 1.43 GB's of size.
• 2.) Fails with: FileNotFoundError: [Errno 2] No such file or directory: 'git' once gpte engineer is executed.` Despite the fact that git is installed.

As a result I propose we keep the simpler version as implemented here.

The cross platform size benefit in my opinion isn't large enough to add the proposed complexity.

I made some updates to #1144:

• change the RUN sudo pip part to RUN pip => per @k1lgor proposal.
• python:3-11-slim to python:3-12-slim
• Added the line: pip install -U langchain-community => We might have to add this one elsewhere.

If there are no objections I'll open the PR again.

from gpt-engineer.

Hey @zigabrencic ,

I believe I may be able to locate the issue. tk tcl curl git are installed in /usr/bin/ since they are dpkg packages. It is necessary to add one COPY.

COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin
+ COPY --from=builder /usr/bin /usr/bin
COPY --from=builder /app .

And in addition, I have added langchain-community = "0.2.0" into pyproject.toml.
Could you test it on your Mac?

from gpt-engineer.

Hey @k1lgor

Sure I can test it just please let me know which branch?

Or if you can provide the full docker file so I test the right thing :)

from gpt-engineer.