flyingcircusio / appenv Goto Github PK

• use (and require) requirements.lock for installing reliable
• implement --appenv-update-requirements: create temporary install from requirements.txt and generate lockfile from there

So, here's a weird one: a) The default values for python -m venv and import venv; venv.create() are not the same. The former uses symlinks=True, but the latter, which is used by batou/appenv, has symlinks=False). b) At least with my brew install python, the variant with symlinks=False has recently(?) stopped working:

$ python3 -m venv venv; venv/bin/python --version; rm -rf venv
Python 3.11.5
$ python3 -c 'import venv; venv.create("venv")'; venv/bin/python --version; rm -rf venv
dyld: Library not loaded: @loader_path/../../../../Python.framework/Versions/3.11/Python
  Referenced from: /Users/schnerring/Downloads/venv/bin/python
  Reason: image not found
fish: Job 1, 'venv/bin/python --version' terminated by signal SIGABRT (Abort)
$ python3 -c 'import venv; venv.create("venv", symlinks=True)'; venv/bin/python --version; rm -rf venv
Python 3.11.5

I don't understand neither (a) nor (b), and the closest related bug I could find was python/cpython#82886 (but that talks about Apple's Python, not Homebrew), so my proposal would be this:

diff --git a/batou b/batou
index e509e153..691d9cca 100755
--- a/batou
+++ b/batou
@@ -69,7 +69,7 @@ def ensure_venv(target):
     version = sys.version.split()[0]
     python_maj_min = ".".join(str(x) for x in sys.version_info[:2])
     print("Creating venv ...")
-    venv.create(target, with_pip=False)
+    venv.create(target, with_pip=False, symlinks=True)

Hi,

with the newest version of AppEnv, I get a pkg_resources.RequirementParseError if the requirement.txt contains a custom index server:

# appenv-python-preference: 3.8,3.7
--extra-index-url https://my.custompypi.de/simple
-e .

When running ./appenv update-lockfile I get:

Traceback (most recent call last):
  File "./appenv", line 515, in <module>
    main()
  File "./appenv", line 507, in main
    appenv.meta()
  File "./appenv", line 280, in meta
    args.func(args, remaining)
  File "./appenv", line 466, in update_lockfile
    spec = list(pkg_resources.parse_requirements(line))[0]
  File "/Users/sweh/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3094, in parse_requirements
    yield Requirement(line)
  File "/Users/sweh/.pyenv/versions/3.7.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 3103, in __init__
    raise RequirementParseError(str(e))
pkg_resources.RequirementParseError: Invalid requirement, parse error at "'--extra-'"

Is there another way to specify a custom index server?

@wosc if you could dig up the issues you had w/ Debian then I'd be happy to investigate a potential generic way of getting environment-specific workarounds available.

Ok. This is going to be painful (for others to watch), but:

If we can't fix the system Python - and I don't want to compile my own - then here's what I'm going to try:

• download the original cPython source packages from github, version specific to whatever python version we are running, extract it in /tmp
• copy lib/venv, lib/... to ~/.appenv/py/lib
• include that in the pythonpath

Hopefully any venvs we create will properly inherit this ... I'll try this out.

The tests of the package should be run on GHA and a separate check running the linter would be nice as the appenv.py is checked in into customer repositories it should not violate flake8 rules.

F-strings are faster and more readable. They got introduced in Python 3.6.

Currently, run() directly exec's the setuptools-generated entrypoint script, which in turn finds the correct python by including the full path to the {env_dir}/bin/python executable as a shebang line.

Shebang lines are limited to 127 characters, which is very easy to exceed with appenv, since the venv hash already takes up 64.

I propose to specify the python executable in appenv itself, instead of relying on the shebang line, like this:

diff --git a/batou b/batou
index 8f03df7..455def7 100755
--- a/batou
+++ b/batou
@@ -200,7 +200,10 @@ def run(argv, meta_args):
     env_dir = _prepare(meta_args)
     # Allow called programs to find out where the wrapper lives
     os.environ['APPENV_BASEDIR'] = meta_args.base
-    os.execv(os.path.join(env_dir, 'bin', meta_args.appname), argv)
+    interpreter = os.path.join(env_dir, 'bin', 'python')
+    entrypoint = os.path.join(env_dir, 'bin', meta_args.appname)
+    argv[0] = interpreter
+    argv.insert(1, entrypoint)
+    os.execv(interpreter, argv)
 

This works fine with batou, but argv[0] is not preserved (since at least under Debian we must set argv[0] to the venv/bin/python, otherwise the venv is not recognized and the spawned python runs with the system site-packages), so programs that are interested in that will have trouble. Not sure if that's something we have to worry about, though.

Currently updates are done by hand, i.e. using curl to get a new version.

Further it would be nice to have at least a notification about a new version when running update-lockfile.
(Maybe update appenv here as well?)

• --appenv-clone to create a new project from an old with the same name, requirements, lock etc.
• entry point for the compatible applications (bin/{app}-init or through entry point lookup) to create "cookie cutter" copies of an environment

• python upgrade
• broken pip
• broken interpreter

• maybe an application package should maintain its own complete fixed version set within the egg that can be used to bootstrap and update installations consistently?

So,

theoretically we could speed things up even further if we include more data in the hash and then could just copy over build results from other locations/machines similar to nix. Obviously this isn't going to be perfect, but maybe good enough to be helpful.

We'd need at least:

• operating system (uname?)
• target path (venvs are not portable)
• python version
• version of appenv (hash of the appenv file?)
• ... others? ...

Pip already automatically checks for hashes, but we don't generate them into the lockfile. Need to check whether we can do that

I was wondering if one could replace zc.buildout environments with appenv, particularly if one's using zc.buildout only for package installation and version pinning.

This depends how we perform releases. We could leverage our venv management and create a temporary install that extracts the appenv script from a distribution.

Looking at the confusion between batou and appenv I suggest we implement a new scheme where appenv becomes its own script and the actual (managed) utilities are symlinks (so that appenv determines which script to run by looking at the argv[0]) by which it was called. This removes the need for a "meta" argument parser:

ctheune@fourteen-3 /tmp/asdf $ ls -lah
total 0
drwxr-xr-x    4 ctheune  wheel   128B Feb 23 06:54 .
drwxrwxrwt  494 root     wheel    15K Feb 23 06:54 ..
-rw-r--r--    1 ctheune  wheel     0B Feb 23 06:54 appenv
lrwxr-xr-x    1 ctheune  wheel     6B Feb 23 06:54 batou -> appenv

This also means we can easily add more symlinks, think harder about updating appenv itself and maybe providing it as a real, versioned package instead of just a curl'ed file.

I get this error but also have python, als well as python3 installed. With which python I get the path /usr/local/bin/python in the appenv script current_python variable has the value /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/bin/python3.9 which is also a working python.

Even though I cannot run the script and get the error Could not find the minimal preferred Python version.

In downstream packages that use appenv (like... batou :) it would be helpful to know which appenv version they bundled. For example, in the version bundled with batou-2.0b9, the appenv-update-lockfile command does not work (it exits with "unrecognized arguments: -u").

I've solved my immediate issue by downloading https://github.com/flyingcircusio/appenv/blob/dc208ac9df334962b013001c60cd93fb88949c98/src/appenv.py and naming it batou, but I think this points to an issue how to manage bundling and versions and somesuch.

I accidentally rsynced an .appenv directory from Mac to Linux and appenv didn't automatically recover from that:

subprocess.CalledProcessError: Command '['./.appenv/unclean/bin/python -m pip install -r requirements.txt --upgrade']' returned non-zero exit status 126.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./batou", line 444, in <module>
    main()
  File "./batou", line 438, in main
    appenv.run(application_name, sys.argv[1:])
  File "./batou", line 234, in run
    self._prepare()
  File "./batou", line 253, in _prepare
    cmd('{env_dir}/bin/python -m pip install -r requirements.txt --upgrade'
  File "./batou", line 43, in cmd
    raise ValueError(e.output.decode("ascii"))
ValueError: /nix/store/9ywr69qi622lrmx5nn88gk8jpmihy0dz-bash-4.4-p23/bin/sh: ./.appenv/unclean/bin/python: cannot execute binary file: Exec format error

• Old revisions
• Old batou venvs that were placed directly into the environment directory

By default we choose /usr/bin/env python3 and suggest users specify a custom specific version. If you'd like to use "whatever is better than X" you could say "3.5 is fine, but use 3.6 if available" so it works even if 3.5 is not available...