Disabled the test in 863d116

Error message:

Retrieved digest for link poetry_project-0.1.0-py3-none-any.whl(md5:6340bed3198ccf181970f82cf6220f78) in poetry.lock metadata ['sha256:a2916a4e6ccb4c2f43f0ee9fb7fb1331962b9ec061f967c642fcfb9dbda435f3', 80a47720d855408d426e835fc6088ed3aba2d0238611e16b483efe8e063d71ee']

This looks like an issue in combination with Pypiserver that we use in the integration test, maybe.

Even if you used pipx install kraken-wrapper with Pipx bound to an older Python versions; if you have a newer version of Python installed on your system we should consider using the latest one installed when creating the build environment.

This will avoid that people run into an unfortunately frequently occurring issue because a lot of Kraken's class member annotations need to be runtime-inspectable but we don't currently/can't yet lint for making sure that class members conform to this requirement, resulting in errors like type object is not subscriptable or unsupported operand type(s) for |: 'type' and 'NoneType.

Once we have this, we can also easily introduce a lower bound >=3.10 for kraken-core because kraken-wrapper will make sure it picks the highest Python version (and we could make it report an error if no Python version equal or higher to 3.10 is found).

When you cancel kraken-wrapper while it is creating the build environment, the env will be broken and you need to manually remove it. If at the end of the initialization we store a flag to say that everything's good, we can discover if that isn't set and re-initialize the env. Otherwise it results in cryptic errors (most commonly "ModuleNotFoundError: kraken.core")

Traceback (most recent call last):                                                                             
 File                                                                                                         
".../.kraken/venv/lib/python3.10/site-packages/kraken/core/cli/main.py", line 203, in _load_build_state                                                                                 
   context.load_project(Path.cwd())                                                                           
 File                                                                                                         
".../.kraken/venv/lib/python3.10/site-packages/kraken/core/system/context.py", line 1[70], in load_project                                                                               
   runner.execute_script(script, {"project": project})                                                        
 File                                                                                                         
".../.kraken/venv/lib/python3.10/site-packages/kraken/common/_runner.py", line 113, in execute_script                                                                                  
   exec(code, vars(module))                                                                                                                                                                                                           
[...]                        
 File                                                                                                         
".../.kraken/venv/lib/python3.10/site-packages/kraken/std/cargo/manifest.py", line 213, in read                                                                                        
   ret = cls.of(path, tomli.load(fp))                                                                         
 File                                                                                                         
".../.kraken/venv/lib/python3.10/site-packages/kraken/std/cargo/manifest.py", line 227, in of                                                                                          
   [Bin(**x) for x in data.get("bin", [])],                                                                   
 File                                                                                                         
".../.kraken/venv/lib/python3.10/site-packages/kraken/std/cargo/manifest.py", line 227, in <listcomp>                                                                                  
   [Bin(**x) for x in data.get("bin", [])],                                                                   
TypeError: Bin.__init__() got an unexpected keyword argument 'required-features'

Seems that using separate dependencies for a binary target via the second strategy here is not supported by kraken.

The task summary could include the version of the package that was published, so that you don't need to scroll up to find the infromation. Example:

  :python.publish.hs-snapshot-publish SUCCEEDED [0.422s] (version: 0.24.4)

Example

> :devops-cli:python.login PENDING
[12/19/23 22:20:38] INFO     $ ['pdm', 'config', 'pypi.hs-artifactory.url', 'MASKED']                                                                             pdm.py:152
                    INFO     $ ['pdm', 'config', 'pypi.hs-artifactory.username', 'MASKED']                                                                        pdm.py:152
                    INFO     $ ['pdm', 'config', 'pypi.hs-artifactory.password', 'MASKED']                                                                        pdm.py:152
Traceback (most recent call last):
  File "/home/coder/git/devops-automation/build/.kraken/venv/bin/keyring", line 8, in <module>
    sys.exit(main())
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/cli.py", line 148, in main
    return cli.run(argv)
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/cli.py", line 76, in run
    return method()
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/cli.py", line 93, in do_set
    set_password(self.service, self.username, password)
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/core.py", line 61, in set_password
    get_keyring().set_password(service_name, username, password)
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/backends/fail.py", line 28, in get_password
    raise NoKeyringError(msg)
keyring.errors.NoKeyringError: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.
                    INFO     $ ['pdm', 'config', 'pypi.hs-playground.url', 'MASKED']                                                                              pdm.py:152
[12/19/23 22:20:39] INFO     $ ['pdm', 'config', 'pypi.hs-playground.username', 'MASKED']                                                                         pdm.py:152
                    INFO     $ ['pdm', 'config', 'pypi.hs-playground.password', 'MASKED']                                                                         pdm.py:152
Traceback (most recent call last):
  File "/home/coder/git/devops-automation/build/.kraken/venv/bin/keyring", line 8, in <module>
    sys.exit(main())
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/cli.py", line 148, in main
    return cli.run(argv)
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/cli.py", line 76, in run
    return method()
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/cli.py", line 93, in do_set
    set_password(self.service, self.username, password)
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/core.py", line 61, in set_password
    get_keyring().set_password(service_name, username, password)
  File "/home/coder/git/devops-automation/build/.kraken/venv/lib/python3.10/site-packages/keyring/backends/fail.py", line 28, in get_password
    raise NoKeyringError(msg)
keyring.errors.NoKeyringError: No recommended backend was available. Install a recommended 3rd party backend package; or, install the keyrings.alt package if you want to use the non-recommended backends. See https://pypi.org/project/keyring for details.
> :devops-cli:python.login SUCCEEDED

Trunk is a tool used to build, bundle and ship Rust WebAssembly frontends. Since WebAssembly frontends need a little bit of glue code to talk to browser APIs, a tool such as Trunk is necessary to turn a WASM binary into something a browser and load and execute.

I believe that we should add support for trunk to kraken, such that it can build WebAssembly frontends without needing too much manual configuration. If possible, we might even be able to enable autoconf to detect it by the presence of a Trunk.toml file.

Such as /bin/python3.8-config, it's definitely not going to be a Python interpreter.

Example

[12/20/23 17:15:02] ERROR    'Install dependencies' failed (exit code: 1, command: $                                    _buildenv_venv.py:84
                             /builds/group/project/build/.kraken/venv/bin/python -m pip install                                 
                             --disable-pip-version-check --no-python-version-warning --no-input --index-url                                 
                             https://[MASKED]:REDACTED
                             READACTED.REDACTED

Cargo, Docker login, Kaniko build and maybe other tasks pass credentials via the CLI, and in verbose mode those commands are logged thus exposing credentials.

See python 3.7 integrationTest here.

https://niklasrosenstein.jfrog.io/artifactory/git/kraken-std-cargo-integration-test.git redirects to:

So it seems like the artifactory that's used for integration testing has been deactivated. This requires those integration tests to be marked as xfail.

That would be used to validate if the source code actually reflects the tagged version.

People that are not very familiar with Kraken that write a .kraken.py for a sub-project will often copy the buildscript() block into that script, but it's only required at the top-level. We should just raise an exception when it is used in any project that is not the root project, as it doesn't add any value there.

This issue occurs if at any point in the serialized object graph there is a module that was loaded with the _frozen_importlib_external._NamespaceLoader. Example:

kraken/hs/storyboard/
  builder/
    api.py
  __init__.py

# kraken/hs/storyboard/__init__.py

from .builder.api import DocumentationBuilder import api

# Somewhere else, producing a lambda that needs to be serialized:
def storyboard_publish():
    # ...
    filename = archive.map(lambda archive: f"{env.canonical_project_id}-{env.git_version}{archive.suffix}")
    # ...

The lambda has a reference to the global scope and causes Dill to try and serialize the kraken.hs.storyboard.builder module which is a namespace module where the __file__ attribute is None, causing a stacktrace like this:

[ ... ]
  File "/home/coder/git/kraken/kraken-hs/.venvs/3.10/lib/python3.10/site-packages/dill/_dill.py", line 1963, in save_function
    _save_with_postproc(pickler, (_create_function, (
  File "/home/coder/git/kraken/kraken-hs/.venvs/3.10/lib/python3.10/site-packages/dill/_dill.py", line 1154, in _save_with_postproc
    pickler._batch_setitems(iter(source.items()))
  File "/home/coder/.pyenv/versions/3.10.9/lib/python3.10/pickle.py", line 998, in _batch_setitems
    save(v)
  File "/home/coder/.pyenv/versions/3.10.9/lib/python3.10/pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/coder/git/kraken/kraken-hs/.venvs/3.10/lib/python3.10/site-packages/dill/_dill.py", line 1758, in save_module
    builtin_mod = _is_builtin_module(obj)
  File "/home/coder/git/kraken/kraken-hs/.venvs/3.10/lib/python3.10/site-packages/dill/_dill.py", line 1733, in _is_builtin_module
    return any(os.path.realpath(module.__file__).startswith(os.path.realpath(getattr(sys, name)))
  File "/home/coder/git/kraken/kraken-hs/.venvs/3.10/lib/python3.10/site-packages/dill/_dill.py", line 1733, in <genexpr>
    return any(os.path.realpath(module.__file__).startswith(os.path.realpath(getattr(sys, name)))
  File "/home/coder/.pyenv/versions/3.10.9/lib/python3.10/posixpath.py", line 394, in realpath
    filename = os.fspath(filename)
TypeError: expected str, bytes or os.PathLike object, not NoneType

Where the save_function() stack is pointing to the lambda function, and the _is_builtin_module() function is pointing to the namespace module.

Kraken is lacking a lock command that re-creates the environment specified in the lockfile

This would make this workflow possible:

• clone a kraken-managed repo
• kraken lock # or any more suitable command name
• tada, a virtual env is setup, with everything specified in the lockfile, so that I can do a kraken run build that will work, even if the newest upstream releases of the dependencies have changed

This is because we only match on digit after the dot.

https://github.com/kraken-build/kraken-build/blob/53bb65e5f0a32a89fc39e61ebc234ceb9357dacf/kraken-common/src/kraken/common/findpython.py#L133

This would allow us to rename a task but keep references to the task's old name in other places still functional (and maybe we can warn users when they're referencing to a task by its alias).

Example:

from kraken.common import buildscript

buildscript(requirements=["dont-exist@."])

https://devguide.python.org/versions/

When creating a Task subclass in a Kraken build script with typed properties, it fails to look up the scope of the build script in sys.modules as it is not a registered Python module.

The workaround atm. is to define the task in a Python module in build-support/ in the project root directory (natively supported by Kraken to load additional modules from).

raise NetworkXError(f"The node {n}             
                             is not in the digraph.") from err                  
                             networkx.exception.NetworkXError: The              
                             node :gitlabci is not in the digraph.

We should not trigger any errors when the desginated task exists, but warn if the task does not exist or would not be run anyway.

I can't quite explain why. The package is uploaded to Cloudsmith just fine.

Failing build: https://github.com/kraken-build/kraken-std/runs/8295784884?check_suite_focus=true

Crate created during this build but failing to be picked up:

https://cloudsmith.io/~niklas-rosenstein-OJj/repos/kraken-std-cargo-integration-test/packages/detail/cargo/hello-world-lib/455.955.370/

It could be something going wrong on our side, or Cloudsmith needs more time to be capable of serving the package in its Cargo index.

Sidecar containers are often required for testing, such as having a Postgres database available while running integration tests. These containers should be configurable in a Kraken build script, and it should be possible to start these containers directly and also have them start for tests.

In the build script, an example container definition may be:

from kraken.std.devenv import sidecar_container

sidecar_container(
  name="postgres",
  image="postgres:latest",
  ports=["5432:5432"],
  env={"POSTGRES_PASSWORD": "alpine"},
)

This should produce two targets, :postgres.start and :postgres.run. The :postgres.start task starts up the container in the background. The :postgres.run command is intended to be used on the CLI to wait for the container to complete or to cancel it when the task is interrupted.

For tests, the :postgres.start command would simply need to be a dependency on the task that runs the tests. It being a background task, the Postgers container will stay alive until the tests have completed.

Running krakenw to install local dependency (such as somepkg @ ../somepkg) fails if the command is not run from the project root. It will attempt to resolve the relative path from the current working directory, instead of from the project root directory.

Exampl: kraken-base-image

Even though kraken query ls shows the tasks as succeeded; the subsequent kraken run --resume re-runs the Docker build tasks.

https://github.com/kraken-build/kraken-base-image/actions/runs/3593226139/jobs/6050255972

Sometimes you may end up with this diff after running krakenw apply because you manually added an entry to .gitignore and your editor stripped trailing whitespace on lines in the file.

tested on linux

cd myproject
krakenw run :cargoClippy
cd ..
mv myproject myproject2
cd myproject2
krakenw run :cargoClippy

the second kraken command will fail, with a mitmproxy error. In our case, it also deleted all the Cargo.tomls, so then cargo explodes due to not being able to compute metadata.

https://setuptools.pypa.io/en/latest/pkg_resources.html

We depend on pytest optionally, but when I install pip install kraken-core it also installs pytest which should not be the case. We only need it as an optional dependency to make sure that if Pytest is available we depend on a minimum version of it and we can use it in kraken.core.testing.

Once pex-tool/pex#1805 ships, passing --pip-version 22.2.2 will have Pip use in-tree builds (as it's the default since 21.3 anyway). This should speed up building a PEX for a Kraken project that has a local requirement.

Currently, local dependencies (such as somepkg @ ../somepkg) can only be installed immediately. This makes development of a Kraken extension difficult in some situations. After modifying files in the source package, a reinstall needs to be triggered with krakenw --upgrade --incremental.

It would be great if we could install the local dependency in editable mode instead (e.g. Pip's -e option).

As a side note: We should add more emphasis on the warning after install (e.g. with --upgrade --incremental) if a Kraken lock file is present but it is outdated. Otherwise, the next krakenw run will install again whatever is in the lock file, and not what the buildscript() definition specifies.

The kraken.hs.cargo.manifest module provides two datasources:

• The plain Cargo.toml file parsed into data structures (CargoManifest, Workspace, Package, Dependencies, Workspace, WorkspacePackage)
• The output of cargo metadata (CargoMetadata, WorkspaceMember, Artifact, Bin and Lib)

It presents multiple shotcomings imho:

• There is no way from a class name to know if it's from the plain Cargo.toml or from cargo metadata.
• The CargoMetadata datastructure looses information, for example the relation between WorkspaceMember and Artifact.

Proposal:

1. Create a new kraken.hs.cargo.metadata module presenting all data from cargo metadata version 1 and migrate all read-only usages from it.
2. Drop cargo metadata datastructures from kraken.hs.cargo.manifest
3. Simplify Cargo.toml representation (for example walking between the different crates of the same workspace might be done using cargo metadata)

Using Pip to install dependencies into an environment with an interpreter constraint is proving to be difficult. Using the --python-version argument to pip install has some restrictions:

• Need to use --no-deps and/or --target options
• Need to specify --only-binary=:all: which doesn't work for a package like termcolor which doesn't come with a wheel distribution
• The --python-version option expects an exact Python version rather than a version constraint (like >=3.7)

Pex seems to be somehow working around this as it is using pip download internally.

FLUP for #71

If Sccache is misonfigured, it will exit with an error and fail the entire build. We should probably allow it to fail, even if it means Rust builds are less than optimal.

Currently, adding a changelog entry is done by appending a TOML section to kraken-build/.changelog/_unreleased.toml. However, this directly leads to merge conflict when MRs are opened concurrently.

This can be partially mitigated by inserting the new changelog entry at a random point in the file, instead of appending to the end. However, this is only possible when the file already contains multiple entries, and the probability of conflict is still high.

A more robust solution would be to have a _unreleased/ directory instead, and creating a new file for each entry. The file name could be either be chosen arbitrarily by the contributor (with a non-negligible chance of collision), or use the UUID from the id field. The files could then be optionally sorted, and merged together for a new release.

Let the user know if a newer version of kraken-wrapper is available.

Can be worked around with by excluding it explicitly.

To reproduce:

1. Define a task whose description is long enough to wrap in your terminal.
2. Make your terminal fairly narrow.
3. Run kraken q ls.
4. Notice that the wrapped lines of the description (that is, every line except the first for each task) is wrapped very narrowly.

This happens because we use the subsequent_ident argument to textwrap.wrap:

https://github.com/kraken-build/kraken-build/blob/fcb55055a56ec703729315b462820cfd363c26bb/kraken-core/src/kraken/core/cli/main.py#L324-L328

And subsequent_indent counts towards the line length of each line except the first. In other words, if the terminal is 120 wide and remaining_width = 60, we'll end up passing 120-60 space characters to subsequent_lines, and those will be counted against the remaining_width we pass as the width argument. Which effectively means there is zero remaining space, and so textwrap dutifully wraps the remaining lines as aggressively as it can.

The solution here is probably to not pass subsequent_indent, and instead just prepend the spaces onto each line (except the first) that textwrap.wrap returns.

Technically, having a lock file should allow us to install an environment without attempting dependency resolution. Unfortunately, that's not quite the reality right now because when you lock an environment on OSX it does not contain optional Linux dependencies.

An example is keyring which has the requirements

• SecretStorage (>=3.2) ; sys_platform == \"linux\"
• jeepney (>=0.4.2) ; sys_platform == \"linux\"

Producing a lockfile on OSX does not write these requirements into the lockfile.

PEX does appear to have a mechanism for writing lock files, but I couldn't yet figure out exactly how to use it so we're still doing the retrospective inspection of the environment's installed distributions.

Maybe we could also leverage Poetry's resolver in some way.

When selecting tasks to run on the command-line, specifying a bare task name matches all tasks with that name in the focus project and all the sub projects. As an example, assuming the following projects and tasks:

: (root project)
  :a (sub project) (focus)
    :a:b (sub project)
      :a:b:task (task) 
    :a:task (task)
  :c (sub project)
    :c:task (task)
  :task (task)

Running krakenw query task with :a as the focus project will select the tasks :a:b:task and :a:task. This is done by transforming the selector from task into .:**:task.

However, krakenw query tas* will only match the task :a:task. I would suggest to also transform this selector from tas* into .:**:tas* automatically as I believe that behavior would be less surprising.

If we do that, the old semantics for matching tas* but only in the current project requires that one explicitly specifies the current project, e.g. .:tas*.

It works perfectly fine locally, I have no clue at the moment as to why it would fail in CI.

This will prevent string formatting from executing if the message is not needed after all.