tgolsson / pants-backends Goto Github PK

7.0 2.0 0.0 491 KB

My plugins for the Pants build system

License: MIT License

Starlark 2.82% Shell 2.31% Python 94.81% Emacs Lisp 0.06%

pants kubernetes mdbook pants-plugins pantsbuild

pants-backends's Introduction

Tom's Pants backends

This is the documentation for my Pants backend, which I use for my own projects - especially managing my own cloud, but also building open-source software.

Please see the documentation for more information!

pants-backends's People

Contributors

Stargazers

Watchers

pants-backends's Issues

Optimize pex build workflows

https://blog.pantsbuild.org/optimizing-python-docker-deploys-using-pants/

Add support for build-args w/ defaults.

Make badges clickable in sub-readmes.

Import docker images from Pants `docker` backend

With the new podman support, this might actually make sense as a workflow.

Automatically derive `uid` and `gid` mappings

Would've prevented the regression in #56 where GHA changed their GID.

Failing to build layer using `resources` leads to Index Out of Range

Describe the bug
Stacktrace:

Traceback (most recent call last):
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/core/goals/repl.py", line 127, in run_repl
    request = await Get(ReplRequest, ReplImplementation, repl_impl)
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "<pants-cache-dir>/named_caches/pex_root/venvs/s/32db4f38/venv/lib/python3.9/site-packages/pants_backend_oci/goals/run.py", line 213, in run_oci_command_repl
    process = await Get(Process, RunImageBundleProcessRequest(request.targets[0], interactive=True))
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 118, in __await__
    result = yield self
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "<pants-cache-dir>/named_caches/pex_root/venvs/s/32db4f38/venv/lib/python3.9/site-packages/pants_backend_oci/goals/run.py", line 93, in prepare_run_image_bundle
    tool, image, rundir, shims = await MultiGet(
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 479, in MultiGet
    return await _MultiGet((__arg0, __arg1, __arg2, __arg3))
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "<pants-cache-dir>/named_caches/pex_root/venvs/s/32db4f38/venv/lib/python3.9/site-packages/pants_backend_oci/language_target/python.py", line 119, in build_python_image
    base_image, umoci, *layers = await MultiGet(
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 464, in MultiGet
    return await _MultiGet((__arg0, __arg1, __arg2, __arg3, __arg4))
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 165, in __await__
    result = yield self.gets
  File "<cache-dir>/bindings/venvs/2.16.0/lib/python3.9/site-packages/pants/engine/internals/selectors.py", line 623, in native_engine_generator_send
    res = rule.send(arg) if err is None else rule.throw(throw or err)
  File "<pants-cache-dir>/named_caches/pex_root/venvs/s/32db4f38/venv/lib/python3.9/site-packages/pants_backend_oci/util_rules/layer.py", line 145, in build_image_layer
    raw_layer_digest, layer_name = real_layers[0]
IndexError: list index out of range

To Reproduce

Include a resources-target in a Python image.

Publishing when no repository set

Describe the bug
When running the publish goal on oci_image_build it will try to publish images even if the repository argument is not set.

To Reproduce

oci_image_build(
    name="image",
    layers=[":layer"],
    base=["//:python-distroless-arm"],
    # repository=repository,
    # tag=image_tags,
    entrypoint="python3",
    args=[f"{dirctory_no_slash}/local.pex"],
    env=[
        f"PEX_PATH=/{dirctory_no_slash}/local.pex",
    ],
)

pants publish ::

Expected behavior
Ignore target when repository not set

Setup (please complete the following information):

https://github.com/tgolsson/pants-backends@064c00b67311675971449b51962ce62e766d500d#egg=pants_backend_oci&subdirectory=pants-plugins/oci
Pants version [e.g. 2.19.0]
umoci nightly mac arm

Fix deprecation warnings

In Pants v2.15 the following warnings appear:

17:40:13.76 [WARN] DEPRECATED: using `SubsystemRule(MdBookTool)` is scheduled to be removed in version 2.17.0dev0.

Use `*MdBookTool.rules()` instead.
17:40:13.80 [WARN] DEPRECATED: `pants.engine.environment.Environment` is scheduled to be removed in version 2.17.0.dev0.

Use `pants.engine.env_vars.EnvironmentVars`.
17:40:13.80 [WARN] DEPRECATED: `pants.engine.environment.EnvironmentRequest` is scheduled to be removed in version 2.17.0.dev0.

Use `pants.engine.env_vars.EnvironmentVarsRequest`.
17:40:13.82 [WARN] DEPRECATED: using `SubsystemRule(SkopeoTool)` is scheduled to be removed in version 2.17.0dev0.

Use `*SkopeoTool.rules()` instead.
17:40:13.82 [WARN] DEPRECATED: using `SubsystemRule(RuncTool)` is scheduled to be removed in version 2.17.0dev0.

Use `*RuncTool.rules()` instead.
17:40:13.82 [WARN] DEPRECATED: using `SubsystemRule(UmociTool)` is scheduled to be removed in version 2.17.0dev0.

Use `*UmociTool.rules()` instead.
17:40:13.86 [INFO] Initializing scheduler...
17:40:13.86 [WARN] DEPRECATED: Setting `Goal.environment_behavior=EnvironmentBehavior.UNMIGRATED` for `Goal` `decrypt` is scheduled to be removed in version 2.17.0.dev0.

See https://www.pantsbuild.org/v2.15/docs/plugin-upgrade-guide

oci_image_artifact and `COPY --from`

This is an ongoing design to allow OCI containers to produce outputs that aren't full containers. There's three use-cases for this:

Export built artifacts to disk
Implement containerized workflows for things that aren't supported in native pants right now (e.g. Rust-compile-in-a-container)
Support equivalents of COPY --from to make Pex workflows faster.

The below is the current overall target design, with some omissions and estimates. Red nodes are commands. Green nodes are effects.

One clinch point I keep hitting is that not everything that can be moved from container-to-container is safe as a package. Thus, we need somewhere along the chain where we can split "packageable" and "non-packageable".

Currenly, it'd just crash on those
One solution is to have a special target for "leaving OCI land".

graph TD
    oci_image_pull --> PullImageBundle
    oci_image_build --- D[ ]:::empty
    ImageLayer ---|0..N| D[ ]:::empty --> BuiltImageBundle

    BuiltImageBundle --x ImageBundle
    PullImageBundle --x ImageBundle

    PEX[pex_binary, etc] --> BuiltArtifact
    ImageBundle --> base
    BuiltArtifact --> PackageLayer --> ImageLayer
    base --- E[ ]:::empty --> oci_artifact_build -->  OciArtifactLayer --> ImageLayer
    command[/command/] --- E[ ]:::empty
    base[/base/] -->  oci_image_build

    export>pants package]:::command --- F[ ]:::empty --> files:::effect;
    OciArtifactLayer --- F
    
    publish>pants publish]:::command --- G[ ]:::empty --> push:::effect;
    ImageBundle --- G

    classDef empty margin:0px,padding:0px,width:0px,height:0px,border:0px;
    classDef command fill:#f96;
    classDef effect fill:#9f6;