Comments (19)
werkt
commented on May 20, 2024
tl;dr: no, dirs != files, but output_directories seem like the way to go
Ah, that's definitely a no-go.
There is, however, this:
However its implementation in bazel 0.9 looks like this:
And I was very excited to find when I went looking for this, that output_directories now has an implementation in the latest unreleased bazel:
For which I will happily mirror an implementation in buildfarm. My guess on documentation for how to add an entry to the output_directories list is that it will also be supplied in 0.10. Thanks @ulfjack!
from bazel-buildfarm.
excavador
commented on May 20, 2024
If I use "declary_directory" instead of "declare_file" I receive following problem with symlink to system python (only under linux, under OSX everything works fine) (just local bazel, not farm)
Is it expected behavior or another bug in bazel?
ERROR: /home/jenkins/backend-ci/plato/BUILD.bazel:45:1: Error while validating output TreeArtifact File:[[/home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX]bazel-out/k8-dbg/bin]plato/tests.virtualenv : /home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX/bazel-out/k8-dbg/bin/plato/tests.virtualenv/lib/python3.6/config-3.6m-x86_64-linux-gnu (Operation not permitted)
ERROR: /home/jenkins/backend-ci/plato/BUILD.bazel:45:1: not all outputs were created or valid
[jenkins@ci-slave-1:Ireland backend-ci]$ ls -al /home/jenkins/.cache/bazel/_bazel_jenkins/cff2319697c0ff2c301458e924e0bef7/execroot/XXX/bazel-out/k8-dbg/bin/plato/tests.virtualenv/lib/python3.6/
total 44
dr-xr-xr-x. 5 jenkins jenkins 4096 Jan 12 19:27 .
dr-xr-xr-x. 3 jenkins jenkins 23 Jan 12 19:27 ..
lrwxrwxrwx. 1 jenkins jenkins 27 Jan 12 19:27 abc.py -> /usr/lib64/python3.6/abc.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 base64.py -> /usr/lib64/python3.6/base64.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 bisect.py -> /usr/lib64/python3.6/bisect.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 _bootlocale.py -> /usr/lib64/python3.6/_bootlocale.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 codecs.py -> /usr/lib64/python3.6/codecs.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 collections -> /usr/lib64/python3.6/collections
lrwxrwxrwx. 1 jenkins jenkins 40 Jan 12 19:27 _collections_abc.py -> /usr/lib64/python3.6/_collections_abc.py
lrwxrwxrwx. 1 jenkins jenkins 49 Jan 12 19:27 config-3.6m-x86_64-linux-gnu -> /usr/lib64/python3.6/config-3.6m-x86_64-linux-gnu
lrwxrwxrwx. 1 jenkins jenkins 28 Jan 12 19:27 copy.py -> /usr/lib64/python3.6/copy.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 copyreg.py -> /usr/lib64/python3.6/copyreg.py
drwxr-xr-x. 3 jenkins jenkins 65 Jan 12 19:27 distutils
lrwxrwxrwx. 1 jenkins jenkins 37 Jan 12 19:27 _dummy_thread.py -> /usr/lib64/python3.6/_dummy_thread.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 encodings -> /usr/lib64/python3.6/encodings
lrwxrwxrwx. 1 jenkins jenkins 28 Jan 12 19:27 enum.py -> /usr/lib64/python3.6/enum.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 fnmatch.py -> /usr/lib64/python3.6/fnmatch.py
lrwxrwxrwx. 1 jenkins jenkins 33 Jan 12 19:27 functools.py -> /usr/lib64/python3.6/functools.py
lrwxrwxrwx. 1 jenkins jenkins 34 Jan 12 19:27 __future__.py -> /usr/lib64/python3.6/__future__.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 genericpath.py -> /usr/lib64/python3.6/genericpath.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 hashlib.py -> /usr/lib64/python3.6/hashlib.py
lrwxrwxrwx. 1 jenkins jenkins 29 Jan 12 19:27 heapq.py -> /usr/lib64/python3.6/heapq.py
lrwxrwxrwx. 1 jenkins jenkins 28 Jan 12 19:27 hmac.py -> /usr/lib64/python3.6/hmac.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 importlib -> /usr/lib64/python3.6/importlib
lrwxrwxrwx. 1 jenkins jenkins 27 Jan 12 19:27 imp.py -> /usr/lib64/python3.6/imp.py
lrwxrwxrwx. 1 jenkins jenkins 26 Jan 12 19:27 io.py -> /usr/lib64/python3.6/io.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 keyword.py -> /usr/lib64/python3.6/keyword.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 lib-dynload -> /usr/lib64/python3.6/lib-dynload
lrwxrwxrwx. 1 jenkins jenkins 33 Jan 12 19:27 linecache.py -> /usr/lib64/python3.6/linecache.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 locale.py -> /usr/lib64/python3.6/locale.py
-rw-r--r--. 1 jenkins jenkins 0 Jan 12 19:27 no-global-site-packages.txt
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 ntpath.py -> /usr/lib64/python3.6/ntpath.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 operator.py -> /usr/lib64/python3.6/operator.py
-rw-r--r--. 1 jenkins jenkins 4 Jan 12 19:27 orig-prefix.txt
lrwxrwxrwx. 1 jenkins jenkins 26 Jan 12 19:27 os.py -> /usr/lib64/python3.6/os.py
lrwxrwxrwx. 1 jenkins jenkins 33 Jan 12 19:27 posixpath.py -> /usr/lib64/python3.6/posixpath.py
drwxr-xr-x. 2 jenkins jenkins 4096 Jan 12 19:27 __pycache__
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 random.py -> /usr/lib64/python3.6/random.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 reprlib.py -> /usr/lib64/python3.6/reprlib.py
lrwxrwxrwx. 1 jenkins jenkins 26 Jan 12 19:27 re.py -> /usr/lib64/python3.6/re.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 rlcompleter.py -> /usr/lib64/python3.6/rlcompleter.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 shutil.py -> /usr/lib64/python3.6/shutil.py
drwxr-xr-x. 66 jenkins jenkins 4096 Jan 12 19:27 site-packages
-rw-r--r--. 1 jenkins jenkins 27543 Jan 12 19:27 site.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 sre_compile.py -> /usr/lib64/python3.6/sre_compile.py
lrwxrwxrwx. 1 jenkins jenkins 37 Jan 12 19:27 sre_constants.py -> /usr/lib64/python3.6/sre_constants.py
lrwxrwxrwx. 1 jenkins jenkins 33 Jan 12 19:27 sre_parse.py -> /usr/lib64/python3.6/sre_parse.py
lrwxrwxrwx. 1 jenkins jenkins 28 Jan 12 19:27 stat.py -> /usr/lib64/python3.6/stat.py
lrwxrwxrwx. 1 jenkins jenkins 30 Jan 12 19:27 struct.py -> /usr/lib64/python3.6/struct.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 tarfile.py -> /usr/lib64/python3.6/tarfile.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 tempfile.py -> /usr/lib64/python3.6/tempfile.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 tokenize.py -> /usr/lib64/python3.6/tokenize.py
lrwxrwxrwx. 1 jenkins jenkins 29 Jan 12 19:27 token.py -> /usr/lib64/python3.6/token.py
lrwxrwxrwx. 1 jenkins jenkins 29 Jan 12 19:27 types.py -> /usr/lib64/python3.6/types.py
lrwxrwxrwx. 1 jenkins jenkins 32 Jan 12 19:27 warnings.py -> /usr/lib64/python3.6/warnings.py
lrwxrwxrwx. 1 jenkins jenkins 31 Jan 12 19:27 weakref.py -> /usr/lib64/python3.6/weakref.py
lrwxrwxrwx. 1 jenkins jenkins 35 Jan 12 19:27 _weakrefset.py -> /usr/lib64/python3.6/_weakrefset.py
from bazel-buildfarm.
werkt
commented on May 20, 2024
I'm going to have to punt to the bazel project for non-buildfarm behaviors around declare_directory
from bazel-buildfarm.
excavador
commented on May 20, 2024
@werkt Yes. My latest comment about local bazel (not a farm).
Can you please give me advise - is it expected behavior or bug to bazel which I should report to bazelbuild/bazel?
from bazel-buildfarm.
werkt
commented on May 20, 2024
Looks like a bazel bug to me.
My clue: Error while validating output TreeArtifact - I usually don't expect to see classname mentions in expected behavior.
from bazel-buildfarm.
excavador
commented on May 20, 2024
For which I will happily mirror an implementation in buildfarm.
@werkt I am very sorry for probably stupid question...
Why you do not want to use bazel as external repo in farm?
As result you can import libraries from bazel directly to farm and 99% to have correct behavior always
from bazel-buildfarm.
werkt
commented on May 20, 2024
Not stupid, I would say that there are two reasons:
- my implementation of this (or almost any other mechanism) will probably be dissimilar, and would need to accommodate a reflection that will not be quite the same during the workers evaluation to anything implemented in bazel; my guidance for how it should behave had no model in an existing implementation for download, which is why I needed some treatment of it from bazel's side to get started (and I keep forgetting to ask the bazel/remote ex team what their intention was)
- the other is that using bazel sources directly without filtering their packaging requires that I pull in dependencies in the groupings that they've defined for them - the remote ex interactions that I'm interested in are not isolated completely in bazel sources. I'd be reliant or committed to maintaining boundaries in their code, and tbh I don't feel we need to be beholden to it. We already share a common endpoint in googleapis, and that has been helpful enough for me to remain consistent with them.
from bazel-buildfarm.
excavador
commented on May 20, 2024
Thank you!
Wait for fix for double check
from bazel-buildfarm.
excavador
commented on May 20, 2024
Any updates?
from bazel-buildfarm.
werkt
commented on May 20, 2024
No movement. Happy to accept a PR if you want this to be accelerated.
from bazel-buildfarm.
werkt
commented on May 20, 2024
You may be interested in bazelbuild/bazel#4668, which I encountered when trying to demonstrate a solution for this. I'll have to proceed without an integration test on this issue until they fix that.
from bazel-buildfarm.
excavador
commented on May 20, 2024
@werkt That's exactly my problem.
That's why my virtualenv rule (and rules_go) use "declare_file" and store directory instead of file in result.
If you replace "declare_directory" by "declare_file" everything would works fine.
from bazel-buildfarm.
werkt
commented on May 20, 2024
Good, glad we converged on that. That means that you are not getting
digests put together for that, so the output cannot be cached
Hopefully this gets some traction in the next couple of days. Otherwise
I'll have to put up a pull request myself
…On Feb 20, 2018 5:30 PM, "Oleg Tsarev" ***@***.***> wrote:
@werkt <https://github.com/werkt> That's exactly my problem.
That's why my virtualenv rule (and rules_go) use "declare_file" and store
directory instead of file in result.
If you replace "declare_directory" by "declare_file" everything would
works fine.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#77 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AABTlsIZLKrg-DSOhMp9q-rQ-S2-41yJks5tW0eSgaJpZM4Rcu1r>
.
from bazel-buildfarm.
excavador
commented on May 20, 2024
@werkt thank you very much for update!
from bazel-buildfarm.
werkt
commented on May 20, 2024
https://github.com/werkt/bazel-buildfarm/tree/output-directories provides an implementation for buildfarm, according to the observed behavior in bazel, and includes some tests. Don't know what to say for in-bazel support for legit output directory specification, but at least we'll be ready...
from bazel-buildfarm.
excavador
commented on May 20, 2024
@werkt good news! I will check in nearest future how buildfarm works now
from bazel-buildfarm.
pauldraper
commented on May 20, 2024
@werkt I tested https://github.com/werkt/bazel-buildfarm/tree/output-directories and it works.
from bazel-buildfarm.
werkt
commented on May 20, 2024
Thanks @pauldraper, the PR is only currently held up by me - there is substantially strange behavior that needs to be considered when working with output directories, specifically intersections with other output files/dirs, and other inputs. The initial behavior will be to reject these outliers, and I have to clean things up a little more in the change to reject those conditions as invalid initially.
from bazel-buildfarm.
werkt
commented on May 20, 2024
Implemented in #128 and landed
from bazel-buildfarm.
Related Issues (20)
- Are workers in RemoteCasWriter fixed whenever any new storage workers are added afterwards? HOT 2
- ci: windows tests fail very often HOT 2
- image bazelbuild/buildfarm-worker:v2.7.0 fails to start with "libfuse.so.2: cannot open shared object file: No such file or directory" HOT 8
- First GRPC type storage tries to create Fuse Exec FS
- Buildfarm is failing at Bazel@HEAD
- Add an optional filter to limit artifact sizes by Action HOT 4
- Post Local Clean Java Coverage Builds Against Remote K8s Build Farm Result In Invalid Digest Recieved HOT 6
- Diffrence between execution and CAS shard worker HOT 3
- Why does clang work, but llvm-ar not? HOT 4
- rules_oss_audit fails to install dependencies on mac
- Set up OSSF security scorecards
- poisson_distribution_test is failing with BAZEL@HEAD HOT 2
- External dependency of buildfarm fails with bzlmod
- Redis Hot Shard issue due to DispatchMonitor HashMap
- skipLoad looping can exhaust file path length
- Heuristics for controlling putDirectory (linkedInputDirectories) per action
- "./examples.bf-run start" fails HOT 2
- How to obtain remote system information? HOT 1
- Helm instructions did not work for me HOT 3
- Health check does not seem to work on shard-worker when redis is down HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bazel-buildfarm.