Comments (12)
There are many errors like that in your log:
(libos_parser.c:1658:buf_write_all) [P7:T178:python3.8] trace: ---- execve("pip", [pip,--disable-pip-version-check,list,], [LD_LIBRARY_PATH=/lib:/lib:/lib/x86_64-linux-gnu:/usr//lib/x86_64-linux-gnu,OMP_NUM_THREADS=4,]) ...
(libos_parser.c:1658:buf_write_all) [P7:T178:python3.8] trace: ---- return from execve(...) = -2
(here it's a bit weird that execve
is called with a relative path, is this a bug in concrete-ml
?)
(libos_parser.c:1658:buf_write_all) [P6:T177:python3.8] trace: ---- execve("/bin/sh", [sh,-c,--,ld --shared -o /tmp/tmp6b4tt02t/sharedlib.so /tmp/tmp6b4tt02t/program.module-0.mlir.o /usr/local/lib/python3.8/dist-packages/concrete_python.libs/libConcretelangRuntime-32c53a6a.so -rpath=/usr/local/lib/python3.8/dist-packa
(libos_parser.c:1658:buf_write_all) [P6:T177:python3.8] trace: ges/concrete_python.libs --disable-new-dtags 2>&1,], [LD_LIBRARY_PATH=/lib:/lib:/lib/x86_64-linux-gnu:/usr//lib/x86_64-linux-gnu,OMP_NUM_THREADS=4,]) ...
(libos_parser.c:1658:buf_write_all) [P6:T177:python3.8] trace: ---- return from execve(...) = -2
-2 is ENOENT
.
You're missing these binaries inside the Gramine namespace (in-enclave virtual filesystem), you're probably missing some mounts in your manifest.
from gramine.
Thanks @mkow for your reply! I'm not sure if it is a bug in concrete-ml but the same code works well in host without Gramine indeed. As you said, if there is something wrong with mounts, what can I do to fix these errors? For example, I should mount to some binaries like /tmp/tmp6b4tt02t/sharedlib.so
in my manifest. Is that right?
from gramine.
For example, I should mount to some binaries like /tmp/tmp6b4tt02t/sharedlib.so in my manifest. Is that right?
No need to specify separate files, just specify whole directories. So in this particular case, a mount like this is enough:
fs.mounts = [
{ type = "tmpfs", path = "/tmp" },
]
Similarly, to enable the files under e.g. /bin/
directory, do this:
fs.mounts = [
{ path = "/bin", uri = "file:/bin" },
]
Read more info here: https://gramine.readthedocs.io/en/stable/manifest-syntax.html#fs-mount-points
from gramine.
I have tried to mount the file and path it need. But there is still the same error. I am not sure if set the config in a wrong way or those configurations and errors are unrelated. Here is my manifest:
# Copyright (C) 2023 Gramine contributors
# SPDX-License-Identifier: BSD-3-Clause
# Python3 manifest example
loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "{{ entrypoint }}"
#loader.log_level = "{{ log_level }}"
loader.log_level = "all"
loader.env.LD_LIBRARY_PATH = "/lib:/lib:{{ arch_libdir }}:/usr/{{ arch_libdir }}"
# Python's NumPy spawns as many threads as there are CPU cores, and each thread
# consumes a chunk of memory, so on large machines 1G enclave size may be not enough.
# We limit the number of spawned threads via OMP_NUM_THREADS env variable.
loader.env.OMP_NUM_THREADS = "4"
loader.insecure__use_cmdline_argv = true
sys.enable_sigterm_injection = true
fs.mounts = [
{ path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
{ path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
{ path = "/usr/{{ arch_libdir }}", uri = "file:/usr/{{ arch_libdir }}" },
{% for path in python.get_sys_path(entrypoint) %}
{ path = "{{ path }}", uri = "file:{{ path }}" },
{% endfor %}
{ path = "{{ entrypoint }}", uri = "file:{{ entrypoint }}" },
{ path = "/etc/hosts", uri = "file:helper-files/hosts" },
{ type = "tmpfs", path = "/tmp" },
{ path = "/usr/local/lib/python3.8/dist-packages/concrete_python.libs" , uri = "file:/usr/local/lib/python3.8/dist-packages/concrete_python.libs" },
{ path = "/bin" , uri = "file:/bin" }
]
sys.stack.size = "2M"
sys.enable_extra_runtime_domain_names_conf = true
sgx.debug = true
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
sgx.enclave_size = "4G"
#sgx.max_threads = {{ '1' if env.get('EDMM', '0') == '1' else '32' }}
sgx.max_threads = 128
sgx.remote_attestation = "{{ ra_type }}"
sgx.ra_client_spid = "{{ ra_client_spid }}"
sgx.ra_client_linkable = {{ 'true' if ra_client_linkable == '1' else 'false' }}
sgx.trusted_files = [
"file:{{ gramine.libos }}",
"file:{{ entrypoint }}",
"file:{{ gramine.runtimedir() }}/",
"file:{{ arch_libdir }}/",
"file:/usr/{{ arch_libdir }}/",
{% for path in python.get_sys_path(entrypoint) %}
"file:{{ path }}{{ '/' if path.is_dir() else '' }}",
{% endfor %}
"file:scripts/",
"file:helper-files/",
]
sgx.allowed_files = [
"file:test.onnx",
"file:.artifacts/",
"file:/usr/local/lib/python3.8/dist-packages/concrete_python.libs"
]
And here is the log.txt.
Actually I have no idea what to do next. Could you please give me some more probably way to solve the error? I will appreciate a lot!
from gramine.
From the log:
(libos_parser.c:1658:buf_write_all) [P2:T5:python3.8] trace: ---- execve("/bin/uname", [uname,-p,], [LD_LIBRARY_PATH=/lib:/lib:/lib/x86_64-linux-gnu:/usr//lib/x86_64-linux-gnu,OMP_NUM_THREADS=4,]) ...
(libos_parser.c:1658:buf_write_all) [P1:T1:python3.8] trace: ---- close(7) = 0x0
(pal_files.c:108:file_open) warning: Disallowing access to file '/bin/uname'; file is not trusted or allowed.
You only added the /bin/
directory into fs.mounts
, but didn't add it into sgx.trusted_files
. So please add:
sgx.trusted_files = [
...
"file:/bin/",
]
from gramine.
Thanks @dimakuv for your help! When added /bin to fs.mounts
and sgx.trusted_files
, the warning file is not trusted or allowed.
have been solved. But the error about ld
command still exists.
manifest:
python.manifest.template.txt
log:
log.txt
from gramine.
This line seems to be problematic:
(libos_parser.c:1658:buf_write_all) [P9:T241:sh] trace: ---- stat("ld", 0x2f885840) = -2
So the application wants to find ld
binary, but it can't (-ENOENT = -2
).
This seems to be because ld
is located under /usr/bin/ld
, but there is no PATH environment variable inside Gramine environment (you didn't specify it in the manifest file).
So please add smth like this in your manifest and try again:
# the subset of paths is taken from default Ubuntu, contains our desired /usr/bin/ld
loader.env.PATH= "/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
from gramine.
Thanks a lot, @dimakuv . It seems that we can use ld now when add the loader.env.PATH
. However, in the same code line, it cannot find the file /tmp/tmpqqe0_3_s/sharedlib.so
. Actually, I have { type = "tmpfs", path = "/tmp" }
in manifest. Is there any other config that I should add to access the file under /tmp
?
Traceback (most recent call last):
File "scripts/test-ml.py", line 21, in <module>
model.compile(X_train)
File "/usr/local/lib/python3.8/dist-packages/concrete/ml/sklearn/base.py", line 575, in compile
self.fhe_circuit_ = module_to_compile.compile(
File "/usr/local/lib/python3.8/dist-packages/concrete/fhe/compilation/compiler.py", line 606, in compile
circuit = Circuit(
File "/usr/local/lib/python3.8/dist-packages/concrete/fhe/compilation/circuit.py", line 67, in __init__
self.enable_fhe_execution()
File "/usr/local/lib/python3.8/dist-packages/concrete/fhe/compilation/circuit.py", line 134, in enable_fhe_execution
self.server = Server.create(
File "/usr/local/lib/python3.8/dist-packages/concrete/fhe/compilation/server.py", line 220, in create
compilation_result = support.compile(mlir, options, compilation_context)
File "/usr/local/lib/python3.8/dist-packages/concrete/compiler/library_support.py", line 172, in compile
self.cpp().compile(
RuntimeError: Can't emit artifacts: Command failed:ld --shared -o /tmp/tmpqqe0_3_s/sharedlib.so /tmp/tmpqqe0_3_s/program.module-0.mlir.o /usr/local/lib/python3.8/dist-packages/concrete_python.libs/libConcretelangRuntime-32c53a6a.so -rpath=/usr/local/lib/python3.8/dist-packages/concrete_python.libs --disable-new-dtags 2>&1
Code:256
ld: cannot open output file /tmp/tmpqqe0_3_s/sharedlib.so: No such file or directory
manifest:
python.manifest.template.txt
log:
log.txt
from gramine.
Yes, it looks like your application creates a bunch of files under /tmp/
that are shared among several processes (of the same application). This sharing is not supported by tmpfs in Gramine.
So you can instead use a classic (chroot) FS mount. Smth like this (replace that tmpfs
mount entry with this new entry):
fs.mounts = [
...
{ path = "/tmp", uri = "file:/tmp" },
]
sgx.allowed_files = [
...
"file:/tmp",
]
This is absolutely insecure (as all files are simply visible to the host), but it should allow you to check the functionality of your application. Until the next problem in your Gramine experiments :)
from gramine.
Thanks again, @dimakuv . In that case, it said FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']
. But we indeed mount the /tmp in manifest.
manifest:
python.manifest.template.txt
log:
log.txt
from gramine.
@thempp66 You added /tmp
to sgx.trusted_files
. These Trusted Files are read-only. I think that's what "FileNotFoundError" complains about.
You actually need to add /tmp
to sgx.allowed_files
instead. Allowed Files are read-write.
from gramine.
@thempp66 You added
/tmp
tosgx.trusted_files
. These Trusted Files are read-only. I think that's what "FileNotFoundError" complains about.You actually need to add
/tmp
tosgx.allowed_files
instead. Allowed Files are read-write.
Thank you all so much! It works now. That's really really important to our project. By the way, sorry about my unfamiliar with gramine and it's config. I'm going to learn more about the way of using Gramine and it's limitation. Thank you!
from gramine.
Related Issues (20)
- bash workload fails with `Error relocating ls: statx: symbol not found` for Alpine Container HOT 3
- [Pal-SGX/LibOS] Move (expensive) creation of (shallow) merkle tree of trusted files out of critical path HOT 3
- Release 1.7 checklist HOT 1
- LibOS test: tcp_einprogress related testcases fails intermittently across distros HOT 2
- Performance degradation seen with secure eventfd HOT 13
- Proper emulation of magic symlinks (e.g. in procfs) HOT 2
- [LibOS] process current working directory is path-based instead of inode-based
- [CI] Don't forget to move all tests to Ubuntu 22.04 and 24.04 when removing 20.04 support
- mknod with S_IFIFO fails to create FIFO pipe, function call return success does not throw any error HOT 1
- The `CI-examples/ra-tcp-mbedtls` does not work OoTB HOT 1
- busybox workload started failing with centos stream 9 HOT 2
- [LibOS] Keep all VMAs of a given LibOS handle on a linked list
- MariaDB Replication Is Broken HOT 35
- error in manifest: required key not provided @ data['loader']['entrypoint'] HOT 2
- Cannot import concrete-ml library in python HOT 5
- Mentorship Opportunity for CCC Projects
- PermissionError: [Errno 13] Permission denied HOT 7
- [LibOS] Move `prot_refresh_mmaped_from_file_handle` and `reload_mmaped_from_file_handle` into a helper func
- [PAL/Linux-SGX] Use `XSAVEC` or `XSAVEOPT` instead of `XSAVE` instruction HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from gramine.