Code Monkey home page Code Monkey logo

Comments (12)

mkow avatar mkow commented on September 3, 2024

There are many errors like that in your log:

(libos_parser.c:1658:buf_write_all) [P7:T178:python3.8] trace: ---- execve("pip", [pip,--disable-pip-version-check,list,], [LD_LIBRARY_PATH=/lib:/lib:/lib/x86_64-linux-gnu:/usr//lib/x86_64-linux-gnu,OMP_NUM_THREADS=4,]) ...
(libos_parser.c:1658:buf_write_all) [P7:T178:python3.8] trace: ---- return from execve(...) = -2

(here it's a bit weird that execve is called with a relative path, is this a bug in concrete-ml?)

(libos_parser.c:1658:buf_write_all) [P6:T177:python3.8] trace: ---- execve("/bin/sh", [sh,-c,--,ld --shared -o /tmp/tmp6b4tt02t/sharedlib.so /tmp/tmp6b4tt02t/program.module-0.mlir.o /usr/local/lib/python3.8/dist-packages/concrete_python.libs/libConcretelangRuntime-32c53a6a.so -rpath=/usr/local/lib/python3.8/dist-packa
(libos_parser.c:1658:buf_write_all) [P6:T177:python3.8] trace: ges/concrete_python.libs --disable-new-dtags 2>&1,], [LD_LIBRARY_PATH=/lib:/lib:/lib/x86_64-linux-gnu:/usr//lib/x86_64-linux-gnu,OMP_NUM_THREADS=4,]) ...
(libos_parser.c:1658:buf_write_all) [P6:T177:python3.8] trace: ---- return from execve(...) = -2

-2 is ENOENT.
You're missing these binaries inside the Gramine namespace (in-enclave virtual filesystem), you're probably missing some mounts in your manifest.

from gramine.

thempp66 avatar thempp66 commented on September 3, 2024

Thanks @mkow for your reply! I'm not sure if it is a bug in concrete-ml but the same code works well in host without Gramine indeed. As you said, if there is something wrong with mounts, what can I do to fix these errors? For example, I should mount to some binaries like /tmp/tmp6b4tt02t/sharedlib.so in my manifest. Is that right?

from gramine.

dimakuv avatar dimakuv commented on September 3, 2024

For example, I should mount to some binaries like /tmp/tmp6b4tt02t/sharedlib.so in my manifest. Is that right?

No need to specify separate files, just specify whole directories. So in this particular case, a mount like this is enough:

fs.mounts = [
  { type = "tmpfs", path = "/tmp" },
]

Similarly, to enable the files under e.g. /bin/ directory, do this:

fs.mounts = [
  { path = "/bin", uri = "file:/bin" },
]

Read more info here: https://gramine.readthedocs.io/en/stable/manifest-syntax.html#fs-mount-points

from gramine.

thempp66 avatar thempp66 commented on September 3, 2024

I have tried to mount the file and path it need. But there is still the same error. I am not sure if set the config in a wrong way or those configurations and errors are unrelated. Here is my manifest:

# Copyright (C) 2023 Gramine contributors
# SPDX-License-Identifier: BSD-3-Clause

# Python3 manifest example

loader.entrypoint = "file:{{ gramine.libos }}"
libos.entrypoint = "{{ entrypoint }}"

#loader.log_level = "{{ log_level }}"
loader.log_level = "all"

loader.env.LD_LIBRARY_PATH = "/lib:/lib:{{ arch_libdir }}:/usr/{{ arch_libdir }}"

# Python's NumPy spawns as many threads as there are CPU cores, and each thread
# consumes a chunk of memory, so on large machines 1G enclave size may be not enough.
# We limit the number of spawned threads via OMP_NUM_THREADS env variable.
loader.env.OMP_NUM_THREADS = "4"

loader.insecure__use_cmdline_argv = true

sys.enable_sigterm_injection = true

fs.mounts = [
  { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
  { path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
  { path = "/usr/{{ arch_libdir }}", uri = "file:/usr/{{ arch_libdir }}" },
{% for path in python.get_sys_path(entrypoint) %}
  { path = "{{ path }}", uri = "file:{{ path }}" },
{% endfor %}
  { path = "{{ entrypoint }}", uri = "file:{{ entrypoint }}" },
  { path = "/etc/hosts", uri = "file:helper-files/hosts" },

  { type = "tmpfs", path = "/tmp" },
  { path = "/usr/local/lib/python3.8/dist-packages/concrete_python.libs" , uri = "file:/usr/local/lib/python3.8/dist-packages/concrete_python.libs" },
  { path = "/bin" , uri = "file:/bin" }
]

sys.stack.size = "2M"
sys.enable_extra_runtime_domain_names_conf = true

sgx.debug = true
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
sgx.enclave_size = "4G"
#sgx.max_threads = {{ '1' if env.get('EDMM', '0') == '1' else '32' }}
sgx.max_threads = 128

sgx.remote_attestation = "{{ ra_type }}"
sgx.ra_client_spid = "{{ ra_client_spid }}"
sgx.ra_client_linkable = {{ 'true' if ra_client_linkable == '1' else 'false' }}

sgx.trusted_files = [
  "file:{{ gramine.libos }}",
  "file:{{ entrypoint }}",
  "file:{{ gramine.runtimedir() }}/",
  "file:{{ arch_libdir }}/",
  "file:/usr/{{ arch_libdir }}/",
{% for path in python.get_sys_path(entrypoint) %}
  "file:{{ path }}{{ '/' if path.is_dir() else '' }}",
{% endfor %}
  "file:scripts/",
  "file:helper-files/",
]

sgx.allowed_files = [
  "file:test.onnx",
  "file:.artifacts/",
  "file:/usr/local/lib/python3.8/dist-packages/concrete_python.libs"
]

And here is the log.txt.
Actually I have no idea what to do next. Could you please give me some more probably way to solve the error? I will appreciate a lot!

from gramine.

dimakuv avatar dimakuv commented on September 3, 2024

From the log:

(libos_parser.c:1658:buf_write_all) [P2:T5:python3.8] trace: ---- execve("/bin/uname", [uname,-p,], [LD_LIBRARY_PATH=/lib:/lib:/lib/x86_64-linux-gnu:/usr//lib/x86_64-linux-gnu,OMP_NUM_THREADS=4,]) ...
(libos_parser.c:1658:buf_write_all) [P1:T1:python3.8] trace: ---- close(7) = 0x0
(pal_files.c:108:file_open) warning: Disallowing access to file '/bin/uname'; file is not trusted or allowed.

You only added the /bin/ directory into fs.mounts, but didn't add it into sgx.trusted_files. So please add:

sgx.trusted_files = [
  ...
  "file:/bin/",
]

from gramine.

thempp66 avatar thempp66 commented on September 3, 2024

Thanks @dimakuv for your help! When added /bin to fs.mounts and sgx.trusted_files, the warning file is not trusted or allowed. have been solved. But the error about ld command still exists.
manifest:
python.manifest.template.txt

log:
log.txt

from gramine.

dimakuv avatar dimakuv commented on September 3, 2024

This line seems to be problematic:

(libos_parser.c:1658:buf_write_all) [P9:T241:sh] trace: ---- stat("ld", 0x2f885840) = -2

So the application wants to find ld binary, but it can't (-ENOENT = -2).

This seems to be because ld is located under /usr/bin/ld, but there is no PATH environment variable inside Gramine environment (you didn't specify it in the manifest file).

So please add smth like this in your manifest and try again:

# the subset of paths is taken from default Ubuntu, contains our desired /usr/bin/ld
loader.env.PATH= "/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

from gramine.

thempp66 avatar thempp66 commented on September 3, 2024

Thanks a lot, @dimakuv . It seems that we can use ld now when add the loader.env.PATH. However, in the same code line, it cannot find the file /tmp/tmpqqe0_3_s/sharedlib.so . Actually, I have { type = "tmpfs", path = "/tmp" } in manifest. Is there any other config that I should add to access the file under /tmp?

Traceback (most recent call last):
  File "scripts/test-ml.py", line 21, in <module>
    model.compile(X_train)
  File "/usr/local/lib/python3.8/dist-packages/concrete/ml/sklearn/base.py", line 575, in compile
    self.fhe_circuit_ = module_to_compile.compile(
  File "/usr/local/lib/python3.8/dist-packages/concrete/fhe/compilation/compiler.py", line 606, in compile
    circuit = Circuit(
  File "/usr/local/lib/python3.8/dist-packages/concrete/fhe/compilation/circuit.py", line 67, in __init__
    self.enable_fhe_execution()
  File "/usr/local/lib/python3.8/dist-packages/concrete/fhe/compilation/circuit.py", line 134, in enable_fhe_execution
    self.server = Server.create(
  File "/usr/local/lib/python3.8/dist-packages/concrete/fhe/compilation/server.py", line 220, in create
    compilation_result = support.compile(mlir, options, compilation_context)
  File "/usr/local/lib/python3.8/dist-packages/concrete/compiler/library_support.py", line 172, in compile
    self.cpp().compile(
RuntimeError: Can't emit artifacts: Command failed:ld --shared -o /tmp/tmpqqe0_3_s/sharedlib.so /tmp/tmpqqe0_3_s/program.module-0.mlir.o /usr/local/lib/python3.8/dist-packages/concrete_python.libs/libConcretelangRuntime-32c53a6a.so -rpath=/usr/local/lib/python3.8/dist-packages/concrete_python.libs --disable-new-dtags 2>&1
Code:256
ld: cannot open output file /tmp/tmpqqe0_3_s/sharedlib.so: No such file or directory

manifest:
python.manifest.template.txt
log:
log.txt

from gramine.

dimakuv avatar dimakuv commented on September 3, 2024

Yes, it looks like your application creates a bunch of files under /tmp/ that are shared among several processes (of the same application). This sharing is not supported by tmpfs in Gramine.

So you can instead use a classic (chroot) FS mount. Smth like this (replace that tmpfs mount entry with this new entry):

fs.mounts = [
  ...
  { path = "/tmp", uri = "file:/tmp" },
]

sgx.allowed_files = [
  ...
  "file:/tmp",
]

This is absolutely insecure (as all files are simply visible to the host), but it should allow you to check the functionality of your application. Until the next problem in your Gramine experiments :)

from gramine.

thempp66 avatar thempp66 commented on September 3, 2024

Thanks again, @dimakuv . In that case, it said FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/']. But we indeed mount the /tmp in manifest.
manifest:
python.manifest.template.txt
log:
log.txt

from gramine.

dimakuv avatar dimakuv commented on September 3, 2024

@thempp66 You added /tmp to sgx.trusted_files. These Trusted Files are read-only. I think that's what "FileNotFoundError" complains about.

You actually need to add /tmp to sgx.allowed_files instead. Allowed Files are read-write.

from gramine.

thempp66 avatar thempp66 commented on September 3, 2024

@thempp66 You added /tmp to sgx.trusted_files. These Trusted Files are read-only. I think that's what "FileNotFoundError" complains about.

You actually need to add /tmp to sgx.allowed_files instead. Allowed Files are read-write.

Thank you all so much! It works now. That's really really important to our project. By the way, sorry about my unfamiliar with gramine and it's config. I'm going to learn more about the way of using Gramine and it's limitation. Thank you!

from gramine.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.