Regression on POSIX ql_syscall_open about qiling HOT 8 OPEN

@iMoD1998, I opened a PR with the necessary fixes.
For some reason it fails on Python 3.12, so if you are not using Python 3.12 you are welcome to pull these changes before they get merged.

from qiling.

Hi there and thanks for reporting this.
Are you sure this is a real issue? Maybe a test case that supports it?

According to POSIX manuals -1 should be returned on error and the program is resonsible to read errno and decide what how to handle the error: "On error, -1 is returned and errno is set to indicate the error".

It is true that we don't set errno anywhere there (and maybe we should), but the fix doesn't seem to be the one you suggested.

from qiling.

Hello, you are correct about libc implementation returning -1 and setting errno but this is the syscall version (has no access to errno) so the error is returned which is then set to errno in libc's wrapper.

This does cause problems as specifically musl libc version 1.2.0 when trying to search through the libary paths will specifically check for ENOENT or something of the sort but will fail outright if it encounters EPERM aka -1.

If you see the mentioned issues i believe for the same reason they are having the same problem.

In the past open has been correct which i why i say regression, one of my emulations has failed from 1.4.5 to 1.4.6 due to this.

from qiling.

Can you please point me to a resource I can look into it? (syscall vs. libc wrapped)
The linked issues are written in Chinese, which I don't understand, so I can't comment.

from qiling.

Yep sure, hopefully musl source would suffice. This is not exclusive to just open, most libc funcs that set errno are done with return value from syscall.

So if you see the implmentation of open in musl:
https://git.musl-libc.org/cgit/musl/tree/src/fcntl/open.c

It calls __syscall_ret.

Which is defined as:
https://git.musl-libc.org/cgit/musl/tree/src/internal/syscall_ret.c

All syscall errors are negative, so they flip the sign and set it to errno.

Which is why in my proposed change i return -errno.

from qiling.

You can also test this on your host machine by calling the syscall directly avoiding libc and see the return value. I can provide that too if you wish.

from qiling.

I believe this is the commit that caused the regression:
0be4620#diff-8dd9355d1cdbad23bf5d2257c46c7fb8310a570a2f45302ce9cf54fed680fa63

from qiling.

Thanks i appricate it.

from qiling.