Comments (14)
This is with Xcode 13.0 (13A233) running on Big Sur 11.5.2.
If no one has ever complained about this before (and that does seem hard to believe), perhaps it's a new problem with the latest versions.
from zfs.
As you say this sounds like the same issue I was asking about on the openzfsonosx.org forums.
To recap on here, I think I've narrowed down a possible cause to something extended attribute related; I've noticed on my projects that the files that will not unlock all have the com.apple.macl
extended attribute set, and removing this can allow them to be edited normally. The only that way I've found to do this so far is to create a terminal command like so:
strip_xattr() { cat "$1" > "$1.tmp" && mv -f "$1.tmp" "$1"; }
This will remove all extended attributes (not just com.apple.macl
which cannot be manipulated normally as it seems to be protected via System Integrity Protection (a common culprit for "Operation not permitted" errors).
What I can't figure out though is why this is happening; as far as I can tell the attribute is being set correctly, and I can access it using xattr -p com.apple.macl /path/to/file
, plus I can set and retrieve other extended attributes using xattr
with no signs of corruption to the data.
In my case I'm using xattr=sa
on all my datasets (for the improved performance, since macOS makes quite heavy use of extended attributes); I haven't had a chance to run any kind of meaningful test to see if the directory based attributes are affected.
I also haven't found a way to find out more about what's going on; is there a way to find out all the file system calls that a process is making, in case there's a possible culprit? Maybe Xcode (or macOS) is using a call other than what xattr
uses and this is where the problem lies? That's pure speculation though.
from zfs.
ok i read https://blog.xpnsec.com/we-need-to-talk-about-macl/ to get familiar with the xattr
from zfs.
The problem was partially solved by enabeling Developer Mode. Xcode can now save changes without problem. However, the inability to track more than one ZFS-hosted project in recently opened persists, nor does it explain why APFS-hosted projects never had a problem to begin with.
from zfs.
ok i read https://blog.xpnsec.com/we-need-to-talk-about-macl/ to get familiar with the xattr
Thanks for the link! Like I say I'm mostly guessing at com.apple.macl
possibly being the culprit; all I know is that the files that won't unlock have it, and removing it "fixes" the problem, but I have no idea if that's just coincidence or not as it could just be the fact that it's now a new file.
One thing I forgot to add is that for me at least this issue only seems to affect projects with source control (git) active, which has always been a mess of weird permissions to begin with (.git folders and files that the current user can't always delete) so it could be something to do with that that's causing Xcode to mess up? If I open a project or a single file that isn't source control managed then it seems to have no problems editing (no failures to unlock).
So it's possible com.apple.macl
is just misleading and that this isn't an xattr issue at all, and the issue lies somewhere within what's going on with git; this is why we really need some way to dig down into what Xcode is actually trying to do when it "unlocks" a file; any ideas how to do that?
from zfs.
I'm having a problem re-producing this locally, Monterey, zfs mounted inside homedir, with mimic=hfs, and I can start/stop xcode as I see fit.
from zfs.
Are you using source control (git) for the Xcode project? In my case it seems to be source control (managed by Xcode) that is part of the equation, though I'm not sure if the same is true for the OP. But for myself at least projects and files without source control are unaffected.
For starting/stopping Xcode I believe the OP's issue is that when a file in an open project is modified and cannot be unlocked for saving, then Xcode will not close (as there are unsaved changes and it still cannot unlock the file).
For my own case a reproduction would look something like:
- Setup a dataset for your Xcode project with
com.apple.mimic=hfs
(andxattr=sa
, though that might not be related). In my own case I'm using the dataset as a user's entire home folder. - Create an Xcode project with some files in it and enable Xcode's source control (git) support.
- Edit a file and save the changes (the first time should work).
- Wait a few seconds for any behind the scenes operations and then try to edit and save the same file again.
If you're triggering the same issue then Xcode will complain that the file is locked and ask if you want to unlock it, then give an unknown error if you do.
I'm not sure if it's related or not but I've also noticed a lot of sandbox related errors for cfprefsd
, resulting in a large number of unsaved .plist
files under /volume/root/.TemporaryItems
on my home folder datasets. I was able to solve this by giving cfprefsd
full disk access, though this should not be necessary (as the purpose of com.apple.macl
is to eliminate the need to give expanded entitlements). It might be possible a similar issue is occurring for Xcode? That said I'm starting to suspect the real issue is with a file somewhere inside the .git
folder for source control, rather than the files themselves (and the way I'm stripping com.apple.macl
is maybe just bypassing the real problem by giving git a new file)?
Also since I didn't mention it here, but I'm macOS Catalina. While it's possible Apple have improved things in Monterey, that still doesn't explain why ZFS is affected in Catalina but APFS/HFS aren't.
from zfs.
@Haravikk for the sake of helping us debug this further, can you please give us the results of running
/usr/sbin/DevToolsSecurity --status --verbose
on your affected system? If the last line is not "Developer mode is currently enabled." then please see if
/usr/sbin/DevToolsSecurity --enable
resolves your Xcode (and Developer Command Line Tools, like /usr/bin/git) file access problems.
from zfs.
I do use git for my projects.
from zfs.
@rottegift Unfortunately enabling developer mode doesn't resolve this issue for me; I dug around in Console.app and found I was getting a lot of the Sandbox related error messages, but granting Xcode Full Disk Access doesn't fix this which is strange (as that's how I've fixed similar issues with cfprefsd
).
In my case at least it seems to be some kind of infuriating interaction with the Sandboxing on macOS which is for some reason behaving differently for ZFS than it did for APFS or HFS+, even though I've always had the bulk of my users relocated onto a different volume so it shouldn't be that, not unless the change in how I've laid out the volumes is part of the problem (I'm using a separate dataset for each user, except admin, whereas before I had a separate users volume for all non-admin users), but in both cases they were located off my main system volume.
Might mean my issue is something slightly different; tomorrow I'll try testing against an APFS/HFS+ formatted zvol and an ordinary APFS volume, each with the same mountpoint, to see if I can eliminate the layout as the problem (in case it's not ZFS but just sandbox misinterpreting paths as being on different devices when they're not).
For @jeffc768's original case, even if developer mode "fixes" the problem, it doesn't explain why it should be needed though does it? I've never needed to enable developer mode before either, as while it makes some things a bit simpler, everything I usually need always worked fine without it on APFS/HFS+.
from zfs.
So I can confirm that the issue I'm seeing is specific to ZFS datasets; I copied all of the affected user's data into an HFS+ formatted zvol and everything worked as normal with no need for developer mode to be enabled, and this resolved all of the other sandboxing issues I've been noticing as well. I had the HFS+ volume mounted at the exact same location that I previously mounted my ZFS dataset (to rule out the user home folder being a volume as a possible cause).
It definitely seems like the problem with files not unlocking in Xcode, and similar issues in other programs (cfprefsd
, Stickies and so-on that I mention in the linked thread) are related to some kind of fault caused by ZFS and how the macOS sandboxing system works.
What I can't figure out is what the problem is exactly; I still can't help but feel like it must be related to the com.apple.macl
attribute somehow, as files without it appear to be unaffected (so long as other sandboxing entitlements are granted, i.e- Downloads folder access or similar), and it feels like the attribute is causing sandboxing to deny file operations even though it's intended to do the opposite. My best guess would be that the sandboxing code sees the attribute is present, but somehow fails to verify it and so rejects it? However as I said, I can seem to manipulate attributes using xattr
without issue, so the problem must have something to do with how sandboxing fetches it? Purely guesswork though.
Are there any other ways I could try to debug this to confirm? Any way to get more detail about what's going on at the filesystem level? It might help if I could isolate some kind of filesystem instruction that could be to blame, but I have no idea how to actually do that.
In the mean time I'll have to either expand the HFS+ zvol and use that for my affected user, or move that one back to my system volume; I'll keep the affected dataset for now so I can test (and because I'd really like to be able to use it).
from zfs.
I found out about the fs_usage
command and decided to give it a try on Xcode while editing and then attempting to re-edit the same file (which causes the unlocking issue in my case).
Unfortunately Xcode seems to be a very noisy program in fs_usage
, probably not helped by me running a quite RAM hungry operation at the same time (which is causing swap related entries to appear as well). I've decided to leave everything in though as I don't want to risk removing anything useful.
You can download my fs_usage
output as a .zip here.
There are two files, the first (fs_usage.xcode.edit.txt
) was taken when performing a normal edit of a file (Jobmodel.php
) which occurred without any issues. The second (fs_usage.xcode.unlock.txt
) shows when I tried to edit the same file a second time and failed to unlock, plus a second attempt to manually unlock (clicking the padlock in Xcode).
Please let me know if these are of no use, and if there's anything else I can do to more effectively debug this issue. I also captured some cleaner fs_usage
output from TextEdit while exhibiting the same basic issue, which you can see on my forum thread, in case that's easier to pick through (it has a lot fewer messages, but is demonstrating only the error case). I can try to get cleaner Xcode output when my video transcoding tasks are done, but that's going to be several days at least.
from zfs.
@lundman is there anything more I can do try to debug this?
I'm a bit out of my depth on this one as I do not know nearly enough about implementing filesystems in general or on macOS specifically.
I had thought maybe if I could figure out where in the code some of the commands reported by fs_usage
are actually implemented I might be able to pick through and find a possible cause? Is there a particular file or group of files that act as the entry point for the commands I see in fs_usage
(like lstat64
, getattrlist
and so-on), but all the ones I've tried to find I haven't had much luck in searching for (I can find places where they're used but not where the ZFS side is defined).
My working theory is still that it's extended attribute related somehow, most likely that either com.apple.macl
is being set incorrectly, or is not being returned correctly by an extended attribute call; I've noticed in the past when debugging Time Machine related problems that it seems to add an extra null character to a lot of its extended attributes, and while these seem irrelevant in Time Machine's case, they do seem like something that stricter programs could stumble upon (expecting a null at the end that isn't there)?
By comparing output from fs_usage
for affected programs to that of fs_usage
for xattr
, one thing I have noticed is that the affected programs seem to use the command getattrlist
, while xattr
seems to use listattr
followed by multiple calls to getattr
. I haven't found a reference to what the difference is, but my guess is that getattrlist
is some kind of combined call that returns keys and values as a single command, and I'm wondering if maybe it's returning a slightly different result on ZFS to that of HFS/APFS and as a result not what macOS, or at least macOS' sandboxing, expects?
from zfs.
lesse if I can catch up again. For getattrlist
I found the tool FSMegaInfo to be handy for it, as you can send the same queries as seen in dtrace/dtruss. I have a decode_getattrlist
to convert the dtruss bits to strings. Then the same queries can be sent to apfs and zfs and compare.
We may be handling xattrs wrong just in general, or a specific one. But hfs only has 1 xattrs they do special work "cprotect". So perhaps it's the generic case. If you do find we add "an extra null" then we should definitely fix that.
Making a new build today for a couple of other issues and to back into the swing of things again.
from zfs.
Related Issues (20)
- diskutil erase ZFS shows -69832: file system formatter failed HOT 1
- Error "no pools available to import" in Big Sur x86_64
- Unexpectedly low performance of VMs with openzfsonosx backing store HOT 2
- Photos cannot import photos to library on zfs HOT 5
- _zfs_findernotify_callback panic HOT 1
- Kernel Panic - OpenZFS 2.1.0 on Apple Silicon HOT 11
- Kernel Panic (Time Out) - OpenZFS 2.1.0 on Apple Silicon HOT 20
- Kernal Panic - 2.1.x arm64 HOT 16
- Kernel Panic: Kernel Data Abort (OpenZFS 2.1.0, M1 Ultra) HOT 15
- how can i create block dev as on linux HOT 1
- 2.1.99: `zsysctl` sets tunable incorrectly HOT 2
- Launchd: no pools available to import HOT 3
- Kernel Panic: OpenZFS 2.1.0 Catalina HOT 1
- System freezes with command: zpool offline
- System restarts with command: zpool export
- System freezes or reboots importing a degraded mirror image HOT 52
- System freezes on import HOT 1
- Mac ZFS triggers duplicating finding programs to fail randomly HOT 9
- `cargo build` breaks with "resource temporarily unavailable (error 35)" (I'm assuming EAGAIN) on v2.2.0 and v2.2.2 HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from zfs.