lsattr and perl is not installed on xigmanas platfform.

lsattr can be installed on xigmanas (login via ssh) by:
pkg install e2fsprogs

perl is only used in teh script to calc the percentage.

just replace line 72:
progress_percent=$(perl -e "printf('%0.2f', ${current_index}*100/${file_count})")
with
progress_percent=$((${current_index}*100/${file_count}))

What effect will block cloning in OpenZFS 2.2 have on this script? The documentation may need to be updated.

When creating a copy of certain files, if the maximum length of the new filename (with .balance appended to it) exceeds the system's maximum defined filename length, the rebalancing operation fails. Would it be possible to modify the script to skip files like these?

Hey there,

Trying to run this on my TrueNAS Scale system. When doing so I just get "./zfs-inplace-rebalancing.sh: line 60: bc: command not found"

Seems to be related specifically to the "progress_percent=$(echo "scale=2; ${current_index}*100/${file_count}" | bc)" line in the code for the script.

Any ideas? Thanks!

Keep track of the lowest pass count in the db and run the script in a loop internally until the target count is reached.

In the case that a filename is something like /pool/projects/[2019] cool project/file.txt, it will constantly be rebalanced, even if a prior run had balanced it.
This rebalancing will often be done silently, but depending on the filename, it will also print an error from grep relating to an invalid regular expression.

Files might be deleted while the script runs. Since this should not affect the rebalancing, it should be checked to avoid an error. Note that the db item has to be removed in that case or it might lead to an infinite loop when #2 is implemented.

Hi all,

I would like to discuss, and expand my knowledge, about potentially enhancing the script to handle hardlinks, rather than simply ignoring them.

In #22 johnpyp mentions "(data) can't be trivially un-hardlinked after without knowledge of which path is in the "balance target" vdev."
I don't quite understand this, could you please expand? Lets say we don't know the path of the balance target vdev and pick a path at random, in the end would it not average out ?

Lets say i have 2vdevs, and they are 60% populated, with files that each have one or more hardlinks, which is balanced 50-50 data usage.
I now add a new empty equal size vdev to the pool and wish to rebalance.

We look at file_1, its got two hardlinks (50%-50%-0)
Could i copy file_1 --> file_1_tmp
Delete the two hardlinks & file_1
Create two new hardlinks to file_1_tmp
Rename file_1_tmp

What does the end data result look like? (15%-15%-70% or so?)

I understand i could be quite naive/crude in this approach, but wish to understand and hope to resolve this, as i'm sure alot of people with similar *arr media setups would appreciate such a feature.

Thanks,
NickyD

root@test-box-not-quiz-box:~# ./zfs-inplace-rebalancing.sh
Usage: zfs-inplace-rebalancing -checksum true -passes 1 /my/pool

See anything strange there?

It appears the the usage shows one dash as part of the parameter requirements not two. That is to say "-checksum" instead of "--checksum" (dash vs dash dash).

I would very respectfully request that this issue be remediated.

Stuart

ShellCheck linting

Would this script work even without any disk remodeling?

I started my Raidz1 with a recordsize of 128k and my files are all huge (11GB up to 1.5TB) I changed it recently to recordsize=1M and the new files read 20x faster than old ones, so I wanted to rewrite the whole 65TB worth of data.

Thanks for this.

./zfs-inplace-rebalancing.sh: line 60: bc: command not found

My guess it is something just missing in the base install of TrueNAS Scale.
What are your thoughts running the script in a Docker container that has the missing component?

I'm using this on truenas Scale (if that matters), trying to rebalance a directory (that I changed checksuming on) and get the following error:

 cp -adxp foo.tif foo.tmp
cp: preserving permissions for ‘foo.tmp’: Operation not permitted

-a and -p don't seem to work, my guess possibly because of windows permissions?

-dx work but this creates new timestamps. The weird thing is, even with the "operation not permitted" message, the file is created with the same timestamp.

-rwxrwxr-x  1 evan root 122878676 Jul 16  2017  foo.tmp
-rwxrwxr-x  1 evan root 122878676 Jul 16  2017  foo.tif.balance
-rwxrwxr-x  1 evan root 122878676 Jul 16  2017  foo.tif

If I try to execute this script in a standard GUI root shell, it fails with the following erorr:

/root/zfs-inplace-rebalancing.sh: 28: Syntax error: "(" unexpected

Is this expected? What can I do to mitigate the issue? Other .sh scripts here execute without issues. This version of TrueNAS runs on FreeBSD13.1 release p9.

I've never gotten the job to complete because if my SSH connection times out in putty or my PC sleeps, when I resume, the script starts from the beginning of the file list, even though it doesn't reprocess the files it's already rebalanced.

I'd like to request an enhancement, which is an option to write a flag for each file that's moved and then have the script resume from that point, rather than starting from the beginning. I realize that doing so would break rebalancing for multiple passes, which is why I suggest it be an option to enable.

Something else needs to be used or the file attributes comparison needs to be removed

Start rebalancing Mon Jul 1 01:48:22 PM MDT 2024:
Copying 'blah' to 'blah.balance'...
Comparing copy against original...
MD5 FAILED: ---------------------- -rw-rwxr--+ user media 5f7973ff9e4152827994d4149d8af39d != ---------------------- -rw-rwxr--+ user user 5f7973ff9e4152827994d4149d8af39d

tried on a few datasets, they all error immediately. Just wondering if its because of fsacl as the md5's do match.

Also feature request to clean up .balance files on exit.

Thank you

Reading through some of the information here on this project I began to contemplate if there might be in any utility in doing something like using dd to read and rewrite each individual block in a file to accomplish the same mission intended by your code?

If so, this would mean that instead of needing enough space to hold the largest file in your directory, you would just need one free block worth of space. Moreover, one could set a number of blocks (if this idea would work), like read 10 blocks write 10 blocks, as it walked through the entire file.

Stuart

TL;DR: What I need is to be able to tell the script to rebalance a directory, but stop after X number of files or, better yet, stop after Y number of bytes. Another command line switch, perhaps?

Read on, for my specific situation that prompted this request...

I just doubled the size of my main pool by adding a second, same-sized vdev to it. As part of the process, the first vdev ended up at a little more than 94% utilization. Not ideal, but hoped to be temporary. So, after adding the new vdev, I was sitting at 94% usage on the first vdev and 0% on the second.

After using du a few times, to look for good candidates for the script to rebalance, I'm currently sitting at 76% / 20% balance. That's certainly good enough. I'm not looking for 47% / 47% balance, here. I know this whole rebalancing process is a bit of a feel good measure, but 94% / 0% just feels icky. What I'm looking for is more like 70% / 24%.

So now I'm facing a dilemma. The only directory I have left, that's big enough, is my "Movies" directory, at 13 TB. That directory is filled with movies, each in a subdirectory, one level down. I must either rebalance the entire Movies directory (13 TB is unnecessarily too much,) or find some other large directory to rebalance (there isn't one.)

I need what the TL;DR says.

It often stops (not always though) with this issue

Comparing copy against original...
MD5 FAILED:
---------------------- -rw-r--r--+ naser wheel 27ca390baf807f9728af92442217769b
!=
---------------------- -rwxr--r--+ naser wheel 27ca390baf807f9728af92442217769b

it seems to me that the MD5 matches, only the x permission is different.
I have issue with permissions sometimes (umask 022)

Hi, Mark

I have a simple question:
I have lots of hardlinked files in ZFS, they have been deduped with Czkawka.

How are they handled? Here's my use case:
/mnt/Zpool/downloads/media/some.ext
/mnt/Zpool/Media Library/Movies/Movie Name (xxxx)/correctname.ext (hardlink to some.ext)

Because I recently expanded my pool from 2x(12x 1.2TB SAS Raid-Z2) to 3x(12x 1.2TB SAS Raid-Z2) my storage is 60+% percent on the first 2x VDEV's (90+% full each) and the last VDEV is sitting empty...