After going through the menu or running heavyscript pvc --mount it shows an empty list. I am on TrueNAS-SCALE-22.12.2 and the most recent version of heavyscript. I have several apps running with PVCs so there should be something displayed.

A recent update of photoprism failed. Now I wanted to manually retry the update (maybe with a longer timeout) but heavyscript is always skipping the previously failed version with the following output:

photoprism
Skipping previously failed version
231011_16.0.3

This can of course make sense as the default behaviour but is there any way to force heavscript to retry the update? I couldn't find anything in the documentation.
In any case, I still want to thank you so much for this amazing tool. It is a true breeze now to keep all my apps up to date.

The section on how to update heavy_script, the example is missing an "_"

Built-In Option (Recommended) bash heavyscript.sh --self-update -b 10 -supr

should be

Built-In Option (Recommended) bash heavy_script.sh --self-update -b 10 -supr

Seems heavyscript delete the current 'myNas01/ix-applications/catalogs' dataset but did not create new one?

When restore a system backup

Starting restore, this will take a LONG time.
[0%] ...
[5%] Basic validation complete...
[15%] Stopped kubernetes...
[17%] Waiting for catalog sync jobs to complete...
Error: Failed to delete dataset: cannot destroy 'myNas01/ix-applications/k3s/kubelet': dataset is busy
Restore failed, exiting..

Scale:

 error: force
Apps have been partially initialized on 'myNas01' pool but it is missing 'myNas01/ix-applications/catalogs' datasets. Specify force to override this and let system re-initialize applications.```

Heavyscript:
[0%] ...
[5%] Basic validation complete...
[15%] Stopped kubernetes...
[17%] Waiting for catalog sync jobs to complete...
Error: Failed to delete dataset: cannot open 'myNas01/ix-applications/catalogs': dataset does not exist
Restore failed, exiting..

Description

Following the recent update to Dragonfish, which removed support for OpenEBS CSI in TrueNAS SCALE ix applications, our ability to backup and restore PVCs using the ix API has been impacted. This issue is critical for handling rollbacks after failed updates, ensuring both applications and their data can be reverted to their previous stable states.

Objective

• Develop a PVC rollback solution: Create a system that not only backs up PVCs but also allows us to rollback these snapshots to recover from failed updates. This system should either integrate with the TrueNAS SCALE rollback API or be a custom-built solution, depending on what's more feasible.

Requirements

• Explore TrueNAS SCALE Rollback API: Determine if we can leverage the existing TrueNAS API for rollbacks and integrate it with our PVC backup system.
• Design and Testing: If a custom solution is needed, design one that ensures seamless rollbacks and test it extensively to guarantee reliability.
• Ensure Compatibility: The solution should work smoothly with our existing infrastructure and applications dependent on persistent storage.

Notes

• Insights or suggestions from the community on leveraging TrueNAS features or on custom rollback solutions would be greatly appreciated.
• This is a wish-list item at the moment. No grantee that this will be added, although I am attempting to configure the backup and restore class to allow for this, implementation may be too complicated to implement and more importantly, maintain..

Hi there,
thx again for the great video and hTruenas Scale - Truecharts Nextcloud - Install & Restore - YouTube @ 20:22.
I would like to finish the conversation we had over there, although it is partly obsolete as it becomes now the official tool as stated in The Future of TrueCharts | TrueCharts --> CONGRATULATIONS (hope it not causes extra workload)
I still feel the conversation hasn't come to proper end, so I do here.

I ask what is the difference to Backup, Migrations and Restore | TrueCharts although TrueTool is mentioned, which you say @ 20:22 is fork/copy of HeavyScript?

Your answer was

Truecharts and I will sometimes work together to solve a project, we worked together to solve the backup/restore guide issue, and integrate fixes within the scripts.
I did most of the coding, so both of the scripts can be used interchangeably with their backup guide.

The only reason they suggest using Truetool, and not heavyscript is because they "verify" their code. Apparently they trust me to develop code, but they don't trust HeavyScript, but they also use most of HeavyScript features? I don't know dude lol.

In my opinion, Truetool is a rebrand of HeavyScript.

Moreover, I asked if you still do a full snapshot of ix-applications as I can find that in Backup, Migrations and Restore | TrueCharts?
I was irritated by the fact their guide still showed the TrueNAS replication options what is not mentioned in your video.

So do you incorporated the data set replication trough TrueNAS itself or do you go 100% with you tool
The only answers in this directions are

It can do some of the same functions in the Truenas SCALE GUI, but that doesn't mean that it is apart of their project.

The only reason you need the truecharts guide, is when you are moving machines, or you are restoring from some sort of remote backup. Theres not really any other reason to do it.

I have one app where heavyscript is removing the docker image but it is needed for an app.

So, I load a docker image in scale using "Manage Docker Images -> Pull Image" as I do for all my containers. In this base, the repo is mine, I have modified the URL to hide it but the image name is repo.mydomain.com:5000/prowlarr/fatula. I have more than a dozen apps like this, and they all pull from the same repo. The image tag, perhaps the tag is somehow an issue with special characters, is 1.7.4.3769-ls28 maybe the dash somehow?

On the "Manage Docker Images" screen, I can now type prowl in the search box and sure enough, it's now there.

Now I run via command line: /root/heavy_script/heavy_script.sh update -v

Now searching for the image shows prowlarr is missing. Trying to start the app fails as there is no image and I have it set to not re-pull (it requires authentication).

Note, all my other apps do not disappear, just this one!

Now the app won't start as it is missing the image. If I re-pull the image, the app then starts but the next run of heavyscript then removes it again. So, the next time a restart occurs of truenas, the app won't start as the image is missing again. It's only this image, all apps use the same repo.

I am running the script as chronjob, sends a mail to mailrise, which ends up as a ntfy notification to my phone.
The output is quite verbose as for a notification. It would be great to have a brief output mode.

Sample:
Heavy Script: latest (v2.9.1) | Heavy Script: updated to v.2.9.1 (new config options)
Catalog sync duration: xx Seconds | not configured
Updates applied:
Plex: 1.40.0.7998_2.0.2 -> 1.40.1.8227_2.0.3
tailscale: 1.60.1_1.0.34 -> 1.62.0_1.0.35
other: failed -> rollback
...

or so.
thanks for the great work!

After I upgraded TrueNas SCALE to dragonfish, I get this error on the nightly app upgrade script by mail:

Warning: Symlink from /root/heavy_script/bin/heavyscript to /usr/local/bin/heavyscript is broken.
Restoring symlink in /usr/local/bin/heavyscript...

ln: failed to create symbolic link '/usr/local/bin/heavyscript': Read-only file system

Which is true /usr in mounted read only:

➜  ~ findmnt -R /usr
TARGET SOURCE                     FSTYPE OPTIONS
/usr   boot-pool/ROOT/24.04.0/usr zfs    ro,relatime,xattr,noacl,casesensitive

Otherwise, heavyscript seems to work fine:

➜  ~ sudo -s
root@nas[/mnt/data/home/user]# heavyscript sync
Warning: Symlink from /root/heavy_script/bin/heavyscript to /usr/local/bin/heavyscript is broken.
Restoring symlink in /usr/local/bin/heavyscript...

ln: failed to create symbolic link '/usr/local/bin/heavyscript': Read-only file system
Please wait while we sync your catalog...
[...]

So is there a way to disable/silence those distracting errors/warnings?

root@truenas[~/heavy_script]# heavyscript -p
🄿 🅁 🅄 🄽 🄴
Error: ReadTimeout(ReadTimeoutError("UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)"))

Hello, I use a heavyscript cron job to update containers at night. Some of my containers are dependent upon others and thus when an update occurs these can breaks and need to be manually restarted.

A feature to address this would be great. I see 2 possible solutions:

1. A cron command with container names to restart post-update or on a timer
2. A full config file to list dependencies

Recently I performed an update to TrueNAS Scale Cobia 23.10.2 and run heavyscript backup to manually backup ix-applications before the OS update. This resulted in following snapshots being created for all ix-applications children datasets:

# zfs list -rt snapshot -o name ssd/ix-applications | less
...
ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27
ssd/ix-applications/default_volumes@ix-applications-backup-system-update--2024-04-30_16:45:51

A replication task I have configured to export ix-applications backups onto a different poll on the same system started failing since then with the following error:

warning: cannot send 'ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27': not an earlier snapshot from the same fs
Replication cannot continue because existing snapshot
ix-applications-backup-system-update--2024-04-30_16:45:51 is newer than
ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27, but has an older date
in the snapshot name. To resolve the error, rename
ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27 with a date that is older than
ix-applications-backup-system-update--2024-04-30_16:45:51 or delete snapshot
ssd/ix-applications/default_volumes@ix-applications-backup-HeavyScript_2024_04_30_18_34_27 from both the source and destination.
cannot receive: failed to read from stream.

Because I made a backup shortly before the OS update it helped me find out that HeavyScript and OS uses different date formatters for snapshot naming:

• ix-applications-backup-HeavyScript uses local (UTC+2) time zone: 18_34_27
• ix-applications-backup-system-update uses UTC time zone: 16:45:51

This causes consecutively created snapshots look out of order failing the replication tasks.

Possible fix

Change HeavyScript timestamp formatter to match the one used by OS.

Workaround

Delete "out-of-order" HeavyScript snapshots:

zfs list -rt snapshot -o name ssd/ix-applications | grep HeavyScript_2024_04_30_18_34_27 | xargs -n1 zfs destroy

To disincentivize allowing root user when not necessary, it'd be great if this script more easily supported use by standard users. Including 1-line deploy script that didn't error if not root and automatic privilege checks/escalation for sudo when necessary.

HeavyScript v2.4.1
TrueNAS-SCALE-22.12.4.2
Cron bash /root/heavy_script/heavy_script.sh update --backup 7 --concurrent 10 --prune --rollback --include-major --self-update
Application created from "Launch Docker Image".
Application stopped from Application "Stop"-button.

Original log part

ntfy
Updated
binwiederhier/ntfy:latest_2308.0.0
binwiederhier/ntfy:latest_2308.0.1
Error: Run Time(500) for ntfy has exceeded Timeout(500)
If this is a slow starting application, set a higher timeout with -t
Reverting update..
Rolled Back
Error: Application did not come up even after a rollback
Manual intervention is required
Stopping, then Abandoning
Stopped

(BTW, where does TrueNAS get those 2308.0.x version numbers from?)
outputfile.txt

k8s-lens cannot connect to the SCALE node by default because the kubernetes API is not exposed. Truetool had an option to expose that, and I've grown quite fond of monitoring the node with lens.

Enabling external access doesn't survive a reboot though, so it has to be activated again after every reboot. It would be great if I could do this with heavyscript too :)

Hello and thank you for your amazing project!

I have TrueNas with a custom app (container tag latest). Everything works fine, but I miss the ability to autoupdate that app the same way is done with TrueNas/Truecharts apps.

It's possible? There is some manual way of doing it?

I could create a PR for that feature if I know more details about how to do it.

BR,
Valentin

Thanks for the very convenient script! I have a suggestion about backup and PVC storage.

I'm one of those people who prefer host paths mainly because if I delete an app, I don't want its data to be deleted. However I do understand the benefits of the PVC storage.

My suggestion is to be able to use backup to extract all the PVC storage of an app in a directory of my choice. With this way I will feel comfortable to use PVC storage for the configs and the database.

instead of doing
heavyscript pod --logs
then selecting app name from the list
make like so
heavyscript pod --logs immich
that way the user can directly select what specific container that they would to pick

https://github.com/Heavybullets8/heavy_script/tree/0729d3d2d59ae58682d84529bd491cecd713d47b/functions/enable

Description

Following the Dragonfish update in TrueNAS SCALE, our script's "enable" functions, such as enabling APT, now require activating "Developer Mode". Previously, these functions operated without the need for any special mode, including features directly in TrueNAS SCALE without additional permissions. The new requirement for Developer Mode, which voids ix support for the system, necessitates a reevaluation of these functions.

Objective

• Decision on 'enable' functions: Determine whether to modify these functions to avoid Developer Mode or provide a clear warning and method for safely enabling it.
• Implement safeguards: If we choose to retain functions that require Developer Mode, implement a robust warning system to inform users of the support implications.

Requirements

• Compatibility Check: Review each function within the 'enable' menu to identify those now requiring Developer Mode.
• User Consent Mechanism: Develop a prompt or warning that fully informs users of the consequences before Developer Mode is activated.
• Documentation Revisions: Update our documentation to clearly state the new requirements and risks associated with using these functions.

Potential Challenges

• Maintaining Functionality vs. Support: Balancing the need for advanced features with the potential loss of official support from ix.
• User Experience: Ensuring that the user interface remains intuitive and that users are fully informed of significant changes in operational prerequisites.

Notes

• This issue is crucial for maintaining the usability and integrity of our scripts in environments where continued support from ix is essential.
• Feedback and suggestions from the community on how to handle this new requirement would be greatly appreciated.

heavyscript enable --disable-api returns Unknown feature: --disbale-api on heavyscript 2.9.0

hi,

The version of my truenas scale is 13.10.0.1,and the Heavyscript is 2.7.7. When installing velero by Heavyscript, it reported an error "Failed to set Velero namespace."
How to solve this error?
Thank you.

When running with -u show proposed and current version when the upgrade is held back due to major upgrade.

When the script runs it runs on all apps and updates them. Once the update is complete apps that we're in a "Stopped" state become active. I don't mind if it updates "Stopped" apps but it would be nice if it could stop them from starting after the update. I have several apps that I only use occasionally and don't want running all the time.
If the script just didn't run on "Stopped" apps that would also be good.

The command heavyscript enable --disable-helm doesn't work and reports Unknown feature: --disable-helm

Hi, since TrueNAS allows pool names with spaces, it would be nice if heavyscript would handle that use-case.
Here's a preview of the issue, but I'm sure there are other instances where the issue might pop up

In this case my pool is called HDD Pool

I think the backup command works fine, which is interesting.

Got this just now (HeavyScript does not know how many replicas this app has, skipping post processing
Please submit a bug report on github so this can be fixed):

🅂 🄴 🄻 🄵
🅄 🄿 🄳 🄰 🅃 🄴
Found a new version of HeavyScript, updating myself...
Updating from: v2.3.1
Updating To: v2.4.0
Changelog:
[Features]

• Starting an app with HeavyScript will now unmount the app if it's mounted.
• CNPG database dumps no longer stop deployments by default, allowing for continuous operation.

[Refactor]

• The CNPG dump process has been refined to keep deployments running.
• Opt-in to stop deployments during the dump using the stop_before_dump option in the config.ini file.
• Your config files should have automatically updated to have this new section.

Running the new version...

🅃 🄰 🅂 🄺 :
-Backing up ix-applications dataset
Please wait..

🄱 🄰 🄲 🄺 🅄 🄿 🅂
Retention: 7

--Snapshots--
New Snapshot Name:
HeavyScript_2023_10_14_00_00_03

Deleted the oldest Snapshot(s) for exceeding limit:
HeavyScript_2023_10_07_00_00_02

🅄 🄿 🄳 🄰 🅃 🄴 🅂
Update(s) Available: 1
Asynchronous Updates: 10
Custom Timeout: 500
Image Updates: Enabled

unifi
Updated Container Image
HeavyScript does not know how many replicas this app has, skipping post processing
Please submit a bug report on github so this can be fixed

🄿 🅁 🅄 🄽 🄴
[0%] ...
[50%] Stopped containers pruned...
[100%] Successfully pruned images/containers...

Unifi app is linuxserver/unifi-controller:latest
TrueNAS-SCALE-22.12.3.3
cronjob command is bash /root/heavy_script/heavy_script.sh update --backup 7 --concurrent 10 --prune --rollback --include-major --self-update

Hi, Truetool had a feature allowing you to enable the k3s api allowing access by external tools like Lens. Will this feature come to heavy_script?

The stop application menu only lets you stop single applications. Using 1-10 or 1,2,3,4 etc does not work. If you wanted to stop all applications or a select number at one time and you have over 30 this becomes very time consuming since you should not be using the stop function in SCALE according to TrueTool.

It would be interesting if we could update just a list of apps, in other words, the opposite of --ignore.

Here's why: I have several apps. All these apps can be updated at night, but only one, Frigate, should not be updated at night but during the day to avoid stopping the recording.

Thx !

I've managed to create a situation where I have mounted directories that I can't delete. The directories have been unmounted using heavyscript, but this empty directory remains. Mounting / unmounting the app again doesn't help, nor does rebooting.

I have switched to the args branch as recommended.

root@scale[/mnt/ixapps/mounted_pvc]# bash -x heavyscript pvc --unmount 
+ set -Eeuo pipefail
+ script_dir=/root/heavy_script
+ [[ ! -d /root/heavy_script ]]
+ cd /root/heavy_script
+ bash ./heavy_script.sh pvc --unmount
find: ‘/mnt/ixapps/ix-applications/releases//volumes/’: No such file or directory
cannot open '': empty component or misplaced '@' or '#' delimiter in name
Failed to unmount tautulli-config.

Hi,
would like to continue the discussion happened in the comments of Truenas Scale - Truecharts Nextcloud - Install & Restore - YouTube in regard to the backup of the Nextcloud database.

TrueCharts is migrating to CloudNativePG - PostgreSQL Operator for Kubernetes and came across Datenbank in neue Nextcloud importieren ... (Import database into new Nextcloud install), which references to Migrating to a different server — Nextcloud latest Administration Manual latest documentation

So I'm wondering could using PostgreSQL native support for PostgreSQL: Documentation: 15: Chapter 26. Backup and Restore and PostgreSQL: Documentation: 15: Chapter 27. High Availability, Load Balancing, and Replication.
The first is used in Backup — Nextcloud latest Administration Manual latest documentation

i want to install heavy script but i get the error ( This script must be run as root. ) does anyone know how i can fix that

I have 1.5.1 running on test instance and all seemed well.

Building up a new box, and have a couple apps running (sonarr, traefik), however DNS List is only showing 1 DNS/app visible...sonarr is blacked out and can't be read on the screen

App Name DNS Name Port
sonarr sonarr.ix-sonarr.svc.cluster.local 8989 <- This is there, you just can't read it on the screen
traefik traefik.ix-traefik.svc.cluster.local 9000

Looking at functions/dns.sh , you are alternating colors, and 1 of them is black (033[1;30m). Lots of people use black background and this gives the illusion that there is nothing there, especially if only a couple apps installed.

I would prefer the header to be say Cyan and all the apps the same color and do away with the alternating row colors. Or at the very least, make them colors that could be displayed in black or white backgrounds :)

All in all a great script though!

Hi, I just upgraded heavyscript on two of my TrueNAS Sclae machines, which went fine so far. But strangely, on one machine I get errors regarding the dump folder, even though I did everything the same. The output is as follows:

...

🅃 🄰 🅂 🄺 🅂 :
-Backing up ix-applications dataset
-Syncing catalog(s)
Please wait for output from both tasks..


find: ‘./database_dumps’: No such file or directory
du: cannot access './database_dumps/*': No such file or directory
🅂 🅈 🄽 🄲
Catalog sync complete

...

The script seems to work fine otherwise, so I'm not sure where this comes from. This happens, no matter if I'm calling the script manually or via parameters, as in a cron job. I called the script as root.

Currently if you run heavyscript enable --apt it does not enable dpkg, which leads to errors like the following when you attempt to install packages

debconf: delaying package configuration, since apt-utils is not installed
Could not exec dpkg!
E: Sub-process /usr/bin/dpkg returned an error code (100)

This can be resolved by using chmod to make /usr/bin/dpkg executable

Hi,
I followed your guide and did a fresh git clone a few hours ago.

I can get your script to work fine except it seems that TrueNAS Scale (22.12.0) doesn't resolve the internal DNS at all.

The strange thing is I was able to set everything up using the local DNS names instead of IPs (like jellyseer, jellyfin, etc) and when I tested the connection using DNS it worked fine.

But now it's no longer working for some reason. I SSH into TrueNAS as root and indeed the lookup fails. I'm running a UDM Special Edition with latest software updates in case that helps.

Name DNS_Name Port

1. heimdall heimdall.ix-heimdall.svc.cluster.local 10014
2. jellyfin jellyfin.ix-jellyfin.svc.cluster.local 8096
3. jellyseerr jellyseerr.ix-jellyseerr.svc.cluster.local 10241
4. netdata netdata.ix-netdata.svc.cluster.local 20489
5. prowlarr prowlarr.ix-prowlarr.svc.cluster.local 9696
6. qbittorrent qbittorrent.ix-qbittorrent.svc.cluster.local 10095
7. radarr radarr.ix-radarr.svc.cluster.local
8. sonarr sonarr.ix-sonarr.svc.cluster.local 8989
9. vaultwarden vaultwarden.ix-vaultwarden.svc.cluster.local 10102

root@truenas[/mnt/tank/software/scripts/heavy_script]# nslookup radarr.ix-radarr.svc.cluster.local
Server: 10.10.4.1
Address: 10.10.4.1#53

** server can't find radarr.ix-radarr.svc.cluster.local: NXDOMAIN

root@truenas[/mnt/tank/software/scripts/heavy_script]#

I am using the example crontask on the documentation page and I receive this error after each run.

error: the server doesn't have a resource type "cluster"

It only started happening after updating to dragonfish and updating heavyscript. As far as I can tell, the script runs normally, it just produces this error after running.

Sometimes with third party registries like truecharts, a question option will change on a chart or be added/removed. As of now there does not seem to be a way to handle this changeover and it results in an inability to directly upgrade a chart. Because these are based in the questions.yaml for the chart, Truenas just tries to apply all of your old values and then gives you nothing but errors when they don't match.

It would be nice if a tool like this could detect these changes and allow for options to be updated.

The latest Dragonfish update (ix applications for TrueNAS SCALE) no longer includes OpenEBS CSI, breaking existing PVC (Persistent Volume Claim) backup and restore functionality powered by the ix API.

We need to replace the ix API solution with our own custom implementation to:

• Support PVC backup and restoration for applications that rely on persistent storage.

As you can see here it's a adding prometheus- suffix before the right url and not showing the port.

Doing heavyscript dns -a output the right url.

As seen in this ticket.

Hi,

I am getting
functions/misc.sh: line 175: export: patchv22120: not a function on both of my TrueNAS-SCALE-22.02.4 boxes.

let heavyscript access the apps question.yaml and values.yaml file that a user fills during the first installation then during updates and backups heavyscript can back those files up as well and then you can expose an option in the cli for hard-reset or hard-upgrade based on those yaml files that way the user doesn't have to refill the same ui form again when reinstalling or doing a custom major update of the app

It would be nice to be able have some kind of flow to view the logs for a container under a running application. I find it very useful to look at logs to see why a container is failing to start and it would be nice to be able to do that easy in the command line rather than clicking through the SCALE UI.

I figure there must be some kind of kubectl incantation that will essentially do this, but it would be nice to have that also be abstracted away just like the other commands in HeavyScript.

An option to purge any PV/PVCs that are no longer being used would be extremely helpful.

k3s kubectl get pv -A | grep Released
zfs destroy -r <pool>/dataset/<pvc>
k3s kubectl delete pv <pvc>