appimagecommunity / appimageupdate Goto Github PK
View Code? Open in Web Editor NEWAppImageUpdate lets you update AppImages in a decentral way using information embedded in the AppImage itself.
Home Page: https://appimage.org
License: MIT License
AppImageUpdate lets you update AppImages in a decentral way using information embedded in the AppImage itself.
Home Page: https://appimage.org
License: MIT License
Since for the "-j" option the zsync client is created on the stack (https://github.com/AppImage/AppImageUpdate/blob/rewrite/src/updater.cpp#L520) and not assigned to zsyncClient
, there's no way to currently pull for error messages https://github.com/AppImage/AppImageUpdate/blob/rewrite/src/updater.cpp#L618.
Should be rather easy to implement since the update information is at a fixed offset.
me@host:~$ xxd /isodevice/Applications/Audacity-2.0.5.glibc2.15-x86_64.AppImage | head -n 1
00000000: 7f45 4c46 0201 0100 0000 0000 0000 0000 .ELF............
me@host:~$ strings /isodevice/Applications/Audacity-2.0.5.glibc2.15-x86_64.AppImage | grep CD001
CD001
As per the AppImageSpec, this is a valid type-1 AppImage (as is any ISO9660 file that is a Linux ELF file at the same time).
Nevertheless, AppImageUpdate fails:
me@host:~$ Downloads/AppImageUpdate-164-6e435c2-x86_64.AppImage /isodevice/Applications/Audacity-2.0.5.glibc2.15-x86_64.AppImage
AppImageUpdate version 1-alpha (commit 6e435c2), build 164 built on 2017-11-21 11:07:44 UTC
Checking for updates...
Invalid magic bytes: 00
Parsing AppImage failed. See previous message for details. Are you sure the file is an AppImage?
In line with other GUI applications (example: Firefox), let's have
This will make the UI look more familiar and in line with most applications.
Screenshot: How Firefox asks when opening files marked as executable (e.g., by appimaged
). Note: This is a Firefox dialog, not something from AppImageKit.
Screenshot: AppImageUpdate
me@host:~$ ./OldImageMagick-git.7a48608-x86_64.AppImage --appimage-updateinformation
gh-releases-zsync|probonopd|ImageMagick|continuous|ImageMagick*-x86_64.AppImage.zsync
AppImageUpdate version 1-alpha (commit 48d9713), build 119 built on 2017-11-07 02:37:53 UTC says that I already have the latest version, except that it isn't:
(Zipped for GitHub Issues)
OldImageMagick-git.7a48608-x86_64.AppImage.zip
AppImageUpdate tends to completely download a new file from the server when the server's and the local filename don't match.
The problem is that zsync (which is called internally) looks for a file matching Filename:
in the .zsync
file. This is totally fine behavior for "normal" download jobs, but does not work very well with the way we use it.
Thus, we should make sure that Filename:
matches argv[1]
, by replacing that (using sed
or a comparable alternative).
me@host:~$ dd if=Downloads/OpenSCAD-Nightly-0-Build1967.1.glibc2.14-x86_64.AppImage of=partial.AppImage bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00188044 s, 558 MB/s
Expected result:
Downloads the rest of the file.
Actual result:
When trying to update this with AppImageUpdate-x86_64.AppImage
via the GUI:
Of course, running this fails:
When trying to update the same partial AppImage via the CLI using the same AppImageUpdate-x86_64.AppImage
:
me@host:~$ Downloads/AppImageUpdate-x86_64.AppImage partial.AppImage
Checking for updates...
Parsing failed! Are you sure the file is an AppImage?
I'd like to spend some time (2+ days) on evaluating the current use of zsync2 with type 2 AppImages, i.e., compare how much has changed file wise (before calling mksquashfs
, that is), and the difference ratio that zsync2 calculates. I feel like it tends to download more data than what has actually changed, but I'd rather perform some measurements before speculating in any way.
User stories:
I as a user expect AppImageUpdate to download only the blocks for a file that is added between two releases.
I as a user expect AppImageUpdate to be able to download blocks for an entire file at max if that file has been changed, and nothing else.
I as a developer hosting releases of my application expect AppImageUpdate to minimize the traffic generated for updates.
At the moment, this is promised, and might be true, but it'd be nice to create a little, meaningful study on that, which we can show to people asking about this. Also, it's a great way to find potential for optimizations. And, as we plan to keep zsync2 as our core functionality (even when using alternatives to the classic server-client architecture, such as peer to peer networks (see AppImage/AppImageKit#175)), now seems to be the right time to investigate these issues.
Factors that might have to be optimized are unequal block sizes for zsyncmake2 and mksquashfs, compression after generation of the squashfs image (as far as I know, mksquashfs pads files to fill up the remaining bytes in the last block of a file), etc. Anything that could lead to equal files being stored in a way so that the hash sums for the occupied blocks are different, basically.
Rule of thumb for block sizes: mksquashfs block size >=
zsyncmake2 block size, mksquashfs block size mod
zsyncmake2 = 0, block sizes should be powers of 2.
To my knowledge, such measurements haven't been performed yet, or at least haven't been set up in a more scientific way, capturing and visualizing the results in a repeatable way.
I think we could e.g., use any random Qt application (bundled with linuxdeployqt and the exact same Qt version) with changes in the main binary only, or some CLI applications bundling just a few but rarely changing libraries.
The goal is to find potential optimizations in our use of squashfs which would decrease the aforementioned difference ratio to save on bandwidth. I think it should be possible to find an acceptable trade-off of higher file space versus more efficient updates.
TODO:
-j
), and some repository containing the AppDirs of which we generate AppImages for with appimagetool, eliminating external dependencies)When you update a CLI app (desktop file has Terminal=true
), then "Run app" should open the app in a terminal.
wget https://download.opensuse.org/repositories/home:/nandub:/MyAppImages/AppImage/aria2-latest-x86_64.AppImage
# Run AppImageUpdate GUI on the just downloaded AppImage
# "Run app" --> Nothing happens in the GUI
Note that this particular AppImage has True
instead of true
:
me@host:~$ /home/me/Downloads/aria2-1.24.0-1.1.Build12.15.glibc2.17-x86_64.AppImage --appimage-extract '*.desktop'
squashfs-root/aria2.desktop
me@host:~$ cat squashfs-root/aria2.desktop
[Desktop Entry]
Type=Application
Name=Aria2
Exec=aria2c
Icon=aria2
Categories=Network;Download Manager
Comment=aria2 is a lightweight multi-protocol & multi-source, cross platform download utility operated in command-line. It supports HTTP/HTTPS, FTP, SFTP, BitTorrent and Metalink
Terminal=True
me@host:~$ desktop-file-validate squashfs-root/aria2.desktop
squashfs-root/aria2.desktop: error: value "Network;Download Manager" for string list key "Categories" in group "Desktop Entry" does not have a semicolon (';') as trailing character
squashfs-root/aria2.desktop: error: value "True" for boolean key "Terminal" in group "Desktop Entry" contains invalid characters, boolean values must be "false" or "true"
me@host:~$ Qulo/AppImageUpdate-x86_64.AppImage /isodevice/Applications/Subsurface-x86_64.AppImage
AppImageUpdate version 1-alpha (commit 6fbc9e9), build 122 built on 2017-11-12 00:58:28 UTC
XOpenIM() failed
Checking for updates...
Fetching release information for tag "continuous" from GitHub API.
... done
Starting update...
Fetching release information for tag "continuous" from GitHub API.
Updating from GitHub Releases via ZSync
zsync2: Reading seed file: /isodevice/Applications/Subsurface-x86_64.AppImage
zsync2: Usable data from seed files: 32.641444%
zsync2: Renaming temp file
zsync2: Fetching remaining blocks
zsync2: Downloading from https://github-production-release-asset-2e65be.s3.amazonaws.com/2319498/25016776-c70a-11e7-82a9-201d837e0259?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20171112%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20171112T063052Z&X-Amz-Expires=300&X-Amz-Signature=be5edc9f6eeba986778e15c31dbb3ade0d1b75fb0f3cdeb5c220918b38550ee2&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3DSubsurface-4d04f7431-x86_64.AppImage&response-content-type=application%2Foctet-stream
zsync2: Verifying downloaded file
zsync2: checksum matches OK
zsync2: used 23252992 local, fetched 47984640
Update successful
CC AppImageKit, AppImageSpec, @probonopd, @adrianschroeter
Some projects like KDE or openSUSE use Mirrorbrain to publish their releases. To make it easier for such projects to add update information to their AppImages, we could add an update type supporting this platform.
Mirrorbrain differs from the random server by:
I already asked in #kde-sysadmin on Freenode about their infrastructure. My initial intention was to make them enforce HTTPS in their mirror infrastructure (in days of Let's Encrypt, this is a valid request), as that also solves the problem "people observing data stream can easily see what files are being requested" (because that's what people don't think of when talking about HTTPS), and obviously, people being too lazy to actually check the files they download, so having the integrity and (server) authenticity checks HTTPS enforces for those initial downloads would be awesome.
I think if we wanted to help them provide updatable AppImages from such an infrastructure, we should add a Mirrorbrain update information type which at least performs the update checks, because I think it Mirrorbrain can redirect automatically if one requests the right URL from the beginning.
Thinking about it again, IIRC, Mirrorbrain returns at least the hashes as HTTP headers, so, to achieve integrity checks based on these (because GPG signatures without verifying the keys aren't anything but pure hash-based integrity checks), we could make use of those by default, if available. This is primarily useful when downloading the .zsync
file from an unauthenticated server (after the redirect from the HTTPS enabled mirrorbrain master): When it's downloaded, check its integrity to make sure the right file has been downloaded. This is similar to how APT works -- they sign and verify the central file which contains hashes of all the files in the repository, and we do the same, as the .zsync
file contains integrity information (file hash) of the target file, hence one can ensure that the target file's integrity check is reliable after verifying the .zsync
file's integrity.
@adrianschroeter once said in the Mirrorbrain configs it was possible to serve certain files from the master server, but we can't/shouldn't rely on this to be configured correctly.
What do you guys think? Personally, I believe we have to improve this situation, especially since we're advertising [OBS], whose artifacts storage is regularly redirecting to HTTP-only servers (I think I posted that somewhere in the OBS repository, if anyone's curious, I don't have the link any more), and therefore we should at least fix one of the problems due to the lack of HTTPS.
Side note: I was analyzing the Kdenlive infrastructure initially, having me analyze the behavior. I verified this with openSUSE's installation because that service is hosting a large share of AppImages. Kdenlive don't provide updates yet, this will be subject of a second issue.
Let's use AppImageUpdate as the user-facing name for the user-visible GUI application. (You will never see "GUI" in the user-facing name of commercial applications; grandma doesn't know what that even means.)
appimageupdate
is the name of the command line tool, following the macOS convention that GUI applications are upper-case while command line tools are lower-case. If this distinction is not sufficient, we could also call the CLI tool appimageupdatetool
.
/isodevice/Applications/
is on a FAT32 partition.
me@host:~$ sudo /isodevice/Applications/AppImageUpdate-162-2720e18-x86_64.AppImage /isodevice/Applications/Quassel_Client-61fc76a-x86_64.AppImage
AppImageUpdate version 1-alpha (commit 2720e18), build 162 built on 2017-11-17 01:19:19 UTC
Checking for updates...
Fetching release information for tag "continuous" from GitHub API.
... done
Starting update...
Fetching release information for tag "continuous" from GitHub API.
Updating from GitHub Releases via ZSync
zsync2: Target file: /isodevice/Applications/Quassel_Client-773227a-x86_64.AppImage
zsync2: Reading seed file: /isodevice/Applications/Quassel_Client-61fc76a-x86_64.AppImage
rename: Invalid cross-device link
zsync2: Usable data from seed files: 80.501089%
zsync2: Renaming temp file
Update failed
Possibly it would be a good idea to have the temp file in the same directory as the target file?
Sparkle Framework on macOS shows these information windows:
But they have an extra feed (AppCast) for this kind of metadata, which is extra work to produce.
QGitHubReleaseAPI seems to be able to put together something similar from the information on GitHub:
A line introduced in 0d169e4 won't work when building out of source tree. The file build-appimages.sh
is meant to be used out of source however, that's why it is written with absolute paths. It even builds out of source in a RAM disk automatically, reducing disk IO and speeding up builds.
It should be moved to the appropriate location, which should fix local builds.
As fixed compile error.
ocs-manager imported AppImageUpdate repos when build time.
Two use cases should be covered:
The former has the higher priority, as @akiraohgaki needs this for OCS-Store.
The launch button can be clicked while the update is still running. It doesn't actually launch the AppImage, but it is kind of odd that it is clickable at all. It should be disabled until the update has finished.
When the file was already 100% downloaded and up to date and the user then tries to update it, a .zs-old
file is left behind:
me@host:~$ ls
appimaged-x86_64.AppImage Desktop Downloads Pictures
appimaged-x86_64.AppImage.zs-old Documents Music Videos
me@host:~$ rm appimaged-x86_64.AppImage.zs-old
me@host:~$ ls
appimaged-x86_64.AppImage Desktop Documents Downloads Music Pictures Videos
me@host:~$ Downloads/appimageupdate-x86_64.AppImage appimaged-x86_64.AppImage
Starting update...
Updating from generic server via ZSync
Update URL: https://github.com/AppImage/AppImageKit/releases/download/continuous/appimaged-x86_64.AppImage.zsync
zsync2: appimaged-x86_64.AppImage found, using as seed file
zsync2: Reading seed file: appimaged-x86_64.AppImage
zsync2: Usable data from seed files: 100.000000%
zsync2: Renaming temp file
zsync2: Fetching remaining blocks
zsync2: Verifying downloaded file
zsync2: checksum matches OK
zsync2: used 245760 local, fetched 0
100% done...
Update successful!
me@host:~$ ls
appimaged-x86_64.AppImage Desktop Downloads Pictures
appimaged-x86_64.AppImage.zs-old Documents Music Videos
https://github.com/jmacd/xdelta
What are its pros/cons vs. zsync?
This issue is created just for reference so that this is not forgotten.
At the moment, the gpg_check method is commented out as you can see in the current implementation:
gpg_check()
{
set +e
if [ "$(which gpg_currently_disabled)" != "" ] ; then
echo "GPG is installed; hence attempting signature verification"
ASC_URL=$( echo "${ZSYNC_URL}" | sed -e 's|.zsync|.asc|g' )
# curl -k -L -O -# "${ASC_URL}" && gpg --verify $(basename "${ASC_URL}") && echo "Signature verified"
else
echo "GPG is not installed; hence skipping signature verification" >&2
fi
set -e
}
After reading AppImage/AppImageKit#238, I came up with the following solution:
gpg_check()
{
set +e
if [[ "$(which gpg2)" != "" && "$(which validate)" != "" ]]; then
echo "GPG is installed; hence attempting signature verification"
validate "${ISO}"
if [[ $? -eq 0 ]]; then
echo "GPG signature verified"
else
echo "GPG verification failed. Please close and retry"
mv -f "${ISO}.zs-old" "${ISO}"
exit 1
fi
else
echo "GPG is not installed; hence skipping signature verification" >&2
fi
set -e
}
All that's needed is to include the validate
binary into the PATH so that appimageupdate
can run it. In my case, I created a self update-able AppImage adding those two plus appimageupdategui
and zsync_curl
.
AppImageUpdate works for non-executable AppImages, which is fine, but when you press Launch, also non-executable files are run. That's some kind of odd behavior, it should be changed to either changing the permissions of the AppImage so that the executable bit is set (I'd prefer this way) or showing an error message if it is not executable (the safer way).
me@host:~$ wget https://transfer.sh/Kljvv/appimageupdategui-x86_64.AppImage
me@host:~$ chmod a+x appimageupdategui-x86_64.AppImage
me@host:~$ /appimageupdategui-x86_64.AppImage appimageupdategui-x86_64.AppImage
Checking for updates...
Speicherzugriffsfehler
Looks like they have a very elaborate and well-thought-out solution in place, what could it offer over zsync for AppImageUpdate?
Using the tool as appimageupdatetool.AppImage <some appimage>
without objdump installed results in the following output:
Checking for updates...
sh: 1: objdump: not found
and an indefinite hang.
I tried to use appimageupdatetool-*.AppImage
from the commandline, but in a very non-standard way:
aiu-tool
.im
pointing to the real AppImage of ImageMagick.It all goes well until about 69%. Then it falls down with a chain of messages saying:
[....]
-1 returned
-1 returned
zsync2: failed to retrieve from ImageMagick-bc5b257-gcc-x86_64.AppImage, status -1
100.00% done (19.40 of 19.40 MiB)...
Update failed!
I'm aware that my AppImage should really carry a proper version information (which it doesn't do this yet) -- but I also like to "kick the tires" in order to test stuff.
The ImageMagick AppImage so far works as expected, even though the Travis build at the end said "Your build exited with 1." and I had to "force" the upload to the GitHub releases page.
However, I submit this issue because you may gain interesting insights into your tool when it has to deal with a possibly faulty AppImage.
I also ran aiu-tool --self-update
and ./aiu-tool ./aiu-tool
, which both worked, despite of using symlinks only.
The updated aiu-tool symlink meanwhile points to appimageupdatetool-180-d89753f-x86_64.AppImage, but this also fails on my ImageMagick AppImage. This most likely means that something is wrong with my ImageMagick AppImage and its .zsync file.
Files:
At the moment, the Vala GUI is just a wrapper for the Bash script, and it can be tricky to make such process wrappers reliable and stable. It works, but to make it become the official updater for AppImages and reach a larger user base, it needs to be improved.
I experience problems with my Red Eclipse AppImages due to this. They are very large, and the UI doesn't really work well together with my AppImages and zsync files. It gets stuck, gets into inconsistent states etc. I fixed one of these states already, but with the current code base, it's quite a PITA due to how it works at the moment.
I think to solve this properly, we have to directly use some kind of zsync_curl API instead of relying on the bash script. This probably improves the experience a lot, and should make the whole thing more reliable and stable. I believe the AppImage project would benefit a lot from such a tool.
My approach would be to write an AppImageUpdate C(++) framework based on the zsync-curl source that has two or more frontends, a CLI and a GUI application. This would make the bash script obsolete, as a side effect.
While working on this, we might also find a solution to integrate it within AppImages directly in a better way. Ideally, you'd specify a url: "https://..."
key in a YML recipe, and the rest happened automagically.
It's probably a good chance to improve my Vala knowledge, so I'd like to contribute. But this is probably going to take some time, I'm busy for the next weeks. If someone else finds this useful though, I'd try to help as good as I can.
This functionality could be handled by zsync_curl as well
As the user, I would like to know
As we want AppImage(Self)Update to look and feel as native as possible on every application platform, we might prefer to work with existing Qt, Gtk+, Electron, etc. updater projects rather than writing all of these GUIs and integrations (for self-updating) on our own.
Objective is to achieve a user experience roughly similar to the Sparkle Framework for macOS, but using AppImage and binary delta updates, for Linux.
Let's collect a list of promising candidates here.
Qt
Electron
Python
Windows
Cross-platform
Currently, the "run" logic is available in the GUI application only, but it might be a good idea to move it out to a module shared between all applications. It might even make sense to move it to the Updater
class, to allow other libraries to make use of its functionality.
Related to #31.
I think it might make sense to add the travis build ID, a monotonically increasing unsigned integer, to the AppImage filenames. This would allow users to see which binary is newer by comparing those numbers by size, which is something the Git commit hash doesn't provide.
CC @probonopd.
The result of this discussion will be applied to linuxdeployqt's and AppImageKit's AppImage releases, too.
Continuation of the discussion in AppImage/AppImageKit#133.
It would be nice to offer some tools allowing AppImages to check for updates and eventually update themselves, which can be used in special cases where this feature is required.
Therefore, a binary called AppImageSelfUpdate
(a modified variant of the new GUI AppImageUpdate
) has been implemented, making self-updates as easy as calling a subprocess.
The idea is that AppImageSelfUpdate
can configure itself from the environment so that the binary just needs to close itself and call that binary using an exec()
call.
A feature that is missing at the moment is to offer AppImages an "update check" using the algorithms implemented in AppImageSelfUpdate
anyway.
We are working on such a concept at the moment in a wiki page here:
https://github.com/AppImage/AppImageUpdate/wiki/Self-updating-AppImages. This page is going to be updated with a more final concept soon.
Self-updating AppImageUpdate does not work as expected when using an older version of AppImgageUpdate (from earlier today) - I can see there is a newer pre-release available on GitHub Releases (since 26 minutes or so).
The update information inside this AppImage is gh-releases-zsync|AppImage|AppImageUpdate|continuous|AppImageUpdate-*x86_64.AppImage.zsync
Move permissions logic from GUI to libappimageupdate
so that users of the library don't have to deal with this in their own code:
Required for @probonopd's UI proposal in https://github.com/opendesktop/ocsstore/issues/8#issuecomment-340486795 (second screenshot) to work.
It should be possible to query the total file size of the new file to be queried using the Updater API. This is fairly easy to implement, and, in combination with the updater.progress()
method, allows for displaying the amount of data that is available locally and/or has been downloaded already.
Despite applying AppImage/AppImageKit#364 in 1ffe5aa we still get
when trying to update an AppImage in a location that the current user has no write rights for (e.g., /opt
).
Perhaps @abhay44 can have a look?
Let's add some tests on Travis CI, e.g.,
A critical bug. objdump
command is missing, hence AppImageUpdate fails to parse the update information.
The call to objdump
needs to be replaced with the ELF related code @probonopd wrote for AppImageKit asap.
Quite a few times I've had to go to the documentation on the web to check for the proper argument to use. I think it would be useful if we had an argument appimage-help
that shows you all the possible commands you can use for that particular AppImage.
What do you think?
Can be reproduced with AppImageUpdate version 1-alpha (commit 6fbc9e9), build 122 built on 2017-11-12 00:58:28 UTC like this:
$HOME/Downloads
(where the browser puts it by default)$HOME/Downloads
, next to the old version$HOME
AppImage contains:
me@host:~$ find /tmp/.mount_AppImaZ67Gwk/ | grep icons
(...)
/tmp/.mount_AppImaZ67Gwk/usr/share/icons/hicolor/scalable/AppImage.svg
Desktop file says:
me@host:~$ cat /tmp/.mount_AppImaZ67Gwk/AppImageUpdate.desktop
[Desktop Entry]
Name=AppImageUpdate
Type=Application
Icon=AppImage
Exec=AppImageUpdate
Probably need to set the _NET_WM_ICON property of toplevel window? In Qt, QApplication::setWindowIcon()
is known to do the trick.
It would be cool if we could somehow find out the version of the locally downloaded AppImageUpdate and appimageupdatetool
.
me@host:~$ Downloads/linuxdeployqt-continuous-x86_64.AppImage --appimage-updateinformation
gh-releases-zsync|probonopd|linuxdeployqt|latest|linuxdeployqt-_*-x86_64.AppImage.zsync
me@host:~$ Downloads/AppImageUpdate-x86_64.AppImage Downloads/linuxdeployqt-continuous-x86_64.AppImage
AppImageUpdate version 1-alpha (commit 513949c), build 108 built on 2017-11-04 00:22:04 UTC
Checking for updates...
Could not find update information in the AppImage! Please contact the author to embed update information!
Investigate Beaker and Dat as a possible way to marry p2p distribution/hosting with binary delta updates.
https://twitter.com/mafintosh/status/925258772406206464
Dat supports variable chunking so you can use the chunker best fitted for your needs. Diffing is not yet impl to focus on basic features 1st
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.