getsolus / eopkg Goto Github PK
View Code? Open in Web Editor NEWeopkg - Solus package manager implemented in python3
eopkg - Solus package manager implemented in python3
Currently, eopkg history -t
uses its own code-path which does not appear to support taking back packages that are present in the Local repo, making testing unnecessarily convoluted when rolling back to a previous state.
It might be smart to ensure that eopkg history -t
uses the same code-path as eopkg install
and eopkg upgrade
.
In addition, it might be smart to ensure that both normal *nix paths such as /var/lib/solbuild/local/eopkg-index.xml.xz' _and_ URI versions of the Local repo path
file:///var/lib/solbuild/local/eopkg-index.xml.xz` both work interchangeably.
Bonus points if eopkg automatically converts the local path version to the URI version?
... And fix shelflive
in said history section.
To enable easy updates and rescue of existing installs, @ReillyBrogan suggested that we attempt to build and distribute our python3 port of eopkg as a single binary compiled with nuitka, using the current system python3 package (3.10 as of this writing).
This issue is a tracker issue for that process, including describing how we plan to handle the situation where we have a system python3 update.
Related
$ sudo eopkg4-bin it --reinstall gzip --debug
DEBUG: InstallDB initialized in 0.0029439926147460938.
DEBUG: PackageDB initialized in 0.2265024185180664.
DEBUG: RepoDB initialized in 0.00019693374633789062.
Updates available, checking reverse dependencies of runtime dependencies for safety.
DEBUG: HistoryDB initialized in 0.002358675003051758.
DEBUG: ComponentDB initialized in 0.0008749961853027344.
DEBUG: RepoDB initialized in 0.0001385211944580078.
DEBUG: InstallDB initialized in 0.0014679431915283203.
DEBUG: PackageDB initialized in 0.1403815746307373.
digraph G {
}
Safety switch forces the upgrade of following packages:
brotli curl libidn2 libunistring
digraph G {
libunistring_32bit[ label = "libunistring-32bit(1.0,8)" ];
curl_devel[ label = "curl-devel(8.6.0,86)" ];
brotli_devel[ label = "brotli-devel(1.1.0,10)" ];
curl[ label = "curl(8.6.0,86)" ];
brotli_32bit[ label = "brotli-32bit(1.1.0,10)" ];
libidn2[ label = "libidn2(2.3.7,11)" ];
libunistring[ label = "libunistring(1.0,8)" ];
brotli[ label = "brotli(1.1.0,10)" ];
libunistring_32bit -> libunistring;
curl_devel -> curl;
brotli_devel -> brotli;
curl -> libidn2;
curl -> brotli;
brotli_32bit -> brotli;
libidn2 -> libunistring;
}
DEBUG: checking libunistring release 8
System error. Program terminated.
<class 'Exception'>: Repo item
not found
Please use 'eopkg help' for general help.
Traceback:
File "/tmp/onefile_103306_1707663789_826088/eopkg-cli", line 73, in <module>
File "/tmp/onefile_103306_1707663789_826088/eopkg-cli", line 70, in main
File "/tmp/onefile_103306_1707663789_826088/pisi/cli/pisicli.py", line 131, in run_command
File "/tmp/onefile_103306_1707663789_826088/pisi/cli/install.py", line 165, in run
File "/tmp/onefile_103306_1707663789_826088/pisi/api.py", line 67, in wrapper
File "/tmp/onefile_103306_1707663789_826088/pisi/api.py", line 500, in install
File "/tmp/onefile_103306_1707663789_826088/pisi/operations/install.py", line 53, in install_pkg_names
File "/tmp/onefile_103306_1707663789_826088/pisi/operations/install.py", line 359, in plan_install_pkg_names
File "/tmp/onefile_103306_1707663789_826088/pisi/pgraph.py", line 38, in add_dep
File "/tmp/onefile_103306_1707663789_826088/pisi/db/packagedb.py", line 96, in get_package
File "/tmp/onefile_103306_1707663789_826088/pisi/db/packagedb.py", line 213, in get_package_repo
File "/tmp/onefile_103306_1707663789_826088/pisi/db/itembyrepo.py", line 40, in get_item_repo
This actually feels fairly straight forward.
Reference: https://www.freedesktop.org/software/systemd/man/latest/systemd.offline-updates.html
/system-update > /var/cache/eopkg/packages/
/system-update
and boots into a minimal state and runs a systemd file to do the upgradeThe systemd file needs to:
Notes:
I'm looking into what the systemd service file looks like soon.
eopkg check
thing, as well as ensuring that eopkg doesn't do anything stupidThe issue:
pkg B v2 is attempting to install /usr/bin/foo but /usr/bin/foo already exists on
the system from pkg A v1
Currently the upgrade operations completely ignores file conflicts and the install operation has a ugly workaround for this edge case. This edge case can be surprisingly common for example functionality getting split off into it's own library.
PiSi has no knowledge of the file lists for remote pkgs and adding support for that is outside of this scope. However, we can
If we encounter a file conflicts when attempting to install a package
files.xml
file to see if the conflicting file(s) still exists in the latest version of the package (the package should already be downloaded at this point)This will make it easier to compare versions.
The following packages will be upgraded:
Please add back bash/fish/zsh completion. So the cli experience will be comparable with eopkg
. Is this even possible with the way eopkg4-bin
is shipped?
Thanks you.
The output of eopkg4-bin --help
is missing options, it seems the list is truncated after "help"
eopkg4-bin
❯ eo4 --help
Usage: eopkg4-bin [options] <command> [arguments]
where <command> is one of:
add-repo (ar) - Add a repository
autoremove (rmf) - Remove eopkg packages
blame (bl) - Information about the package owner and release
build (bi) - Build eopkg packages
check - Verify installation
clean - Clean stale locks
configure-pending (cp) - Configure pending packages
delete-cache (dc) - Delete cache files
delta (dt) - Creates delta packages
disable-repo (dr) - Disable repository
enable-repo (er) - Enable repository
fetch (fc) - Fetch a package
help (?) - Prints help for given commands
Use "eopkg4-bin help <command>" for help on a specific command.
Options:
--version show program's version number and exit
-h, --help show this help message and exit
eopkg
❯ eopkg --help
Usage: eopkg [options] <command> [arguments]
where <command> is one of:
add-repo (ar) - Add a repository
autoremove (rmf) - Remove eopkg packages
blame (bl) - Information about the package owner and release
build (bi) - Build eopkg packages
check - Verify installation
clean - Clean stale locks
configure-pending (cp) - Configure pending packages
delete-cache (dc) - Delete cache files
delta (dt) - Creates delta packages
disable-repo (dr) - Disable repository
enable-repo (er) - Enable repository
fetch (fc) - Fetch a package
help (?) - Prints help for given commands
history (hs) - History of pisi operations
index (ix) - Index eopkg files in a given directory
info - Display package information
install (it) - Install eopkg packages
list-available (la) - List available packages in the repositories
list-components (lc) - List available components
list-installed (li) - Print the list of all installed packages
list-newest (ln) - List newest packages in the repositories
list-pending (lp) - List pending packages
list-repo (lr) - List repositories
list-upgrades (lu) - List packages to be upgraded
rebuild-db (rdb) - Rebuild Databases
remove (rm) - Remove eopkg packages
remove-orphans (rmo) - Remove orphaned packages
remove-repo (rr) - Remove repositories
search (sr) - Search packages
search-file (sf) - Search for a file
update-repo (ur) - Update repository databases
upgrade (up) - Upgrade eopkg packages
Use "eopkg help <command>" for help on a specific command.
Options:
--version show program's version number and exit
-h, --help show this help message and exit
The repository name is "package-management". The internal references are "pisi", given eopkg is a fork of pisi. This is confusing IMO. I'd like to rename everything into "eopkg" since that's how we've always called our package manager.
If you think backward reference matters, we can fork this very repository and call the new repo "eopkg", while archiving this one once the Python 3 port is over.
@GZGavinZhao @silkeh thoughts? This is low priority anyway.
When switching back and forth between eopkg4-bin and eopkg pisi py2, this error is quite common:
System error. Program terminated.
unsupported pickle protocol: 4
We should ensure that eopkg4 py3 uses the version 2 pickple protocol that eopkg py2 uses for compatibility purposes.
If you just run eopkg info somepkg then it'll you the "Installed Version" as well as "Package found in Solus Repository".
However, if you pass -f to info to show installed files it'll no longer show you the "Installed Version".
It looks like long stripe doesn't get displayed properly on eopkg4-bin
. Pay attention to "Summary" and "Description".
This is eopkg4-bin
:
alfisya ~ eopkg4-bin info font-firago-otf
font-firago-otf package is not installed
Package found in Solus repository:
Name : font-firago-otf, version: 1.000, release: 3
Summary : An independent Open Source typeface (OTF) ? FiraGO
Description : An independent Open Source typeface (Open Type) ? FiraGO
Licenses : OFL-1.1
Component : desktop.font
Dependencies :
Distribution : Solus, Dist. Release: 1
Architecture : x86_64, Installed Size: 23.07 MB, Package Size: 7.81 MB
Reverse Dependencies:
This is eopkg
:
alfisya ~ eopkg info font-firago-otf
font-firago-otf package is not installed
Package found in Solus repository:
Name : font-firago-otf, version: 1.000, release: 3
Summary : An independent Open Source typeface (OTF) — FiraGO
Description : An independent Open Source typeface (Open Type) — FiraGO
Licenses : OFL-1.1
Component : desktop.font
Dependencies :
Distribution : Solus, Dist. Release: 1
Architecture : x86_64, Installed Size: 23.07 MB, Package Size: 7.81 MB
Reverse Dependencies:
Ikey Doherty
Btw the current third party, is a deprecation window scheduled...?
Or could we do an absolute cheeky and yml convert it for temporary lease of life
And then teach ypkg to add the comar .py bits into the correct part of the archive
And drop eopkg build completely
Ie if it finds comar/*.py shove them in
Add relevant pspec pieces too
alternatively do the above and hack eopkg build to call out to ypkg/solbuild and clone the repo to a staging location,
So that the existing instructions continue to work sorta
Mad adhd mental dump ^ but may give some ideas rather than continuing to give life support for the old format which I never got around to fully murdering.
Some Linux distros let users know of required updates in case they wouldn't know for sure if a kernel module was present among the update candidates. For example, Ubuntu uses a red hue around the Power/Session icon.
I think that's an idea that could be useful to users, especially those who have the habit of making total/unfiltered updates (i.e. via sudo eopkg up
). These users might ignore that a kernel module was updated, if for instance they've let the update finish in the background. Notifying them rules out the possibility that an unchecked kernel update is still to be applied.
https://www.freedesktop.org/wiki/Software/systemd/inhibit/
Call the inhibitor lock over dbus during installation/upgrade operations to prevent the system sleeping, locking or shutting down.
sudo eopkg4-bin bi --ignore-safety pspec.xml
[sudo] password for harvey:
System error. Program terminated.
<class 'TypeError'>: '>=' not supported between instances of 'ImportError' and 'int'
Please use 'eopkg help' for general help.
Traceback:
File "/tmp/onefile_65661_1712092891_34243/eopkg-cli", line 82, in <module>
File "/tmp/onefile_65661_1712092891_34243/eopkg-cli", line 78, in main
File "/tmp/onefile_65661_1712092891_34243/pisi/cli/pisicli.py", line 131, in run_command
File "/tmp/onefile_65661_1712092891_34243/pisi/cli/build.py", line 209, in run
File "/tmp/onefile_65661_1712092891_34243/pisi/api.py", line 951, in build
File "/tmp/onefile_65661_1712092891_34243/pisi/atomicoperations.py", line 707, in build
File "/tmp/onefile_65661_1712092891_34243/pisi/operations/build.py", line 1778, in build
File "/tmp/onefile_65661_1712092891_34243/pisi/operations/build.py", line 268, in __init__
File "/tmp/onefile_65661_1712092891_34243/pisi/operations/build.py", line 425, in set_build_type
File "/tmp/onefile_65661_1712092891_34243/pisi/operations/build.py", line 678, in load_action_script
File "/tmp/onefile_65661_1712092891_34243/traceback.py", line 180, in print_exc
File "/tmp/onefile_65661_1712092891_34243/traceback.py", line 124, in print_exception
File "/tmp/onefile_65661_1712092891_34243/traceback.py", line 728, in __init__
File "/tmp/onefile_65661_1712092891_34243/traceback.py", line 406, in _extract_from_extended_frame_gen
If we deprecate third party now this is not a problem. All in favour? :P
Discovered here getsolus/packages#2196
The package.yaml description, which includes some non-ascii characters, gives a garbled output is the pspec.xml. From the polylith
package:
package.yml line:
The poly tool helps you manage your polylith projects which introduces the architectural concept of “service level building blocks” which can be combined like LEGO® bricks to build our services and systems.
pspec.xml out:
<Description xml:lang="en">The poly tool helps you manage your polylith projects which introduces the architectural concept of “service level building blocks” which can be combined like LEGO® bricks to build our services and systems.
The transition to Python 3 is pretty invasive, and on top of it we also removed the piksemel dependency in favor of lxml. To find out bugs and prove we're production-ready, here's our User Acceptance Test suite:
⚠️ That might break your Solus installation. Unless you're experienced with boot rescue and the terminal, experiment in a virtual machine instead of in your local machine directly.
We'll first create a fake system root in a directory of your choice. We'll call this directory ROOT
.
sudo ./eopkg-cli ur -D ROOT # This will automatically add the Solus stable repository, and will also synchronize it.
sudo ./eopkg-cli lc -l -D ROOT # List all components (AKA groups) in the Solus repository.
sudo ./eopkg-cli la -l -D ROOT # List all available packages in the Solus repository.
sudo ./eopkg-cli it -c system.base -D ROOT # Install the base system.
# The installation above will fail because of two missing symlinks. Fix them.
sudo ln -s lib64 ROOT/lib
sudo ln -s ../run ROOT/var/run
sudo ./eopkg-cli it -c system.base -D ROOT # This time it will complete.
Team members have created a package eopkg4
that can be installed in parallel with the Python 2 eopkg (the pisi
package). Note: software center is untested and may be broken if you install eopkg4
. This may include the ability to install and update 3rd-party packages! So you must be familiar with the terminal to test eopkg4
.
# Add the Grocery repo
sudo eopkg ar Grocery https://repo-cdn.gzgz.dev/eopkg-index.xml.xz
# Install the Python 3 eopkg!
sudo eopkg it eopkg4
As of the current date (2023-10-26), some manual steps are needed to use eopkg4
:
/var/lib/eopkg/info/
. There should be a file called files.db
in it.sudo mv files.db files.db.3
sudo eopkg4 rdb
to rebuild the files database.eopkg4
in place of eopkg
.Any thing eopkg
can do, eopkg4
should be able to do it too. If you notice any discrepancies, please open an issue or comment below.
sudo eopkg4 dc
to clear the cache./var/lib/eopkg/info
. You should have a db backup files.db.3
. Restore that backup by running cp -f files.db.3 files.db
.sudo eopkg rdb
to rebuild the files database.eopkg4 help
displays fewer commands than eopkg help
, but all of the commands are still there. They just don't appear there for some reason.If any issues arise in this process, feel free to comment below or open a new issue and ping @GZGavinZhao.
Python 2
$ sudo eopkg it xiphos --dry-run
Updates available, checking reverse dependencies of runtime dependencies for safety.
Following packages will be installed:
biblesync enchant16 gtkhtml libwebkit-gtk mesalib mesalib-32bit mesalib-dbginfo qt5-base sword xiphos
Total size of package(s): 216.64 MB
$ sudo eopkg it xiphos --dry-run --ignore-revdeps-of-deps
Following packages will be installed:
biblesync enchant16 gtkhtml libwebkit-gtk sword xiphos
Python 3
$ sudo eopkg4-bin it xiphos --dry-run
Updates available, checking reverse dependencies of runtime dependencies for safety.
System error. Program terminated.
Repo item
not found
Please use 'eopkg help' for general help.
Use --debug to see a traceback.
$ sudo eopkg4-bin it xiphos --dry-run --ignore-revdeps-of-deps
Following packages will be installed:
biblesync enchant16 gtkhtml libwebkit-gtk sword xiphos
Total size of package(s): 36.15 MB
From the python 2 version we can see that mesalib mesalib-32bit mesalib-dbginfo qt5-base
are also wanting to be installed but with python 3, adding these dependencies to the DAG, it chokes.
Very, similar to #42.
Traceback
Traceback:
File "/home/ninya/eopkg/./eopkg-cli", line 73, in <module>
main()
File "/home/ninya/eopkg/./eopkg-cli", line 70, in main
cli.run_command()
File "/home/ninya/eopkg/pisi/cli/pisicli.py", line 131, in run_command
self.command.run()
File "/home/ninya/eopkg/pisi/cli/install.py", line 165, in run
pisi.api.install(packages, ctx.get_option("reinstall") or reinstall)
File "/home/ninya/eopkg/pisi/api.py", line 67, in wrapper
ret = func(*__args, **__kw)
^^^^^^^^^^^^^^^^^^^^^
File "/home/ninya/eopkg/pisi/api.py", line 500, in install
return pisi.operations.install.install_pkg_names(packages, reinstall)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ninya/eopkg/pisi/operations/install.py", line 53, in install_pkg_names
G_f, order = plan_install_pkg_names(A)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ninya/eopkg/pisi/operations/install.py", line 359, in plan_install_pkg_names
G_f.add_dep(name, revdep)
File "/home/ninya/eopkg/pisi/pgraph.py", line 38, in add_dep
pkg2 = self.packagedb.get_package(depinfo.package)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ninya/eopkg/pisi/db/packagedb.py", line 96, in get_package
pkg, repo = self.get_package_repo(name, repo)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ninya/eopkg/pisi/db/packagedb.py", line 213, in get_package_repo
pkg, repo = self.pdb.get_item_repo(name, repo)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ninya/eopkg/pisi/db/itembyrepo.py", line 40, in get_item_repo
raise Exception(_("Repo item %s not found") % str(item))
```
eopkg info --xml nano
System error. Program terminated.
Invalid attribute name 'xml:lang'
Hi there.
I'm thinking to help Solus project with porting codes of eopkg to python3. I'm currently running in my fork
A long time ago I did it as well (porting Pardus2011's pisi code and I renamed it as inary)
I also wanna help you to support an independent distro.
I you wanna be familiar with that issue I can help you.
Bests regards.
piksemel doesn't work with Python 3, plus we want to stay as close as possible to the standard library. I began converting the codebase to use the xml
module. The codebase is pretty vast, I'm collecting here files to be processed.
These files contain the "piksemel" string:
Other files depending on piksemel without explicitly importing it:
I'm trying to assemble a package. There are no obvious problems with the assembly. I get two packages at the output. Trying to install - I get a batch manager error.
name : freetds
version : 1.4.6
release : 1
source :
- https://www.freetds.org/files/stable/freetds-1.4.6.tar.bz2 : 813802a1c6bc02fe1696b6ea31aa535225719777736b5bfc23a3a17858956ac0
homepage : https://www.freetds.org
license :
- GPL-2.0-or-later
- LGPL-2.0-or-later
component : database
summary : Tabular Datastream Library
description: |
Library for accessing Sybase and MS SQL Server databases
builddeps :
- unixodbc-devel
- pam-krb5
- pkgconfig(readline)
- pkgconfig(openssl)
setup : |
./configure --prefix=/usr \
--sysconfdir=/etc/freetds \
--enable-msdblib \
--enable-krb5 \
--with-unixodbc=/usr \
--with-openssl \
--enable-odbc
build : |
%make
install : |
%make_install
Installing the packages
# eopkg it freetds-1.4.6-1-1-x86_64.eopkg freetds-devel-1.4.6-1-1-x86_64.eopkg
Updates available, checking reverse dependencies of runtime dependencies for safety.
Unhandled internal exception.
Please file a bug report to <https://github.com/getsolus/package-management/>.
<class 'pisi.graph.CycleException'>: Encountered cycle ['freetds-devel', 'freetds']
Please use 'eopkg help' for general help.
Traceback:
File "/usr/bin/eopkg", line 78, in <module>
cli.run_command()
File "/usr/lib/python2.7/site-packages/pisi/cli/pisicli.py", line 138, in run_command
self.command.run()
File "/usr/lib/python2.7/site-packages/pisi/cli/install.py", line 109, in run
pisi.api.install(packages, ctx.get_option('reinstall') or reinstall)
File "/usr/lib/python2.7/site-packages/pisi/api.py", line 65, in wrapper
ret = func(*__args,**__kw)
File "/usr/lib/python2.7/site-packages/pisi/api.py", line 447, in install
return pisi.operations.install.install_pkg_files(packages, reinstall)
File "/usr/lib/python2.7/site-packages/pisi/operations/install.py", line 245, in install_pkg_files
order = G_f.topological_sort()
File "/usr/lib/python2.7/site-packages/pisi/graph.py", line 133, in topological_sort
self.dfs(lambda u: list.append(u))
File "/usr/lib/python2.7/site-packages/pisi/graph.py", line 101, in dfs
self.dfs_visit(u, finish_hook)
File "/usr/lib/python2.7/site-packages/pisi/graph.py", line 109, in dfs_visit
self.dfs_visit(v, finish_hook)
File "/usr/lib/python2.7/site-packages/pisi/graph.py", line 118, in dfs_visit
raise CycleException(cycle)
Shannon (stable)
Budgie
OS: Solus x86_64
Kernel: 6.5.11-263.current
No response
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.