archlinux / asp Goto Github PK

If $ASPROOT (~/.cache/asp) has been purged for some reason, asp update PACKAGE does not recreate the local branch for the pkgbase. This creates a problem because any existing asp checkouts become broken until another (pointless) checkout has been made:

$ asp checkout linux 
remote: Enumerating objects: 9771, done.
remote: Counting objects: 100% (1912/1912), done.
remote: Compressing objects: 100% (781/781), done.
remote: Total 9771 (delta 1583), reused 1210 (delta 1131), pack-reused 7859
Receiving objects: 100% (9771/9771), 2.28 MiB | 2.20 MiB/s, done.
Resolving deltas: 100% (3994/3994), done.
From https://github.com/archlinux/svntogit-packages
 * branch            packages/linux -> FETCH_HEAD
 * [new branch]      packages/linux -> packages/packages/linux
Cloning into 'linux'...
done.


$ (cd ~/.cache/asp; git --no-pager branch)
  packages/packages/linux

$ (cd ~/.cache/asp; git --no-pager branch --all)
  packages/packages/linux
  remotes/packages/packages/linux

$ (cd linux; git pull)                    
Already up to date.

After purging the cache:

$ rm -rf ~/.cache/asp

$ asp update linux  
remote: Enumerating objects: 9771, done.
remote: Counting objects: 100% (1912/1912), done.
remote: Compressing objects: 100% (781/781), done.
remote: Total 9771 (delta 1583), reused 1210 (delta 1131), pack-reused 7859
Receiving objects: 100% (9771/9771), 2.28 MiB | 8.86 MiB/s, done.
Resolving deltas: 100% (3994/3994), done.
From https://github.com/archlinux/svntogit-packages
 * branch            packages/linux -> FETCH_HEAD
 * [new branch]      packages/linux -> packages/packages/linux

$ (cd ~/.cache/asp; git --no-pager branch --all)
  remotes/packages/packages/linux

$ (cd ~/.cache/asp; git --no-pager branch)      

$ (cd linux; git pull)
fatal: couldn't find remote ref refs/heads/packages/packages/linux

$ (mkdir tmp; cd tmp; asp checkout linux)
mkdir: created directory 'tmp'
Cloning into 'linux'...
done.

$ (cd linux; git pull)                   
Already up to date.

Due on how is build if someone run list-all any package not belong to [community] is prefixed as package

packages/zlib
packages/zsh
packages/zvbi
packages/zziplib
community/0ad
community/0ad-data
community/4ti2
community/6tunnel
community/9base

this dificult indentify from which repo belong a package and could cause confusion.

Should be possible to make it distingish and properly show [code] [extra] [comunity] [multilib] like abs used to do?

The git protocol provides no authentication and thus is vulnerable to man-in-the-middle attacks. git.archlinux.org supports cloning via HTTPS so please use that by default.

This isn't a bug and perhaps not even a feature request, but I think it may be useful in the context of what asp brings to those hacking on official packages, and my hope that this is the right place to document it.

I'm trying to establish a workflow to deal with package updates, particularly those built from custom PKGBUILDs. The AUR lends itself to this - you can just fork the git repo, make your changes there and merge in any updates to the PKGBUILDs as they happen.

The official PKGBUILDs don't lend themselves to this as easily, mainly because they all live in a single repo. asp helps with this by essentially checking out locally those branches mapped to packages. However the layout (trunk and repo folders) as well as the location (local) make it difficult to integrate into the same workflow as would be used (by me) for AUR packages. I feel that the AUR layout works well (and hear grumblings about it being where the official repo of PKGBUILDs is heading), so I've tried to figure out a way to reproduce that layout, so that I can fork off of it like I do with the AUR. This is what I've come up with.

First to start:

#clone the first package via its branch to save downloading the whole repo
git clone -b packages/termite --single-branch https://github.com/archlinux/svntogit-community.git
cd svntogit-community
#should already be here but just in case
git checkout packages/termite
#use a magic command to create a new branch with just the contents of trunk
git subtree split -P trunk/ -b trunks/termite
#push it somewhere, rename the branch if required
git push http://git-server/arch-packages/termite.git trunks/termite:master

Later to update:

git checkout packages/termite
# pull should fast forward
git pull 
#the same magic command will update the existing branch
git subtree split -P trunk/ -b trunks/termite
#push it to the same place.
git push http://git-server/arch-packages/termite.git trunks/termite:master

To add further packages:

#register the new package and fetch it
git remote set-branches --add origin packages/fish
git fetch
#check it out
git checkout packages/fish
#then the same magic command
git subtree split -P trunk/ -b trunks/fish
#and the same push
git push http://git-server/arch-packages/fish.git trunks/fish:master

This obviously doesn't use asp as it's doing something different, but should be easily scriptable and importantly remain idempotent. I'm not sure if asp itself would be the right tool to offer similar subtree functionality, but it would be cool to have something that abstracts away some of the internals of the svntogit repo.

Would like to use tig for viewing the PKGBUILD commit log (asp difflog). Currently I use a wrapper script for this, but it doesn't cope well with e.g. split packages (which asp already handles internally).

tig accepts more or less the same CLI syntax as git log, e.g.

tig "$remote/packages/$pkgname" -- trunk/

Hey there,

I'm currently trying to figure out if it's possible to save bandwidth by modifying this specific line: https://github.com/falconindy/asp/blob/5d64421cd9538cce28b9f554e2116495df9be49f/package.inc.sh#L170

Currently, the line does a git clone --single-branch --branch ... behind the scenes and I'm wondering if a --depth=1 would also be enough as I would probably not want to revert the history and this would save me a shitload of bandwidth on my third-country-like connected machines.

So I guess my questions are:

• Is a full git history necessary and used somewhere else by asp?

I was wondering what was the purpose/usecase of these lines.
if pkgbase=$(archweb_get_pkgbase "$1") && package_find_remote "$pkgbase" "$2"; then log_info '%s is part of package %s' "$1" "$pkgbase" pkgname=$pkgbase return 0 fi
Obviously this serves a very specific purpose, apparently dealing with package names which somehow have a related 'pkgbase'. what are the usecases for this line, which seems like an special case.

Also, I'm not asking just because, I think I caught a bug in that function.
When this block gets executed without a network connection available, and pkgname does NOT exist, pkgbase will remain empty-valued and pkgname(global) will get overwritten by '', which in turn will return the message:
$ asp checkout
error: unknown package:

Instead of,
$ asp checkout
error: unknown package:
If you're busy I can write a patch, but I'd need to know what is the usecase of that section of code.

61.0.3163.100 is being exported while 62.0.3202.94 is available in repos.

Hello,

I really like this tools' idea, so I have begun reading the code. I have noticed that the command checkout exists according to the manual and code, but not according to the usage() in asp.in. Could you update that ?

Have a great day

asp export community/python-tomlkit results in:

error: package 'python-tomlkit' not found in repo 'community-x86_64'

python-tomlkit only has a 'repos/trunk' and a 'repos/community-any'

Same happens for 'extra' (example 'asp export extra/perl-http-date').

Hello,

I was reading and playing with asp and noticed you only list short options for options (-h, -v) but there is no long option.
Is there a possibility to add a long options too like the traditional (that I keep imputing on asp) --help or --version?

some users cof prefer the long options other for muscle-memory, scripts, not mistake, consistency.
even pacman have long and short so it will be on line with what is on Arch.

Note: also https://git.archlinux.org/still show abs as official arch linux project.

Hello,
please add the option to choose between http or git protocol. This is useful for people behind a http proxy which blocks git protocol.

Regards

$ asp export mpv
error: unknown package: mpv
$ asp export community/mpv
error: unknown package: mpv

At the top of asp I see ARCH_GIT_REPOS=(packages community), so I assume that fetching from community is supported. I've tried with other packages, and using checkout instead but no luck.

edit: it might be worth mentioning asp list-all yields packages from community

$ asp list-all | grep mpv
community/mpv

Excellent tool folks! Is there a reason why it is not in the A.U.R.? Would you mind if I worked at setting that up?

When trying to checkout or export with fresh ~/asp and current version (2f04d93):

$ asp export vim
fatal: Couldn't find remote ref vim
$ asp export packages/vim
fatal: Couldn't find remote ref vim

Today I just tried to do a

asp export gvim-python3

However, this outputs:

error: unknown package: gvim-python3

But

asp export vim

works okay. Now, the PKGBUILD of vim contains:

pkgbase=vim
pkgname=('vim-minimal' 'vim' 'vim-python3' 'gvim' 'gvim-python3' 'vim-runtime')

So you see the issue: asp only recognizes pkgbase. Now, it would be nice to be able to export other packages from package name.

Did you keep asp this way to be simple (feature) or it should have been able to export gvim-python3 (bug)?

When running asp checkout or export (I only tried those two) the mentioned error message shows up, and then asp prints error: unknown package: yadayada. A quick look into asp shows that in_array isn't defined anywhere.

'asp update' returns error message:

==> updating remote 'packages'
==> updating remote 'community'
fatal: '* master' is not a valid branch name

I found this issue when trying to resholve asp. It seems that asp is actually using GNU Tar, instead of bsdtar provided by libarchive as described in the depends array of the Arch package. A further digging shows that the dependency of the Arch package was introduced in archlinux/svntogit-packages@2cac577, which fixed FS#54507, but asp switched to GNU Tar in a72de1d and included this commit in the next release, while libarchive remains in the depends array.

There seems to be a bug with the branch names:

$ asp update
==> updating remote 'packages'
remote: Counting objects: 27, done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 27 (delta 4), reused 17 (delta 1)
Unpacking objects: 100% (27/27), done.
From git://git.archlinux.org/svntogit/packages
 * branch            packages/gcc    -> FETCH_HEAD
 * branch            packages/mutt   -> FETCH_HEAD
 * branch            packages/quagga -> FETCH_HEAD
   1426092..4e7639e  packages/gcc    -> packages/packages/gcc
   6c4f5b8..21f5bf7  packages/mutt   -> packages/packages/mutt
   1cdcf4b..b2e3107  packages/quagga -> packages/packages/quagga
==> updating remote 'community'
remote: Counting objects: 104, done.
remote: Compressing objects: 100% (75/75), done.
remote: Total 104 (delta 48), reused 64 (delta 18)
Receiving objects: 100% (104/104), 44.71 KiB | 8.94 MiB/s, done.
Resolving deltas: 100% (48/48), completed with 3 local objects.
From git://git.archlinux.org/svntogit/community
 * branch            packages/cowfortune     -> FETCH_HEAD
 * branch            packages/linux-hardened -> FETCH_HEAD
 * branch            packages/nrpe           -> FETCH_HEAD
 * branch            packages/python-magic   -> FETCH_HEAD
   b403d78..ebfb078  packages/linux-hardened -> community/packages/linux-hardened
   f4a03ef..4be6dba  packages/nrpe           -> community/packages/nrpe
fatal: 'community/packages/cowfortune?[m' is not a valid branch name.
fatal: 'community/packages/linux-hardened?[m' is not a valid branch name.
fatal: 'community/packages/nrpe?[m' is not a valid branch name.
fatal: 'community/packages/python-magic?[m' is not a valid branch name.
fatal: 'packages/packages/gcc?[m' is not a valid branch name.
fatal: 'packages/packages/mutt?[m' is not a valid branch name.
fatal: 'packages/packages/quagga?[m' is not a valid branch name.

If I download packages via repository/package format I expect to get the exact same PKGBUILD that was used for the package in the repos and not the git/svn head.

However if we use fdupes as example i get:

[arch@arch asp]$ asp export community/fdupes
==> exporting fdupes:repos/community-x86_64
[arch@arch asp]$ pacman -Sii fdupes | tail -n 3
SHA-256 Sum     : 3582304aae42f6bdaa2af4ed2e11ebedf4ed830a8fa790f46108b4804f4678c5
Signatures      : 1D1F0DC78F173680

[arch@arch asp]$ sha256sum fdupes/PKGBUILD 
f5b90cc367b3574aff4d254eff08f025824d80ed996c4a6ece133fcfe65a8b79  fdupes/PKGBUILD

One hash used "SKIP" so I fixed it manually in trunk:
https://git.archlinux.org/svntogit/community.git/commit/trunk/PKGBUILD?h=packages/fdupes&id=dec317ec364d8aec33b9a0637b50b02bae240ae1

However the initial PKGBUILD that was used to build fdupes was the old one. Asp should download this one and not my fixed PKGBUILD (otherwise i should have bumped the release variable).

I wrote a small bash script that can search for PKGBUILDs in older versions. I only tried this myself on the whole package/community git, but it takes way too long to search through the whole history. I never tried that with branched, as I found asp. I though asp would solve this issue.

#!/bin/bash

CHECKSUM=$1
FILE=$2
ALGO="sha256"
LIMIT="1000000000000000000"

if [[ -z "$CHECKSUM" || -z "$FILE" ]]; then
    echo "Usage: $0 ${ALGO} file"
    exit 1
fi

# Check if valid git repo
ROOT=$(git rev-parse --show-toplevel)
if [[ $? -ne 0 ]]; then
    echo "Not a valid git repo."
    exit 1
fi
cd "$ROOT"

# Check each revision for checksum, stop on first match
git log --pretty=%H -n "${LIMIT}" "${FILE}" \
    | xargs -n1 -P0 -iX bash -ec \
    "${ALGO}sum <(git show X:${FILE}) | grep -q ${CHECKSUM} && echo X && exit 255" \
    2>/dev/null

My final goal is to get the PKGBUILDs of all packages in the current repositories at the state when the packager also built it (not the trunk). Maybe if we switch to git we can later tag those branches and have a better version schema. This way we could make builds also better reproducible.