dpb587 / gget Goto Github PK

Hi,
Thank you for this tool. I am trying to download any file with the .ipa extension using gget. It works when I use the example used in the readme but does not work with other repos. I suspect it has to do with the need for authentication using a token for private repos. I tried to add it in the same way fetch does, but it didn't work. Below is an example of what I get when I try to run gget:

./gget github.com/jerry977015/Aisa
gget: error: resolving ref: resolving latest: getting latest release: GET https://api.github.com/repos/jerry977015/Aisa/releases/latest: 404 Not Found []

Thank you for your help!

Experiment with the idea of allowing --version to receive an optional version constraint as a value. This could then be used for dependencies wanting to verify compatibility or check if they should reinstall gget.

$ gget --version
version 0.3.0

$ gget --version='>=0.4'
version 0.3.0
gget: version constraint not satisfied: >=0.4
exit code 1

$ if ! gget --version='>=0.3' >/dev/null 2>&1 ; then install_gget ; fi

The --cd option seems like it should specify an output directory, but instead seems to specify the resulting file's location:

$ gget github.com/pivotal-cf/pivnet-cli --executable '*linux*' --cd ~/c/go/bin
Found 1 file (10.6M) from github.com/pivotal-cf/[email protected]
X pivnet-linux-amd64-2.0.2 errored (processing step 2: renaming: rename /home/wjesse/c/go/.gget-679800536 /home/wjesse/c/go/bin: file exists)  
gget: error: transfers failed: pivnet-linux-amd64-2.0.2
$ gget github.com/pivotal-cf/pivnet-cli --executable '*linux*' --cd ~/c/go/bin/pivnet
Found 1 file (10.6M) from github.com/pivotal-cf/[email protected]
√ pivnet-linux-amd64-2.0.2 done (executable)

This might be because applySettings() is called before the refs are resolved.

Heya there, i just discovered this project and i like it very much, thank you for the effort. While using it, i was thinking that it would be greate to extend the "--show-resources" flag with an interactive mode --i. Something like:

gget github.com/dpb587/gget --show-resources --i

1. gget-0.2.0-darwin-amd64
2. gget-0.2.0-linux-amd64
3. gget-0.2.0-windows-amd64.exe

What do you want to gget?
Accepted user input: 1,2,3,1-2,*

Explanation:

1= download first resource
2= download second resource
3= download third resource
1-2= download 1 AND 2
*= download all

Another small cosmetic suggestion: what about to rename the --show-resources to a more easy and fast typing --list?

That's it. What do you think about it?

Currently, when REF is defaulted, only stable, non-pre-release, non-draft releases can be downloaded. For dev/test/opt-in cases, it'd be helpful to support fetching those assets.

I think pre-releases are currently only a GitHub feature and not supported in GitLab nor BitBucket.

Maybe a flag? Or special ref value? Does specifying pre-release imply only pre-release, or in addition to stable? Consider a future feature supporting semver parsing to detect rc/beta/alpha? Some ideas:

$ gget github.com/dpb587/gget --latest=stable --latest=pre-release
$ gget github.com/dpb587/gget --latest=any

$ gget github.com/dpb587/gget --stability=any

$ gget github.com/dpb587/gget@latest-pre
$ gget github.com/dpb587/gget@latest-unstable
$ gget github.com/dpb587/gget@latest-any

$ gget github.com/dpb587/gget --allow-pre-release
$ gget github.com/dpb587/gget --include-pre-release
$ gget github.com/dpb587/gget --github-pre-release

Currently checksum verification always happens if a checksum is detected for the file. In case of bugs, this should be something that can be disabled.

Additionally, a --verify-checksum-like option can support making it mandatory, or using a specific algorithm as well.

• auto (default) to verify if found
• required (missing value default) to force some checksum to be used (or fail)
• no/--no-verify-checksum to disable any verification
• algorithm (e.g. sha256) to require and verify a specific checksum

Consider how to represent "sha256 or better" as a value. Or a list of acceptable algorithms.

Consider a future, potential --verify-signature option and if those option name/value conventions will be similar.

I saw that you're planning to add support for sha256sums.txt and sha512sums.txt files in v0.4.0 so I thought it might be helpful to share some of the variations I've encountered while working on my script.

This is not meant to be a feature request to add support for these variations, just a list that you can use as a reference in case you do want to support more variations in gget.

*.md5
*.sha256
Example: https://github.com/genuinetools/reg/releases/tag/v0.16.1

*_sha256.txt
*_sha512.txt
Example: https://github.com/hairyhenderson/gomplate/releases/tag/v3.7.0

shasums
Example: https://github.com/gdrive-org/gdrive/releases/tag/2.1.0

CHECKSUM
Example: https://github.com/vmware-tanzu/velero/releases/tag/v1.4.0

MD5SUMS
SHA1SUMS
SHA256SUMS
Example: https://github.com/rclone/rclone/releases/tag/v1.52.1

md5sum.txt
sha256sum.txt
Example: https://github.com/docker/machine/releases/tag/v0.16.2

For the common case of downloading an asset from a published release, the current program flow makes two unnecessary API requests as it continues to resolve the release down to the commit. Restructure to avoid those extra calls since the initial response has all the data it technically needs.

$ go run . -vv github.com/dpb587/gget
INFO[0000] starting gget/0.0.0+dev (commit unknown; built 2020-06-21T15:28:16Z) 
INFO[0000] http: GET https://api.github.com/repos/dpb587/gget/releases/latest (status: 200 OK) 
INFO[0000] http: GET https://api.github.com/repos/dpb587/gget/git/refs/tags%2Fv0.4.0 (status: 200 OK) 
INFO[0000] http: GET https://api.github.com/repos/dpb587/gget/git/tags/96621bbdd27b21c618245448cce7a7e80fde39a7 (status: 200 OK) 
Downloading 3 files (33M) from github.com/dpb587/[email protected]
...

Potentially related to #13 since those extra calls provide metadata shown in that output.

There are some cases where it may be desirable to use variable strings in different executions of the same gget command, e.g. as part of a case statement in bash.

Take a repository with the following release resources:

logcli-linux-amd64.zip
logcli-linux-arm.zip
logcli-linux-arm64.zip
loki-linux-amd64.zip
loki-linux-arm.zip
loki-linux-arm64.zip
promtail-linux-amd64.zip
promtail-linux-arm.zip
promtail-linux-arm64.zip

If I format the statement like this

case "${APP_NAME}" in
  "logcli" | "loki" | "promtail")
    gget github.com/grafana/loki '*-linux-amd64*'
  ;;
esac

all three matching resources are downloaded by gget, as expected.

However, if I only want to download one of the three, it would be helpful to have support for variables in the RESOURCE definition, like this:

case "${APP_NAME}" in
  "logcli" | "loki" | "promtail")
    gget github.com/grafana/loki '${APP_NAME}-linux-amd64*'
  ;;
esac

If you run that statement with the current version of gget, it will exit with

gget: error: no resource matched: ${APP_NAME}-linux-amd64*

because the variable name is being parsed literally.

For non-TTYs, automated environments, and more basic output, add a --no-progress-like option which disables the live-updating download progress. When enabled, it should print a similar, but simpler result line after a download completes.

$ gget --no-progress github.com/dpb587/gget
Downloading 1 file from github.com/dpb587/[email protected]
gget-0.4.0-linux-amd64 (executable; sha256 OK)

This is different than the --quiet behavior which should continue to not print any results.

First, let me say that the new website for gget is gorgeous. I love the design, the color choices, the flow. Fantastic work! Having said that, there are two reasons why I think that the source for it should be public.

The first one is altruistic: because it would allow people to submit pull requests for it to improve/update the documentation for gget when necessary.

The second one is selfish: because this gave me the idea of making a website for my little script as well (primarily because .sh is an actual TLD, making https://ginstall.sh/ a valid domain name and how neat would that be?) and I would love to use that design as a template for it.

It would be helpful to report what reference was ultimately used when downloading. For example: when auditing historical logs; latest isn't explicit; versions are not implied in file names; seeing canonical user/repo capitalization; or seeing the expanded commit hash.

Might be a good place to include the number of files (and total download size, if known), too? Maybe with a non-present continuous verb? Probably should not be printed in --quiet mode. Review messaging with wget/curl output. Some ideas:

$ gget github.com/dpb587/gget
Downloading 3 files (34 MB) from github.com/dpb587/[email protected]...
...

$ gget github.com/dpb587/gget@36c7 README.md
Found github.com/dpb587/gget@36c7ac4821a5ce437303c3da920efbbadcb2a873
...

$ gget github.com/DPB587/GGET
Using github.com/dpb587/[email protected]
...

$ gget github.com/dpb587/gget
...
Downloaded 3 files from github.com/dpb587/[email protected]

The --show-ref and --show-resources cannot easily be used together (since they have different formats), and they cannot be used in the same command as a download. They're primarily intended for debugging and automation. Find a way to support a single command to download files while still having a method to capture metadata useful for external tools.

Maybe introduce a --print-info-like option with a machine-friendly output format like below. It would be helpful to be support sending this to a non-STDOUT/STDERR descriptor to continue to maintain a nice user output:

origin	resolved	github.com/dpb587/[email protected]
origin	service	github
origin	server	github.com
origin	owner	dpb587
origin	repository	gget
origin	ref	v0.4.0
metadata	tag	v0.4.0
metadata	commit	d75361c1316c6004cde13c04460c57552a839d57
resource	gget-0.4.0-darwin-amd64
resource	gget-0.4.0-linux-amd64
resource	gget-0.4.0-windows-amd64.exe

Probably requires introducing a dedicated --no-download-like option to allow external tools to continue gathering info before an actual download.

Should respect http_proxy, https_proxy, no_proxy. Maybe use https://golang.org/pkg/net/http/#ProxyFromEnvironment.

For enterprise instances and self-hosted services, it would be helpful to not need --service=. Should be able to fallback to auto-detecting based on server headers or API endpoint checks.

Came across this issue when running dpb587/gget in GitHub actions.

Error log:

panic: downloading delegatebot-0.9.0-linux-amd64: processing step 2: renaming: rename /tmp/gget-tmpfile-168433803 delegatebot: invalid cross-device link
goroutine 37 [running]:
github.com/dpb587/gget/cmd/gget.(*Command).Execute.func2(0xc00000e230, 0xc0001d6eb0, 0xa02260, 0xc000094020)
	/root/cmd/gget/command.go:248 +0x243
created by github.com/dpb587/gget/cmd/gget.(*Command).Execute
	/root/cmd/gget/command.go:241 +0x1193

It seems that the use of os.Rename() is the problem, see relevant code.