whiteinge / ok.sh Goto Github PK
View Code? Open in Web Editor NEWA Bourne shell GitHub API client library focused on interfacing with shell scripts
License: BSD 3-Clause "New" or "Revised" License
A Bourne shell GitHub API client library focused on interfacing with shell scripts
License: BSD 3-Clause "New" or "Revised" License
Describe the bug
Add support for github deployments API
Set up GitHub Pages web site with usage & examples.
Describe the bug
A clear and concise description of what the bug is and what the expected
behavior is.
Include logs
ok.sh
with the -vvv
flag to look for any potential problems in theAuthorization
header first!ok.sh -x [...command here...] 2> ok-shdebug.log
and attach that file.Environment (please complete the following information)
ok.sh
release.
curl version.
jq version.
What operating system and version.
What shell.
If you are unsure, download
whatshell.sh and run
/usr/bin/env sh /path/to/whatshell.sh
.
Additional context
Add any other context about the problem here.
Couple things:
local
is a bashism and definitely not POSIX. It doesn't exist in compliant shells like ksh
, yash
, many others. Interestingly ok.sh is ok in oksh.
The shebang is not POSIX, either, strictly speaking. You can begin a script with a colon instead.
ok.sh: line 200: local: not found
ok.sh: line 201: local: not found
ok.sh: line 202: local: not found
ok.sh: line 203: local: not found
ok.sh: line 204: local: not found
ok.sh: line 205: local: not found
ok.sh: line 206: local: not found
ok.sh: line 208: local: not found
ok.sh: line 154: local: not found
ok.sh: line 156: local: not found
No command given. Available commands:
$
toybox
is much more compliant than busybox, fwiw../ok.sh list_issues nixos/nix
...
bash -x ./ok.sh -vvv list_issues nixos/nix 2>&1 | tee "$log_file"
..
<= Recv header, 153 bytes (0x99)
0000: Link: <https://api.github.com/repositories/3386088/issues?page=2
0040: >; rel="next", <https://api.github.com/repositories/3386088/issu
0080: es?page=33>; rel="last"
+ read -r hdr val
+ '[' Link = $'\r' ']'
+ val='<https://api.github.com/repositories/3386088/issues?page=2>; rel="next", <https://api.github.com/repositories/3386088/issues?p
age=33>; rel="last"'
+ case "$hdr" in
++ printf %s '<https://api.github.com/repositories/3386088/issues?page=2>; rel="next", <https://api.github.com/repositories/3386088/issues?page=33>; rel="last"'
++ awk '
BEGIN { RS=", "; FS="; "; OFS=": " }
{
sub(/^rel="/, "", $2); sub(/"$/, "", $2)
sub(/^[[:space:]]*</, "", $1); sub(/>$/, "", $1)
print "Link_" $2, $1
}'
+ headers='http_version: 1.1
status_code: 200
status_text: OK
Server: GitHub.com
Date: Thu, 22 Nov 2018 16:02:38 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 93284
Status: 200 OK
X-RateLimit-Limit: 60
X-RateLimit-Remaining: 55
X-RateLimit-Reset: 1542905660
Cache-Control: public, max-age=60, s-maxage=60
Vary: Accept
ETag: 9dc7bc9fda118e6e7066512ebc3da67d
X-GitHub-Media-Type: github.v3; format=json
Link_next: <https://api.github.com/repositories/3386088/issues?page=2
Link_last: <https://api.github.com/repositories/3386088/issues?page=33
realpath $(which awk )
/usr/bin/mawk
mawk -W version
mawk 1.3.3 Nov 1996, Copyright (C) Michael D. Brennan
messing about abit
i guess this is the expected result !!
printf %s '<https://api.github.com/repositories/3386088/issues?page=2>; rel="next", <https://api.github.com/repositories/3386088/issues?page=33>; rel="last"' | awk '
BEGIN { RS=", "; FS="; "; OFS=": " }
{
sub(/^rel="/, "", $2); sub(/"$/, "", $2)
sub(/^ *</, "", $1); sub(/>$/, "", $1)
print "Link_" $2, $1
}'
Link_next: https://api.github.com/repositories/3386088/issues?page=2
Link_last: https://api.github.com/repositories/3386088/issues?page=33
... so this appears a
problem with mawk /^[[:space:]]*</
I used to test this against Busybox installed to a local directory, sandboxed via env -i
. I know Busybox isn't exactly POSIX but it's pretty close, and keep the POSIX docs nearby when you're wondering about a particular flag or command. Anyway I can't think of a better way to get a near-POSIX environment set up. (Suggestions very welcome if someone has an idea here.)
It's been a while since I've done that and a few non-compatible things have snuck in. Busybox doesn't seem to have pr
(even though it's POSIX), and there's a syntax error (unclosed }
) somewhere in the file that Busybox's sh
is choking on that dash
is otherwise fine with.
So, either:
This function is inconsistent with the others. See the note in 0066212.
Create issues, create milestones, make a pull request, etc.
Describe the bug
As of 502011c, creating a repo with ok.sh create_repo foo
fails with:
martinmac:ok.sh brightm$ ./ok.sh create_repo foo
awk: newline in string OK_SH_ACCEPT
OK_SH_U... at source line 1
awk: newline in string OK_SH_ACCEPT
OK_SH_U... at source line 1
awk: newline in string OK_SH_ACCEPT
OK_SH_U... at source line 1
Client Error: 400 Bad Request
martinmac:ok.sh brightm$
Environment (please complete the following information)
ok.sh
release: 502011cProductName: Mac OS X
ProductVersion: 10.14.3
BuildVersion: 18D109
GNU bash, version 5.0.2(1)-release (x86_64-apple-darwin18.2.0)
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Additional context
I git bisected the problem to: 502011c.
Now ok.sh uses .netrc
file for authorization verification, This is available in most cases.
But in some special cases, this doesn't work very well. like Travis CI.
If Travis CI set a no display value environment, unable to generate .netrc
file. and then call some api will return Client Error: 401 Unauthorized
Is it possible to add an environment variable?
like this
https://github.com/whiteinge/ok.sh/blob/master/ok.sh#L697
curl -nsSig \
-H "Accept: ${OK_SH_ACCEPT}" \
-H "Content-Type: ${content_type}" \
${etag:+-H "If-None-Match: \"${etag}\""} \
${GITHUB_TOKEN:+-H "Authorization token ${GITHUB_TOKEN}"} \
${has_stdin:+--data-binary @-} \
${trace_curl:+--trace-ascii /dev/stderr} \
-X "${method}" \
"${path}"
Github's releases UI allows you to create new tags as part of creating a new release. Could this functionality also be exposed via the create_release
command?
Extract help-text out into generated (?) README file.
Small functions that present a nice interface on making a request and processing the output for easy use with other common CLI tools.
Allow passing in an ETag to do an If-None-Match request.
Needed to avoid a couple gotchas since the README is generated and the test suite is custom.
cc #28
I'm struggling to find information about this. What values can I inject?
Edit: I assume it's ordering. But it should be clarified in the docs if that is the case.
Please add meta data to ok.sh.
For example
#!/bin/bash
#
# One line description for this script.
#
# VERSION :semver
# DATE :2017-12-31
# URL :https://github.com/szepeviktor/debian-server-tools
# AUTHOR :Viktor Szépe <[email protected]>
# LICENSE :The MIT License (MIT)
# BASH-VERSION :4.2+
# DEPENDS :apt-get install package
# REFS :url
# DOCS :url
# SOURCE :url
# UPSTREAM :url
# LOCATION :/usr/local/sbin/script.sh
# OWNER :root:root
# PERMISSION :0755
# SYMLINK :/usr/local/sbin/alias.sh
# CRON-HOURLY :/usr/local/sbin/script.sh
# CRON-DAILY :/usr/local/sbin/script.sh
# CRON-WEEKLY :/usr/local/sbin/script.sh
# CRON-MONTHLY :/usr/local/sbin/script.sh | mailx -E -s "subject" root
# CRON.D :2 2<-->* * *<->root<-->/usr/local/sbin/script.sh
# CONFIG :~/.config/script/configuration
Some API calls use a single user/repo
arg and some use separate user
& repo
args. This is annoying to work with because you have to look up the docs for each. Should ok.sh normalize on one format?
These are a PITA to work with via the shell. How hard would it be to add a simple parser to ok.sh?
Pros:
Cons:
_filter
overrides can already use this syntax. ok.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/ok.sh b/ok.sh
index 4bf3b97..87407e4 100755
--- a/ok.sh
+++ b/ok.sh
@@ -1734,7 +1734,7 @@ list_releases() {
#
# Keyword arguments
#
- local _filter='.[] | "\(.name)\t\(.tag_name)\t\(.id)\t\(.html_url)"'
+ local _filter='.[] | [.name, .tag_name, .id, .html_url] | @tsv'
# A jq filter to apply to the return data.
shift 2
The basic (non-GitHub, non-HTTP-request) functionality should be friendly for unit testing. The HTTP requests should be covered by a couple basic integration tests.
I don't think I'll test the actual GitHub-centric functions because there's not a ton of value there if the other things are working right. Plus I'd have to send actual requests against the GitHub API to do so. Feedback welcome.
Likely a user problem and not an issue. I am not able to get any useful output from the tool.
Following the one example in the wiki
ok.sh list_repos octocat _filter='.[] | select(.fork == false) | .clone_url'
returns nothing (with or without the filter).
adding -v
+ curl -nsSi -H Accept: application/vnd.github.v3+json -H Content-Type: application/json -X GET https://api.github.com/users/octocat/repos?AWKLIBPATH=%2Fusr%2Flib%2Fx86%5F64%2Dlinux%2Dgnu%2Fgawk
+ set +x
ok.sh INFO: Follow_next: 1
ok.sh INFO: Remaining next link follows: 50
once I add -vvv
I get lots of json output but this is hardly usable.
I'm not sure if I am doing something wrong or if I misunderstood what the expected output should be. Any help would be appreciated.
I can create a release but when I try to upload a asset to release run
./scripts/ok.sh -v upload_asset $USER $REPO 13452549 javadoc.jar application/x-tar < ./build/libs/javadoc.jar
in run into
parse error: Invalid numeric literal at line 1, column 9
jq parse error; invalid JSON.
ok.sh INFO: Remaining next link follows: 50
./scripts/ok.sh: line 1652: upload_url: Upload URL could not be retrieved.
I use jq
version jq-1.5
Any ideas ?
When running upload_asset, jq reports an error:
$ ./ok.sh upload_asset https://uploads.github.com/repos/ArturKlauser/raspberrypi-rstudio/releases/23039592/assets?name=foo1 ok.sh mime_type=application/octet-stream
parse error: Invalid numeric literal at line 1, column 9
jq parse error; invalid JSON.
The error goes away when -q is specified.
Some digging shows that apparently the HTTP headers are piped into jq before the actual json data:
$ printf ':\ncat -\n' > myjq; chmod 755 myjq
$ OK_SH_JQ_BIN=./myjq ~/Downloads/ok.sh upload_asset https://uploads.github.com/repos/ArturKlauser/raspberrypi-rstudio/releases/23039592/assets?name=foo2 ok.sh mime_type=application/octet-stream
HTTP/1.1 201 Created
Cache-Control: no-cache
Content-Security-Policy: default-src 'none'
Content-Type: application/json; charset=utf-8
Etag: W/"75c6982651342ea7db8fa9949fc7b3b4"
Last-Modified: Wed, 22 Jan 2020 14:59:26 GMT
Strict-Transport-Security: max-age=31557600
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
X-Accepted-Oauth-Scopes: repo
X-Content-Type-Options: nosniff
X-Frame-Options: deny
X-Github-Media-Type: github.v3; format=json
X-Oauth-Scopes: repo
X-Xss-Protection: 1; mode=block
Date: Wed, 22 Jan 2020 14:59:26 GMT
Transfer-Encoding: chunked
X-GitHub-Request-Id: CE13:5182:68E29:8648C:5E28634C
{"url":"https://api.github.com/repos/ArturKlauser/raspberrypi-rstudio/releases/assets/17506513","id":17506513,"node_id":"MDEyOlJlbGVhc2VBc3NldDE3NTA2NTEz","name":"foo2","label":"","uploader":{"login":"ArturKlauser","id":4790123,"node_id":"MDQ6VXNlcjQ3OTAxMjM=","avatar_url":"https://avatars0.githubusercontent.com/u/4790123?v=4","gravatar_id":"","url":"https://api.github.com/users/ArturKlauser","html_url":"https://github.com/ArturKlauser","followers_url":"https://api.github.com/users/ArturKlauser/followers","following_url":"https://api.github.com/users/ArturKlauser/following{/other_user}","gists_url":"https://api.github.com/users/ArturKlauser/gists{/gist_id}","starred_url":"https://api.github.com/users/ArturKlauser/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ArturKlauser/subscriptions","organizations_url":"https://api.github.com/users/ArturKlauser/orgs","repos_url":"https://api.github.com/users/ArturKlauser/repos","events_url":"https://api.github.com/users/ArturKlauser/events{/privacy}","received_events_url":"https://api.github.com/users/ArturKlauser/received_events","type":"User","site_admin":false},"content_type":"application/octet-stream","state":"uploaded","size":60861,"download_count":0,"created_at":"2020-01-22T14:59:25Z","updated_at":"2020-01-22T14:59:26Z","browser_download_url":"https://github.com/ArturKlauser/raspberrypi-rstudio/releases/download/untagged-dc669712e7a6ef40f473/foo2"}
Include logs
ok.sh -x [...command here...] 2> ok-shdebug.log
and attach that file.Environment (please complete the following information)
ok.sh
release
Hi,
I'm having a problem using the upload_asset command with a github organization.
Here is the output of -vvv: https://hastebin.com/iquqitoqep
And this is the tail end of -vvv -j: https://hastebin.com/itutoribat
The first one has been edited heavily to remove the zip binary and ssl data, just for ease of reading.
Second one is just the end, mostly so you can see this:
/home/folk/.local/bin/ok.sh: 1561: local: "url":: bad variable name
Here is the relevant line from my lua script:
oksh("-vvv", "-j", "upload_asset", userId, repo, id, zip, "filename=" .. zip, "mime_type=application/zip")
userId = stellaris-mods
repo = test for the first paste, testing on the short 2nd paste
zip = folk_testmod.zip
My head is fried at the moment, hopefully I can take a closer look soon, and come back with some more information.
From reading the docs and source to ok.sh, I didn't see any way to create an issue? I'm looking to file an issue for every warning in my code; I think I would be able to do so with that.
raspberrypi2> jq
jq - commandline JSON processor [version ]
Usage: jq [options] <jq filter> [file...]
jq is a tool for processing JSON inputs, applying the..
....
Running any command:
qx@raspberrypi2 ~ $ ok.sh list_releases toyfoundry rebadge
/usr/local/bin/ok.sh: 399: local: |: bad variable name
With debug:
qx@raspberrypi2 ~ $ ok.sh org_repos toyfoundry
ok.sh DEBUG: Running command org_repos.
ok.sh DEBUG: Formatting 0 parameters as urlencoded
/usr/local/bin/ok.sh: 399: local: |: bad variable name
ok.sh DEBUG: Processing response.
ok.sh DEBUG: Response status is: 200 OK
ok.sh DEBUG: Outputting requested header 'status_code'.
ok.sh DEBUG: Outputting requested header 'status_text'.
ok.sh DEBUG: Outputting requested header 'Link_next'.
ok.sh INFO: Remaining next link follows: 50
ok.sh DEBUG: Command org_repos exited with 0.
Any ideas would be appreciated!
So I try to use create_release with these arguments: stellaris-mods testing 5.20170918 body="* Test\n * Test2"
And I get this result:
0000: {. "message": "Invalid request.\n\nFor 'properties/tag_name', 5
0040: .20170918 is not a string.",. "documentation_url": "https://dev
0080: eloper.github.com/v3".}.
Sent data:
=> Send data, 109 bytes (0x6d)
0000: {"tag_name": 5.20170918, "body": "* Updated readme.md with chang
0040: es from steam.bbcode.\n* Skogfisk eplepai."}.
== Info: upload completely sent off: 109 out of 109 bytes
I get the same result if I quote the argument on the command line, like:
stellaris-mods testing '5.20170918' body="* Test\n * Test2"
I have not tried using " to quote it, and I don't have time to do so right now (because I deleted the repository already and I need to go make dinner).
Would be helpful to display the error message from GitHub at the CLI. E.g.
{
"message": "Validation Failed",
"errors": [
{
"resource": "Milestone",
"code": "missing_field",
"field": "title"
},
{
"resource": "Milestone",
"code": "missing_field",
"field": "slug"
}
],
"documentation_url": "https://developer.github.com/v3/issues/milestones/#create-a-milestone"
}
On Kubuntu 14.04, when I try to create a repo, I get these errors:
$ octokit.sh create_repo testing_octokit
/home/user/bin/octokit.sh: 397: local: |: bad variable name
/home/user/bin/octokit.sh: 577: /home/user/bin/octokit.sh: [[: not found
I guess it is because the shebang is #!/usr/bin/env sh
and sh
does not support local
or [[
.
When I run it with bash
, I get another error:
$ bash octokit.sh create_repo testing_octokit
Client Error: 422 Unprocessable Entity
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.