juliaweb / gitforge.jl Goto Github PK
View Code? Open in Web Editor NEWUnified interface for interacting with Git forges
License: MIT License
Unified interface for interacting with Git forges
License: MIT License
At no point in the docs does it mention that @paginate
is basically required to be used for all calls that might return more than one thing.
Possibly we should actually always return a iterator like @paginate
does .
But right now we don't.
I was wondering why get_pull_requests(forge, project_id)
was only returning 20 things even though i knew their are like 100 pull reqests on that repo.
Docs for get_pull_requests
never mentioned @paginate
.
No example of it being used was anywhere,
but I tried
@paginate get_pull_requests(forge, project_id)
and that returned all my pull requests.
using GitForge
lab_token = GitForge.GitLab.PersonalAccessToken("{REDACTED}")
lab_api = GitForge.GitLab.GitLabAPI(token=lab_token, url="https://{REDACTED}/api/v4")
repo = get_repo(lab_api, "{REDACTED_OWNER}", "{REDACTED_PROJECT}")
prs = get_pull_requests(lab_api, repo.val.id; state="opened", per_page=100, page_limit=100)
length(GitForge.value(prs)) # 7
prs = @paginate get_pull_requests(lab_api, repo.val.id; state="opened", per_page=100, page_limit=100)
arr = []
for x in prs
push!(arr, x)
end
length(arr) # 14
I'm really not sure what's going on here doing arr[1] == arr[8]
returns false, but looking at the values they're all the same. I haven't come across this w/ GitHub pagination of pull requests.
This issue is used to trigger TagBot; feel free to unsubscribe.
If you haven't already, you should update your TagBot.yml
to include issue comment triggers.
Please see this post on Discourse for instructions and more details.
If you'd like for me to do this for you, comment TagBot fix
on this issue.
I'll open a PR within a few hours, please be patient!
I don't like the Result
monad thing anymore. We can just pack the HTTP response and rate limiting info into an exception when things go bad and throw that instead.
Currently GitForge does not support Codeberg, therefore packages hosted there cannot be automatically registered via Registrator.jl.
It would be nice to have it supported.
I'm willing to help working on this.
Im guessing GitLab doesn't send these rate limit headers, and seeing this warning (https://github.com/JuliaWeb/GitForge.jl/blob/master/src/ratelimits.jl#L31) for every request seems somewhat misleading and noisy.
Is there a better solution?
It seems that with GitLab 12.1.2 the DateTime format has changed:
ERROR: ArgumentError: Unable to parse date time. Expected directive Delim(Z) at char 24
Stacktrace:
[1] macro expansion at /Users/omus/Development/Julia/1.0/usr/share/julia/stdlib/v1.0/Dates/src/parse.jl:104 [inlined]
[2] tryparsenext_core(::String, ::Int64, ::Int64, ::Dates.DateFormat{Symbol("y-m-dTH:M:S.sZ"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.Delim{Char,1}}}, ::Bool) at /Users/omus/Development/Julia/1.0/usr/share/julia/stdlib/v1.0/Dates/src/parse.jl:40
[3] macro expansion at /Users/omus/Development/Julia/1.0/usr/share/julia/stdlib/v1.0/Dates/src/parse.jl:150 [inlined]
[4] tryparsenext_internal at /Users/omus/Development/Julia/1.0/usr/share/julia/stdlib/v1.0/Dates/src/parse.jl:127 [inlined]
[5] parse(::Type{Dates.DateTime}, ::String, ::Dates.DateFormat{Symbol("y-m-dTH:M:S.sZ"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.Delim{Char,1}}}) at /Users/omus/Development/Julia/1.0/usr/share/julia/stdlib/v1.0/Dates/src/parse.jl:282
[6] Type at /Users/omus/Development/Julia/1.0/usr/share/julia/stdlib/v1.0/Dates/src/io.jl:432 [inlined]
[7] #read#48 at /Users/omus/.julia/packages/JSON2/J5VUB/src/read.jl:210 [inlined]
[8] (::getfield(JSON2, Symbol("#kw##read")))(::NamedTuple{(:dateformat, :datetimeformat),Tuple{Dates.DateFormat{Symbol("y-m-d"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'}}},Dates.DateFormat{Symbol("y-m-dTH:M:S.sZ"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.Delim{Char,1}}}}}, ::typeof(JSON2.read), ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{Dates.DateTime}) at ./none:0
[9] #read#32(::Base.Iterators.Pairs{Symbol,Dates.DateFormat,Tuple{Symbol,Symbol},NamedTuple{(:dateformat, :datetimeformat),Tuple{Dates.DateFormat{Symbol("y-m-d"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'}}},Dates.DateFormat{Symbol("y-m-dTH:M:S.sZ"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.Delim{Char,1}}}}}}, ::Function, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Union) at /Users/omus/.julia/packages/JSON2/J5VUB/src/read.jl:84
[10] (::getfield(JSON2, Symbol("#kw##read")))(::NamedTuple{(:dateformat, :datetimeformat),Tuple{Dates.DateFormat{Symbol("y-m-d"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'}}},Dates.DateFormat{Symbol("y-m-dTH:M:S.sZ"),Tuple{Dates.DatePart{'y'},Dates.Delim{Char,1},Dates.DatePart{'m'},Dates.Delim{Char,1},Dates.DatePart{'d'},Dates.Delim{Char,1},Dates.DatePart{'H'},Dates.Delim{Char,1},Dates.DatePart{'M'},Dates.Delim{Char,1},Dates.DatePart{'S'},Dates.Delim{Char,1},Dates.DatePart{'s'},Dates.Delim{Char,1}}}}}, ::typeof(JSON2.read), ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Union) at ./none:0
[11] macro expansion at /Users/omus/.julia/packages/JSON2/J5VUB/src/JSON2.jl:173 [inlined]
[12] #read#499(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(JSON2.read), ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Type{GitForge.GitLab.Commit}) at /Users/omus/.julia/packages/JSON2/J5VUB/src/JSON2.jl:154
[13] read at /Users/omus/.julia/packages/JSON2/J5VUB/src/JSON2.jl:154 [inlined]
[14] postprocess(::GitForge.JSON, ::HTTP.Messages.Response, ::Type{GitForge.GitLab.Commit}) at /Users/omus/.julia/dev/GitForge/src/request.jl:80
[15] #request#9(::Array{Pair{SubString{String},SubString{String}},1}, ::Dict{Any,Any}, ::Dict{Any,Any}, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::GitLabAPI, ::Function, ::GitForge.Endpoint) at /Users/omus/.julia/dev/GitForge/src/request.jl:162
[16] get_commit(::GitLabAPI, ::String, ::String, ::String) at /Users/omus/.julia/dev/GitForge/src/request.jl:122
[17] top-level scope at none:0
I'll note I'm seeing dates that look like "2019-08-02T16:33:01.000+00:00"
and I believe we're expecting "2019-08-02T16:33:01.000Z"
currently. I am reluctant to change the DateTime format as this would fix my issue but not for people using older versions of GitLab. Ideally we could support a few different formats or pick a format based upon the GitLab version.
Usage of functions is often subtly different for different forges, it would be nice to offer something over top to abstract that away for simple cases.
Since we're starting to maintain this again (for use in CompatHelper and eventually TagBot too), there are a couple of things that I would like:
Response{T}
design pattern that I used (c6bbc7b does this but I never got it into master for whatever reason)@paginate
, we return a lazy paginator by default. The logic of the pagination itself can probably be cleaned up too. (#18, #17)After Pkg.add
:
ERROR: LoadError: LoadError: LoadError: UndefVarError: defaultkwargs not defined
Stacktrace:
[1] getproperty(::Module, ::Symbol) at ./sysimg.jl:13
[2] top-level scope at /home/nkottary/.julia/dev/JSON2/src/JSON2.jl:63
[3] include at ./boot.jl:326 [inlined]
[4] include_relative(::Module, ::String) at ./loading.jl:1038
[5] include at ./sysimg.jl:29 [inlined]
[6] include(::String) at /home/nkottary/.julia/packages/GitForge/O3rZs/src/forges/GitHub/GitHub.jl:3
[7] top-level scope at none:0
[8] include at ./boot.jl:326 [inlined]
[9] include_relative(::Module, ::String) at ./loading.jl:1038
[10] include at ./sysimg.jl:29 [inlined]
[11] include(::String) at /home/nkottary/.julia/packages/GitForge/O3rZs/src/GitForge.jl:1
[12] top-level scope at none:0
[13] include at ./boot.jl:326 [inlined]
[14] include_relative(::Module, ::String) at ./loading.jl:1038
[15] include(::Module, ::String) at ./sysimg.jl:29
[16] top-level scope at none:2
[17] eval at ./boot.jl:328 [inlined]
[18] eval(::Expr) at ./client.jl:404
[19] top-level scope at ./none:3
in expression starting at /home/nkottary/.julia/packages/GitForge/O3rZs/src/forges/GitHub/users.jl:1
in expression starting at /home/nkottary/.julia/packages/GitForge/O3rZs/src/forges/GitHub/GitHub.jl:110
in expression starting at /home/nkottary/.julia/packages/GitForge/O3rZs/src/GitForge.jl:22
ERROR: Failed to precompile GitForge [8f6bce27-0656-5410-875b-07a5572985df] to /home/nkottary/.julia/compiled/v1.1/GitForge/A2yPU.ji.
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1197
[3] _require(::Base.PkgId) at ./loading.jl:960
[4] require(::Base.PkgId) at ./loading.jl:858
[5] require(::Module, ::Symbol) at ./loading.jl:853
I would like it if this worked.
julia> prs = get_pull_requests(forge, owner, repo)
ERROR: GitLabAPI has not implemented this function
Here's a (non-final) list of what I consider to be bare minimum to make this useful (slightly specialized with Registrator in mind):
For some endpoints, you can specify a project via ID (12345) or path (owner/name). It's preferred to use the latter since it's more compatible with GitHub. At some point, I should do a sweep of all implemented GitLab endpoints to make sure that we're exposing the path form whenever possible.
The test suite for this package is really lacking. Much of that is due to its dependence on HTTP responses, so we should mock those bits out with SimpleMock.
Pagination definitely needs a good test suite since the logic is pretty finicky.
I'm getting an error, which I suspect is because of this definition of keywordargs
:
Lines 13 to 20 in 1fd93e1
I believe keywordargs
should return a named tuple associating field names with keyword values but the definition does not have any field names in it.
The offending date string was not printed in the log but the format string was: "yyyy-mm-dd\THH:MM:SS.s" which is not the GitHub default date string: "y-m-dTH:M:S\Z"
Here's the error message:
ERROR: GitForge.PostProcessorError{ArgumentError}(HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Server: GitHub.com
Date: Fri, 16 Sep 2022 13:26:52 GMT
Content-Type: application/json; charset=utf-8
Transfer-Encoding: chunked
Cache-Control: private, max-age=60, s-maxage=60
Vary: Accept, Authorization, Cookie, X-GitHub-OTP
ETag: W/"ee47c6975a415059782dfe8b3c71c02b079bd89d1892c21113ccd6098f073078"
Last-Modified: Wed, 14 Sep 2022 15:48:33 GMT
X-OAuth-Scopes: repo, user
X-GitHub-Media-Type: github.v3; format=json
X-RateLimit-Limit: 5000
X-RateLimit-Remaining: 4998
X-RateLimit-Reset: 1663338283
X-RateLimit-Used: 2
X-RateLimit-Resource: core
Access-Control-Expose-Headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubdomains; preload
X-Frame-Options: deny
X-Content-Type-Options: nosniff
X-XSS-Protection: 0
Referrer-Policy: origin-when-cross-origin, strict-origin-when-cross-origin
Content-Security-Policy: default-src 'none'
Vary: Accept-Encoding, Accept, X-Requested-With
Content-Encoding: gzip
X-GitHub-Request-Id: 84DC:64CD:3A2E004:3B22A11:6324799B
{"login": ***PERSONAL DATA REMOVED FROM LOG***
⋮
1409-byte body
""", ArgumentError("Invalid DateTime string"), Base.StackTraces.StackFrame[parse(#unused#::Type{Dates.DateTime}, s::String, df::Dates.DateFormat{Symbol("yyyy-mm-dd\\THH:MM:SS.s"), Tuple{Dates.DatePart{'y'}, Dates.Delim{Char, 1}, Dates.DatePart{'m'}, Dates.Delim{Char, 1}, Dates.DatePart{'d'}, Dates.Delim{Char, 1}, Dates.DatePart{'H'}, Dates.Delim{Char, 1}, Dates.DatePart{'M'}, Dates.Delim{Char, 1}, Dates.DatePart{'S'}, Dates.Delim{Char, 1}, Dates.DatePart{'s'}}}) at parse.jl:277, DateTime at io.jl:483 [inlined], #construct#18 at StructTypes.jl:458 [inlined], construct at StructTypes.jl:458 [inlined], #construct#16 at StructTypes.jl:456 [inlined], construct at StructTypes.jl:456 [inlined], read(::StructTypes.StringType, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, ::Type{Dates.DateTime}; kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) at structs.jl:148, read at structs.jl:124 [inlined], macro expansion at structs.jl:75 [inlined], #read#21 at structs.jl:67 [inlined] … repl_backend_loop(backend::REPL.REPLBackend) at REPL.jl:200, start_repl_backend(backend::REPL.REPLBackend, consumer::Any) at REPL.jl:185, run_repl(repl::REPL.AbstractREPL, consumer::Any; backend_on_current_task::Bool) at REPL.jl:317, run_repl(repl::REPL.AbstractREPL, consumer::Any) at REPL.jl:305, (::Base.var"#874#876"{Bool, Bool, Bool})(REPL::Module) at client.jl:387, #invokelatest#2 at essentials.jl:708 [inlined], invokelatest at essentials.jl:706 [inlined], run_main_repl(interactive::Bool, quiet::Bool, banner::Bool, history_file::Bool, color_set::Bool) at client.jl:372, exec_options(opts::Base.JLOptions) at client.jl:302, _start() at client.jl:485])
Stacktrace:
[1] request(f::GitHubAPI, fun::Function, ep::GitForge.Endpoint; headers::Vector{Pair{SubString{String}, SubString{String}}}, query::Dict{Any, Any}, request_opts::Dict{Any, Any}, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ GitForge ~/.julia/packages/GitForge/TsFLH/src/request.jl:166
[2] request
@ ~/.julia/packages/GitForge/TsFLH/src/request.jl:126 [inlined]
[3] #get_user#21
@ ~/.julia/packages/GitForge/TsFLH/src/helpers.jl:8 [inlined]
[4] get_user(f::GitHubAPI)
@ GitForge ~/.julia/packages/GitForge/TsFLH/src/helpers.jl:8
[5] top-level scope
@ REPL[7]:1
caused by: ArgumentError: Invalid DateTime string
Stacktrace:
[1] parse(#unused#::Type{Dates.DateTime}, s::String, df::Dates.DateFormat{Symbol("yyyy-mm-dd\\THH:MM:SS.s"), Tuple{Dates.DatePart{'y'}, Dates.Delim{Char, 1}, Dates.DatePart{'m'}, Dates.Delim{Char, 1}, Dates.DatePart{'d'}, Dates.Delim{Char, 1}, Dates.DatePart{'H'}, Dates.Delim{Char, 1}, Dates.DatePart{'M'}, Dates.Delim{Char, 1}, Dates.DatePart{'S'}, Dates.Delim{Char, 1}, Dates.DatePart{'s'}}})
@ Dates /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Dates/src/parse.jl:277
[2] DateTime
@ /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Dates/src/io.jl:483 [inlined]
[3] #construct#18
@ ~/.julia/packages/StructTypes/AK4aM/src/StructTypes.jl:458 [inlined]
[4] construct
@ ~/.julia/packages/StructTypes/AK4aM/src/StructTypes.jl:458 [inlined]
[5] #construct#16
@ ~/.julia/packages/StructTypes/AK4aM/src/StructTypes.jl:456 [inlined]
[6] construct
@ ~/.julia/packages/StructTypes/AK4aM/src/StructTypes.jl:456 [inlined]
[7] read(::StructTypes.StringType, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, ::Type{Dates.DateTime}; kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ JSON3 ~/.julia/packages/JSON3/GoF7x/src/structs.jl:148
[8] read
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:124 [inlined]
[9] macro expansion
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:75 [inlined]
[10] #read#21
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:67 [inlined]
[11] read
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:67 [inlined]
[12] #_#48
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:554 [inlined]
[13] StructClosure
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:553 [inlined]
[14] applyfield
@ ~/.julia/packages/StructTypes/AK4aM/src/StructTypes.jl:872 [inlined]
[15] read(::StructTypes.UnorderedStruct, buf::Base.CodeUnits{UInt8, String}, pos::Int64, len::Int64, b::UInt8, ::Type{GitForge.GitHub.User}; kw::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ JSON3 ~/.julia/packages/JSON3/GoF7x/src/structs.jl:615
[16] read
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:565 [inlined]
[17] #read#16
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:40 [inlined]
[18] read
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:33 [inlined]
[19] #read#14
@ ~/.julia/packages/JSON3/GoF7x/src/structs.jl:13 [inlined]
[20] read(io::IOBuffer, ::Type{GitForge.GitHub.User})
@ JSON3 ~/.julia/packages/JSON3/GoF7x/src/structs.jl:13
[21] postprocess(p::GitForge.JSON, r::HTTP.Messages.Response, #unused#::Type{GitForge.GitHub.User})
@ GitForge ~/.julia/packages/GitForge/TsFLH/src/request.jl:85
[22] request(f::GitHubAPI, fun::Function, ep::GitForge.Endpoint; headers::Vector{Pair{SubString{String}, SubString{String}}}, query::Dict{Any, Any}, request_opts::Dict{Any, Any}, kwargs::Base.Iterators.Pairs{Union{}, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ GitForge ~/.julia/packages/GitForge/TsFLH/src/request.jl:164
[23] request
@ ~/.julia/packages/GitForge/TsFLH/src/request.jl:126 [inlined]
[24] #get_user#21
@ ~/.julia/packages/GitForge/TsFLH/src/helpers.jl:8 [inlined]
[25] get_user(f::GitHubAPI)
@ GitForge ~/.julia/packages/GitForge/TsFLH/src/helpers.jl:8
[26] top-level scope
@ REPL[7]:1
It would be nice to be able to create or update a file on Github through the API, as GitHub.jl does.
I'd be willing to work on this at some point, but right now I'm a bit swamped.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.