sindresorhus / got Goto Github PK
View Code? Open in Web Editor NEW🌐 Human-friendly and powerful HTTP request library for Node.js
License: MIT License
🌐 Human-friendly and powerful HTTP request library for Node.js
License: MIT License
Take the following:
var url = 'http://github.com/mattdesl/graphics-resources/blob/master/LICENSE.md'
require('got')(url, {
timeout: 4000
}, function (err, result, res) {
if (err) {
console.error('err', err.message)
} else {
console.log('result', result)
}
})
In node 0.10.x it runs, but results in an err
saying the request failed.
In node 0.12.7 the method does not run, since it reaches this error before requesting:
_http_client.js:75
throw new Error('Protocol "' + protocol + '" not supported. ' +
^
Error: Protocol "https:" not supported. Expected "http:".
at new ClientRequest (_http_client.js:75:11)
at Object.exports.request (http.js:49:10)
at Object.exports.request (https.js:136:15)
at get (/projects/npmutils/gh-readme-scrape/node_modules/got/index.js:138:16)
at ClientRequest.<anonymous> (/projects/npmutils/gh-readme-scrape/node_modules/got/index.js:163:5)
at ClientRequest.g (events.js:199:16)
at ClientRequest.emit (events.js:129:20)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:426:21)
at HTTPParser.parserOnHeadersComplete (_http_common.js:111:23)
at Socket.socketOnData (_http_client.js:317:20)
I'm guessing it's because the link should actually be:
https://github.com/mattdesl/graphics-resources/blob/master/LICENSE.md
Any ideas how to fix this? Ideally I'd like the http://
request to follow to the https://
url.
From: #105 (comment)
Passing Basic Auth in the URL is deprecated and not even possible anymore in Chrome and IE: http://serverfault.com/questions/371907/can-you-pass-user-pass-for-http-basic-authentication-in-url-parameters https://tools.ietf.org/html/rfc3986#section-3.2.1
In the next major version of got
we should throw and tell the user to use the auth
option instead.
Hi,
I see difference in tarballs available for download via npm repository and this github releases. Can we have at lease "license" as well as "test.js" included in npm tarball?
Thanks.
If we use this behind a corporate proxy will fail with ECONNREFUSED
and I'm not sure how this can be updated to take a proxy option when it's called.
It seems like got
silently swallows errors.
var got = require('got')
got('http://josephdykstra.com', function (err, data) {
console.log('i see this message')
throw new Error('lolwut')
console.log('but not this message')
})
process.on('uncaughtException', function(err) {
console.log('error: ' + err)
})
C:\Users\Michael\Github\javascript\just-login-emailer\test>node test2.js
i see this message
C:\Users\Michael\Github\javascript\just-login-emailer\test>
This is my code:
var opts = {
method: 'POST',
encoding: 'utf8',
headers: {
'Content-Type': 'application/json'
},
body: {
"email": "[email protected]",
"password": "mypassword"
}
}
got(base_url + '/url_to_post', opts, function(err, data, res) {
expect(res.statusCode).toBe(200);
done();
});
I have a problem with send jason data because in node server I get this error:
Error: invalid json
at parse (...../node_modules/body-parser/lib/types/json.js:84:15)
.....
If I try to print the body in ...../node_modules/body-parser/lib/types/json.js
.....
function parse(body) {
console.log(body);
if (body.length === 0) {
// special-case empty json body, as it's a common client-side mistake
// TODO: maybe make this configurable or part of "strict" option
return {}
}
.....
I get this log:
email=my%40mail.com&password=mypassword
Some ideas?
right now I can require('got')
and require('http')
to do e.g. var agent = new http.Agent({ maxSockets: 1 })
and then pass in the agent to the options when I do a got
request, but IMO it would be nicer to not have to require('http')
:)
I was given an API ID and password for a service. The password has a #
in it. When using got to make a put request, it bugs out. I have traced the problem back to the #
in the password. Here is something to recreate it (at least using the API service I am using..., but you should be able to simulate something similar with some other service you make requests to) :
// First lets do a normal https request to show that this works:
var id = "test";
var pass = "45d3ps#453";
var https = require('https');
var options = {
'method': 'POST',
'host': 'account.myservice.com',
'path': '/api/token',
'auth': id + ":" + pass,
'headers': {
'Content-Type': 'application/x-www-form-urlencoded'
}
};
var req = https.request(options, function(res){
console.log(res.statusCode);
console.log(res.headers);
res.on('data', function(data){
process.stdout.write(data);
});
});
req.on('error', function(err){
console.log("Error", err);
});
req.write('request=credentials');
req.end();
I get the expected response:
200
{ 'content-type': 'application/json; charset=UTF-8',
'cache-control': 'no-store',
date: 'Fri, 18 Sep 2015 22:05:09 GMT',
'accept-ranges': 'bytes',
vary: 'Accept-Charset, Accept-Encoding, Accept-Language, Accept',
pragma: 'no-cache',
connection: 'close' }
{"expires_in":1799,"token_type":"api-test","access_token":"dk393kd-3i9dk3-39dk"}
So that works fine. Now with got
:
var id = "test";
var pass = "45d3ps#453";
var got = require('got');
var url = "https://" + id + ":" + pass + "@account.myservice.com/api/token";
var options = {
'body': 'request=credentials',
'headers': {
'Content-Type': 'application/x-www-form-urlencoded'
}
}
got.post(url, options, function(err, data, res){
if (err) {
console.log("Error", err);
} else {
console.log(data, res);
}
});
I get:
Error { [RequestError: getaddrinfo ENOTFOUND test]
message: 'getaddrinfo ENOTFOUND test',
code: 'ENOTFOUND',
host: 'test',
hostname: 'test',
method: 'POST',
path: '/:45d3ps' }
The first thing I noticed was that the return headers path had part of the password in it. It didn't take long to figure out that the #
hash was causing some issues with the url. Honestly, I'm not sure (according to internet w3c standards) if it's even valid to have a #
hash in the username or password section of a URL since a hash is typically used at the end of a url.... But it's working for Node's vanilla https module so I dunno...
Any thoughts on this?
Would be useful.
Now got
without callback behaves as ReadableStream:
got('http://todomvc.com').pipe(fs.createWriteStream('index.html'));
But (in case of non-empty body) there is no way to write data from stream to request body like this:
stream.pipe(got('http://todomvc.com'));
Some thoughts, that should be considered before implementing:
got
should call req.end()
in stream mode?got
be pipe-able in callback mode?i see this in the code
Line 76 in 41d147b
but its not in the README, just wanted to double check that it should be documented. if so, im happy to send a PR adding docs
While thinking about redirects, I notice, that got
is ignoring cookies on redirects - and we have no way to do something about it in userland.
We could resend (Cookies are host-sensitive and redirect can change host.)set-cookie
header as cookie
on redirect - all gathered cookies on redirects path could be merged in set-cookie
header in response.
This take me a wile to figure out, that I don'' have network problems from this error message:
Error: Couldn't connect to http://subhost.yandex.net.
Could we change it to somethig more accurate (that may include status code):
Error: Response status code from http://subhost.yandex.net is not OK (403).
Example code:
'use strict'
const got = require('got')
const url = 'https://github.com/xxx.jpg'
got(url, function (err) {
console.log(err.statusCode) // 406
})
got(url)
.catch(function (err) {
console.log(err.statusCode) // 406
})
got.stream(url)
.on('error', function (err) {
console.log(err.statusCode) // output nothing, seams that there is no error event
})
I believe it's an anti-pattern to synchronously throw errors in promises.
Example:
var mightBeFalse = false;
got(mightBeFalse)
.then(doStuff)
.catch(handleRejectedPromise);
got
synchronously throws
Parameter `url` must be a string or object, not boolean
Instead of rejecting the promise which would be handled by handleRejectedPromises
.
I'm using multistream
to stream multi API pages through the same stream (as suggested in issue #108) but I've ran into an issue. The API returns an empty array for pages that exceed the dataset so I need to "peek" at the data in the stream to know if I've reached the end. Using the example under stephenplusplus/stream-faqs as reference, if I add a stream.on('data', function() { ... })
, no output is piped to process.stdout
when using got.stream()
. The issue does not occur when using request.
var MultiStream = require('multistream')
var got = require('got')
//var request = require('request')
var count = 0;
function factory (cb) {
if (count > 3) return cb(null, null)
count++
var stream = got.stream('https://www.random.org/integers/?num='+count+'&min=1&max=6&col=1&base=10&format=plain&rnd=new')
//var stream = request('https://www.random.org/integers/?num='+count+'&min=1&max=6&col=1&base=10&format=plain&rnd=new')
stream.on('data', function() {
})
setTimeout(function () {
cb(null, stream)
}, 100)
}
MultiStream(factory).pipe(process.stdout);
14.23 Host
A client MUST include a Host header field in all HTTP/1.1 request messages
Seems like we must support this?
It seems that http(s).request(url)
uses url.path
while urlLib.format(url)
uses url.pathname
.
I do not have the time to look further but I think it can cause some issues (for instance when displaying the URL in an error message).
For now we are using certificates from request/request. We need to generate our own :)
Except for a few tests that specifically tests the callback interface.
We can then use async functions, which are so nice:
test('retry - timeout errors', async t => {
t.is(await got(`${s.url}/knock-twice`, {timeout: 1000}), 'who`s there?');
});
More info: https://github.com/sindresorhus/ava#async-function-support
I'm not sure why this happens, but I try to communicate with an API and I have to send a body argument like:
var options = {
body: {
txt: 'my string'
}
}
under the request:
got.post('myapiendpoint', options, cb);
when the size of the txt
param is tiny, the req is resolved successful:
silly: » Calling API...
TXT LENGTH 29878
silly: » Parsing body...
silly: » Handling body...
silly: » successful req and res
but when the size is more large I have a socket hang up error:
silly: » Calling API...
TXT LENGTH 153438
[ { [RequestError: socket hang up]
message: 'socket hang up',
code: '...',
host: '...,
hostname: '...',
method: 'POST',
path: '...' } ]
For now got
is not returning request
object, that can be aborted on timeout. It could be added (see floatdrop@bfb865e), by it is not quite obvious, which request object should be returned on redirect?
Any ideas?
Turns out some people use this with browserify
, but by default got
is huge when browserified, since it includes the buffer
browser polyfill and other things. We should look into if there's any way to get it down without too much work.
Relevant: https://twitter.com/mattdesl/status/624703076159590400
Not sure, but this might be useful: https://github.com/miguelmota/buffer-to-arraybuffer
If not already defined by the user of course.
This is just a proposal - feel free to share ideas, criticize, etc.
Get Promise API into got
with polyfill for Promise in Node.JS 0.10
like this:
got('google.com'); // => Promise
got('google.com', function (err, body) {
// ...
});
✨Generators✨ are awesome for flow-control and we finally can forget about callback hell and move on (with some help of co
-like libraries) to yield
:
co(function *() {
var results = yield [got('google.com'), got('github.com')];
console.log(results);
});
To support this, we should return Promise from got.
Also new fetch API
based on Promises landed in Chrome 42.
When ES7 will come to play, we will be able to go even further with async
/await
:
async function () {
let site = await got('google.com');
};
Promise API conflicts with Stream API
Atm I have no idea how to merge them (and if it is possible, I would vote against it), so one way is to move Stream API under stream
property, like this:
var stream = got.stream('google.com');
This will be breaking change, so "brace yourselves, major release is coming" (hah). What we can do in 3.x
version, to warn users - add promise
property first and deprecation warning of Stream API.
There are two ways, with which Promise can resolve request. This will affect, how results can be destructed into variables:
Resolve into Array of response
and body
(like then-got):
// Assume co-wrapper in all examples below ↓
var [response, body] = yield got('google.com');
Resolve into response Object with body
property (like got-promise):
var {body, headers} = yield got('google.com');
It seems like resolving to Object make much more sens - order of vars is not matters and user can access body, without response (or vice versa with different order).
TS: @sindresorhus
Related: request/request#796
I tried to use gh-got as a promise, but since there's no callback, it's seen as a stream and that's not compatible with json for some reason.
Should it be possible to use got as a promise but not treated as a stream? Please let me know if I'm misunderstanding something.
im porting some request code over that looks like this
req.on('response', handle)
in request, handle
gets called when the 'final' redirect gets called, so e.g. if the first response was actually a 302, and the 2nd one is the a 200, only the 200 response will be emitted as the response
event.
in got, response
is emitted each time their is a response, because if this line is truthy multiple times, the recursive call in there causes it to emit many times
the problem I have is that I only care about the last response, but there doesn't seem to be an event for that, but maybe i'm missing something!
https://github.com/sindresorhus/got/blob/master/index.js#L64. Could set a custom property (.resCode
) with the error code and supply a user friendly error message like Couldn't connect to http://github.com (code 500)
.
The resCode
property would need to be well documented since many will want too look for it.
fetch library have option to disable gzip. Since we enforcing ungzip on response, wouldn't it be nice to have option to disable it?
For now we don't have parsing on responses, that are not okay by them selves:
read(res, encoding, function (err, data) {
err = new GotError(url + ' response code is ' + statusCode + ' (' + status[statusCode] + ')', err);
err.code = statusCode;
cb(err, data, response);
});
But it is not convinient, when API returns JSON (like github does) and we need to parse it by hands.
On other side we have huge amount of services, that proxy'ed and return 50x codes with default nginx error page.
So question is - maybe we should parse JSON on non-200 responses silently? Anyway got
is already in error state.
Just putting it in an issue instead of a inline TODO
.
Lines 93 to 102 in e3897a0
request
has this. Could be a nice shortcut.
It basically JSON.stringify()
the input and sets the header 'content-type': 'application/json'
and JSON.parse()
the output.
@kevva @floatdrop @julien-f Thoughts?
I wanted to create a PR via the GitHub interface but I inadvertently committed directly instead.
Feel free to revert the commit if it does not seem correct to you.
Sorry for this :/
Does got
support cookies? If so, how do I pass it a cookie jar? Where do I get a cookie jar? The README doesn't seem to explain this.
Context: I'm switching a project of mine from request to got, which uses a cookie jar (it doesn't appear to work without one). When I finished switching it over, I got a "Not Authorised" response from a url in a sequence of 3 which I should have got a success response instead. This leads me to think that got
isn't storing cookies correctly.
I'm new to this library, but I thought it was weird that I had to specify method: 'PUT'
in my request options when I called got.stream.put()
. I figured that put
was implied:
var options = {
method: 'PUT'
}
fs.createReadStream(fileInput).pipe(got.stream.put(url, options, function(err, res){
console.log(res);
}));
Seems like the method called should automatically alter the method type in the options.
Is honoring the default proxy behaviour ("npm config" or env variables) intended or is this supposed to be a low-level API that will never do it?
Thanks.
Before major release happen - maybe we should subclass error a little bit more, than GotError
? For example we have parsing errors, http request errors, maxredirects errors, etc...
Sometimes it is useful to differentiate them by class, not by message.
One more concern is internal structure of errors - should we mimic error properties from Node 0.12 by default?
//cc @sindresorhus @kevva @julien-f
Now according to doc:
If
body
is a plain Object, it will be stringified withquerystring.stringify
and sent asapplication/x-www-form-urlencoded
.
But if I need post json data there is no option for this. And I have to use Promise to throw out SyntaxError exception:
Client.prototype.create = function create(resource, options, data) {
resource = this.makeUrl(resource, options);
options = this.processHeaders(options);
options.headers['Content-Type'] = 'application/json';
try {
options.body = JSON.stringify(data);
} catch (ex) {
return new Promise(function () { throw ex; });
}
return got.post(resource, options);
};
https://github.com/sindresorhus/got/blob/master/index.js#L69
Error event emits only with err
argument and there is no way to acces to body of response.
How about somthing like
cb = function () {
var args = Array.prototype.slice.call(arguments, 0);
args.unshift('error');
proxy.emit.apply(proxy, args);
};
I am using got
in the browser via Browserify. The URL of my web app is http://localhost:8000/app/#/
. I'd like to send a GET request to http://localhost:8000/app/ping
.
With xhr
, one can specify the request URL as ./ping
, and it works. I quite like this because then I don't need to worry about the hostname and whatnot. But with got
, it seems to make the request to http://./ping
.
Just wondering if we could mimic xhr
's functionality in this instance?
request lib also has this issue: request/request#1596
Current implementation works if using pathname
but not with path
.
I think the URL object should be normalized as soon as possible to avoid conflicts between:
host
and hostname
path
and pathname
Networks suck. They fail all the time and are fragile. Having functionality for got
to retry in case of failure would make it a lot more solid.
@vdemedes made this great module we could use a inspiration: https://github.com/vdemedes/got-retry
Question: Should it be on by default?
@floatdrop Thoughts?
Related: request/request#1300
I attempted to use got.stream()
with continue-stream to stream multiple pages of API requests into a single stream but it appears to not work. The first page will pull down, but additional calls will not work. Here's is the snippet I'm using for testing (pulled from their README). It works fine with the request library's stream that the example uses.
var continueStream = require('continue-stream')
var request = require('request')
var got = require('got')
var pumpify = require('pumpify')
var JSONStream = require('JSONStream')
var page = 1
function next(callback, previousStream) {
if (page >= 4) return callback()
/*
var req = request({
url: 'https://api.github.com/repos/joyent/node/events?page=' + (page++),
headers: {'user-agent': 'pug'}
})
*/
var req = got.stream('https://api.github.com/repos/joyent/node/events?page=' + (page++), {
headers: {'user-agent': 'pug'}
})
var stream = pumpify.obj(req, JSONStream.parse('*'))
callback(null, stream)
}
continueStream.obj(next)
.on('data', function(data) {
console.log(data);
})
Most POST requests (by default) are made with the application/x-www-form-urlencoded
Content-Type. We (node) are using text/plain
by default. I think it's worth changing this to use a more expected behavior.
IMHO, it is too bad that this module cannot be used for binary content due to the setEncoding('utf-8')
.
I think it should instead return a buffer and let the user call toString()
to obtain a string if necessary.
See https://gist.github.com/julien-f/9cc9976ce768890eb274#file-get-js for inspiration, maybe :)
We are indirectly using got.js as an indirect dependency in our mobile application (Cordova, AngularJS and Ionic). The CI build inside the corporation is throwing the following exception which does not impact the IPA/APK files that get generated as an output from the CI build but it is annoying to see that in a CI build where folks usually expect perfect log statements. Here's the stack trace:
[email protected] postinstall /Users/hudson/.hudson/jobs/Mobile-Dashboard-Pilot/workspace/node_modules/ionic-minify/node_modules/mozjpeg-stream/node_modules/mozjpeg
node lib/install.js
⚠ Redirected 10 times. Aborting.
⚠ mozjpeg pre-build test failed
ℹ compiling from source
✖ MaxRedirectsError: Redirected 10 times. Aborting.
at ClientRequest. (/Users/hudson/.hudson/jobs/Mobile-Dashboard-Pilot/workspace/node_modules/ionic-minify/node_modules/mozjpeg-stream/node_modules/mozjpeg/node_modules/bin-build/node_modules/download/node_modules/got/index.js:37:23)
at ClientRequest.g (events.js:199:16)
at ClientRequest.emit (events.js:107:17)
at HTTPParser.parserOnIncomingClient as onIncoming
at HTTPParser.parserOnHeadersComplete (_http_common.js:111:23)
at Socket.socketOnData (_http_client.js:317:20)
at Socket.emit (events.js:107:17)
at readableAddChunk (_stream_readable.js:163:16)
at Socket.Readable.push (_stream_readable.js:126:10)
at TCP.onread (net.js:538:20)
[email protected] postinstall /Users/hudson/.hudson/jobs/Mobile-Dashboard-Pilot/workspace/node_modules/ionic-minify/node_modules/pngout-bin
node lib/install.js
✔ pngout pre-build test passed successfully
[email protected] postinstall /Users/hudson/.hudson/jobs/Mobile-Dashboard-Pilot/workspace/node_modules/ionic-minify
node scripts/install.js
Copying minifier file to project's hooks/after_prepare...
Copying configuration file to project's hooks/ folder...
Copying chalk to your node_modules/ folder...
Copying clean-css to your node_modules/ folder...
Copying mozjpeg-stream to your node_modules/ folder...
Copying ng-annotate to your node_modules/ folder...
Copying pngout-bin to your node_modules/ folder...
Copying uglify-js to your node_modules/ folder...
Copying win-spawn to your node_modules/ folder...
Finished installing. Experience the awesomeness ;)
Previous issue #7. I couldn't care less about this, but it's useful to have a place to refer people that open issues elsewhere.
A good PR would be accepted, but won't happen from my part.
I would prefer for the solution to be an external module got
can just depend on as I don't want to have to maintain it. Maybe https://github.com/mikeal/tunnel-agent.
I strongly believe this is something that should be a part of Node.js and not every module doing HTTP. +1
this issue if you agree: nodejs/node#8381
(In stream mode) on error event got
emits error
and then response
events (in this order). This seems kind of awkward - because error
will read response body and this will lead to confusion in request
event handler.
@sindresorhus thoughts?
It seems like there an open socket left after got has redirect reply from server. Setting {agent: false}
fixes test, but feels like it should be working by default.
Default http agent in node is limited by 5 open sockets. Read more in hyperquest readme.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.