follow-redirects / follow-redirects Goto Github PK

See #30 (comment)

Brings me to another problem BTW, that a call to abort() should probably abort everything in the entire chain.

As per RFC:

307 Temporary Redirect

The 307 (Temporary Redirect) status code indicates that the target resource resides temporarily under a different URI and the user agent MUST NOT change the request method if it performs an automatic redirection to that URI. Since the redirection can change over time, the client ought to continue using the original effective request URI for future requests.

The server SHOULD generate a Location header field in the response containing a URI reference for the different URI. The user agent MAY use the Location field value for automatic redirection. The server's response payload usually contains a short hypertext note with a hyperlink to the different URI(s).

Note: This status code is similar to 302 (Found), except that it does not allow changing the request method from POST to GET. This specification defines no equivalent counterpart for 301 (Moved Permanently) ([RFC7238], however, defines the status code 308 (Permanent Redirect) for this purpose).

Right now GET is always being used.

function defaultMakeRequest(options, cb, res) {
  if (res) {
    // This is a redirect, so use only GET methods
    options.method = 'GET';
  }

  var req = options.nativeProtocol.request(options, cb);

  if (res) {
    // We leave the user to call `end` on the first request
    req.end();
  }

  return req;
}

Any chance you would consider changing the license to MIT or apache 2?

BSD requires attribution in any marketing done for a product build with this module.

I think follow-redirects default behavior should be to mimic the behavior of the http and https as used in the browser via browserify. XMLHtppRequest follows redirects transparently, and there is no way to disable that. Therefore the browserified versions of http and https automatically have a follow-redirects like behavior, but the server side counterparts do not (obviously, there would be little point to this module otherwise). However, this behavior breaks one of the key goals of browserify; the ability to write code that behaves identically on the client and server. This isn't the fault of browserify, due to the way XMLHttpRequest behaves they can't possibly emulate the behavior of the native modules.

I am definitely not suggesting we turn follow-redirects into a client side library, but instead help the server behave more like the client. This is backwards from the goal of most browserify modules, but as previously stated, it's just not possible to emulate the server in this case. Developers seeking to create a cross platform solution would use follow-redirects on the server, and have browserify automatically swap it out with browserify-http when creating the client bundle. Indeed, I have already detailed instructions for doing just that in the README.

We are already very close. For many scenarios the library will work as is. However, there are still a few points to be addressed.

1. Issue #12 needs to be fixed. Specifically, our current logic for deciding when to redirect is pretty basic. It's not always OK to redirect for every 300 level code. The spec notes many scenarios where typical browser behavior is actually not to spec. Rather than strictly conforming to the spec, we should follow whatever the most common behavior is.
2. Also related to #12, we currently always perform a GET on redirection, even if the original request was a POST or otherwise. This is the correct response in most situations, but (I believe) not all.
3. Handling errors. Currently we emit an error on the returned client request object if we exceed the maximum number of redirects. We also forward any errors we encounter on subsequent redirected requests to the original one. If we encounter a 3XX status code without a Location header, we do not throw an error, but return the result to the registered callback. This all makes logical sense, but we should work to make our behavior the same as the client side modules.
4. Make all this configurable. There is already an (optional and as yet undocumented) makeRequest callback that allows us to change these behaviors on master. It's probably wise to keep a "legacy" version of that callback around that continues to mimic the behavior of the library as it has existed for years.
5. We should do all this while still maintaining API compatibility with the core modules (nodes native http and browserify-http, etc). Plenty of cross-platorm options exist if you are willing to deal with a different API. The goal here should be an easy transitions for those using the core modules.
6. Be as lightweight on the client side as possible. Ideally, this means not existing at all on the client, and instead differing to the core browserify implementations that are likely already bundled.

According to this comment, we should handle the infinite loops here, in this library.

Connections don't close on node v0.12.2 due to this line. The node server will eventually stop responding. In a production app I fixed it by commenting the line out.

Maybe putting a conditional for node version would be a better solution?

Thanks!

If I try to use a proxy and access URL that sends a single 302 redirect I'm ending up hitting the redirect limit.

The following code shows the problem.

node test.js <url of proxy> <url to test>

I've been using squid in a vm to test, if I access http://google.com it trys to redirect to http://www.google.com

GET http://google.com/ HTTP/1.1
Host: google.com
Connection: close

HTTP/1.1 302 Found
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Location: http://www.google.co.uk/?gfe_rd=cr&ei=Gnm5WL3dDqTS8Aec2IPwAg
Content-Length: 261
Date: Fri, 03 Mar 2017 14:09:30 GMT
X-Cache: MISS from ubuntu
X-Cache-Lookup: MISS from ubuntu:3128
Via: 1.1 ubuntu (squid/3.5.12)
Connection: close

var followRedirect = require('follow-redirects');
var http = followRedirect.http;
var urllib = require('url');

var proxy = "http://192.168.122.250:3128/";
var target = "http://google.com";
if (process.argv.length == 3) {
	proxy = process.argv[2];
} else if (process.argv.length == 4) {
	proxy = process.argv[2];
	target = process.argv[3];
}

var options = urllib.parse(proxy);
var targetOpt = urllib.parse(target);

options.headers = {
	"Host": targetOpt.host
}

options.pathname = target;
options.path = target;

console.log(options);

var req = http.request(options,function(res){
	var body = "";
	res.on('data',function(chunk){
		body += chunk;
	});

	res.on('end',function(){
		console.log("All Donei %d", res.statusCode);
		console.log(body);
	});
});

req.setTimeout(120000, function(){
	console.log("timeout");
	req.abort();
});

req.on('error', function(err){
	console.log(err);
});

req.end();

Hey there,

I like your approach to plug in at a low level with http and https. I want to take the same approach for client http caching. I've spent a fair amount of time searching and didn't find any existing module, apart from cache-http which has a weird API and only look at If-Modified-Since.

My question to you is are you aware of a library I may have missed and is there anything like connect but for http clients?

it is only using _.extend.

It could be replaced by merge. merge is less than 1kB when minified (as opposed to 17 from underscore).

It might be better to just write an implementation ourselves though. merge has a recursive option which we do not need.

Got some problems on redirecting from a 302 POST request, basically the server refused to send back any response. After some investigation, I found it was caused by the Content-Length header, which probably makes the server think it's an invalid GET request.

My current makeshift is deleting the Content-Length field from this._options.headers for all GET requests inside _processResponse. Any better solution?

Looking at the HTTP spec, I think it is entirely possible that the server will consume at least part of the request body before issuing a 307 redirect. When that happens, we are unable to redirect because whatever portion of the stream has been consumed is lost to us.

Perhaps we should buffer the request body when it's a stream (at least to a point).

Someone sent me this email and I referred them to this issue:

Hi James
I have been using "follow-redirects" for making ReST calls and its very good. Thanks for the wonderful package. I am however stuck up when the site does a redirect. In that case, something very strange happens. Instead of getting a 307, for some very weird reason I keep getting 400 response.

Now I know this sounds very crazy as its the server which sends the response, and not the client, but whenever I use the normal "http" package, in my client code, instead of using the "follow-redirects", I get a 307 properly. Only for "follow-redirects" do I get a 400.

So is there any additional step that I need to do, to get this resolved.

When trying to use this in Mashery's I/O Docs I see the following error:

TypeError: Object # has no method 'request'
at /home/warp/code/iodocs/node_modules/follow-redirects/index.js:60:40
at unsecuredCall (/home/warp/code/iodocs/app.js:568:23)
at processRequest (/home/warp/code/iodocs/app.js:481:9)
at callbacks (/home/warp/code/iodocs/node_modules/express/lib/router/index.js:272:11)

etc...

https://github.com/warpr/iodocs/blob/coverartarchive/app.js#L564

Do you have any idea if this Is a bug, or am I doing something wrong?

I spent half a day tracking this down. I'm not sure if this qualifies as a bug, but if you accept my suggested fix, I'll submit a PR.

The problem I was having was that one of my tests on my project was failing (timing out, specifically). Needless to say, there were many layers above follow-redirects to comb through for the error. But I simply couldn't find anything. The problem I was seeing was that when my server returned a redirect, my client would hang. Hitting the server with a browser showed a proper Location header and status code. But the client (using axios) just would not return. It was making the request and the server was sending back a response, but the promise in Axios was never resolved. I presumed that the redirect was being processed, but it never seemed to get a response.

So I rolled up my sleeves and tried to track down exactly where the process was breaking down. My original request was a POST and follow-redirects was converting that to a GET. That by itself is what I expected, so that was no problem. I could change my original method to take a GET instead and then the problem went away. So that fixed my problem (tests ran to completion), but it really bugged me that I didn't understand why this wasn't working and I didn't want to just leave it at that and just hope not to run into it again.

It clearly had something to do with redirects, so I turned on debugging output with DEBUG=follow-redirects and took a look at what was going on. I could see follow-redirects making the redirect request. I assumed, incorrectly, that perhaps when a POST request was made, the POST method was used on the redirect (that would have cause an error). But that wasn't the issue. In fact, if that were the case the request should have completed (which wasn't happening). This provides a subtle clue into what actually was the problem.

I compared the options being passed in both the working and non-working cases. There were only two differences in those requests and they were related to headers. This is because my original request (which came through axios had headers normally associated with a POST in them. Specifically Content-Type and Content-Length. When follow-redirects switches the method to GET in its 307 handling, it leaves those there. So the resulting redirected GET request has a Content-Length header in it. Since a GET request has no body, this makes no sense.

I noticed that with the Chrome extension DHC, if I make a GET request with a Content-Length header, it removes it automatically. But it appears what is happening is that follow-redirects/http are keeping that header in there and when it hits the Express server, the express server sits and waits until it gets all the bytes for the body. But, of course, there is no body.

What I did to fix this was to simply change this:

	if (!(this._options.method in safeMethods)) {
		this._options.method = 'GET';
	}

...to this...

	if (!(this._options.method in safeMethods)) {
		this._options.method = 'GET';
                // Remove a Content-Length header if one is present
		delete this._options.headers["Content-Length"];
	}

This seems perfectly reasonable to me and it is very much inline with what DHC does. But I wanted to first check here to see if a) this seems like a reasonable approach and b) whether you want me to create a PR for this.

I can't see what harm this would do, but you guys know more about the HTTP implications of this than I do. For that matter, there may be other headers to consider here as well (i.e., that would never apply to a GET and therefore should be removed).

When no redirect happens, we would expect response.fetchedUrls[0] to contain the full URL for the request that was just executed. However, it only contains the protocol + the hostname.

would be nice. Either add it to the response object or add a second argument to the callback function.

Description

When a request is done using the request method, and when no redirect was made, the
response.responseUrl will only contain the base url and not the full url with the path included.

Reproduce

var http = require('follow-redirects').http;
var URLParser = require('url');
// Does not work
 var parsedUrl = URLParser.parse('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A00');
var settings = { hostname: parsedUrl.hostname, port: parsedUrl.port, path: parsedUrl.path, method: 'GET'};
http.request(settings, function (response) {
  console.log(response.responseUrl);
}).end();

// Works as it should:
http.get('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A00', function (response) {
  console.log(response.responseUrl);
}).on("error", function (err) {
  console.error(err);
});

// Works as it should: (request to a URL that gets redirected)
 var parsedUrl = URLParser.parse('http://graph.spitsgids.be/connections/?departureTime=2017-06-12T11%3A01');
var settings = { hostname: parsedUrl.hostname, port: parsedUrl.port, path: parsedUrl.path, method: 'GET'};
http.request(settings, function (response) {
  console.log(response.responseUrl);
}).end();

Hi there,

I use node-rest-client module which uses your module in dependance.
When I wan't to make a HTTP request, it returns an error Object.assign is not a function from your module (index.php lines 169 and 134).

My app is under Node 0.10.44 (I can't upgrade).

Please fix it asap (you can find a great polyfill on MDN or for NPM)

I'm getting the following error when running a test that uses nock.

TypeError: Cannot read property '_redirectable' of undefined
      at eventHandlers.(anonymous function) (node_modules/follow-redirects/index.js:22:7)
      at OverriddenClientRequest.RequestOverrider.req.once.req.on (node_modules/nock/lib/request_overrider.js:195:7)
      at Writable.RedirectableRequest._performRequest (node_modules/follow-redirects/index.js:69:12)
      at Writable.RedirectableRequest (node_modules/follow-redirects/index.js:46:7)
      at Object.wrappedProtocol.request (node_modules/follow-redirects/index.js:202:10)
      at Object.wrappedProtocol.get (node_modules/follow-redirects/index.js:207:33)
      at getLatestUrl (lib/update.js:21:13)
      ...

The error doesn't happen when I run the program regularly. See this travis build for more context.

HEADs where being converted to GETs, meaning HEADing a relayed server results in the entire payload being dumped unceremoniously on the requester.

I also had an issue with being stuck in an endless relay loop.

my quick and dirty hack to make it work can be cut and pasted to the top of any jsnode file, (don't worry with npm or other bollocks, just have assert,debug and consume installed.

enjoy.

'use strict';

var assert = require('assert');
var debug = require('debug')('follow-redirects');
var consume = require('stream-consume');
// follow-redirects is taken and butchered from code obtained using
// "npm install follow-redirects"
// ie: https://www.npmjs.com/package/follow-redirects
// it had enough bugs in it to warrant fixing, but
// it's not worth doing
var followRedirects = (function (_nativeProtocols) {

  var nativeProtocols = {};

  var publicApi = {
    maxRedirects: 5
  };

  for (var p in _nativeProtocols) {
    /* istanbul ignore else */
    if (_nativeProtocols.hasOwnProperty(p)) {
      // http://www.ietf.org/rfc/rfc2396.txt - Section 3.1
      assert(/^[A-Z][A-Z\+\-\.]*$/i.test(p), JSON.stringify(p) + ' is not a valid scheme name');
      generateWrapper(p, _nativeProtocols[p]);
    }
  }

  return publicApi;

  // to redirect the result must have
  // a statusCode between 300-399
  // and a `Location` header
  function isRedirect(res) {
    return (res.statusCode >= 300 && res.statusCode <= 399 &&
    'location' in res.headers);
  }


  function execute(options, callback) {
    var req_method=options.method;
    var fetchedUrls = [];
    var clientRequest = cb();

    // return a proxy to the request with separate event handling
    var requestProxy = Object.create(clientRequest);
    requestProxy._events = {};
    requestProxy._eventsCount = 0;
    if (callback) {
      requestProxy.on('response', callback);
    }
    return requestProxy;

    function cb(res) {
      // skip the redirection logic on the first call.
      if (res) {
        var fetchedUrl = url.format(options);
        fetchedUrls.unshift(fetchedUrl);

        if (!isRedirect(res)) {
          res.fetchedUrls = fetchedUrls;
          requestProxy.emit('response', res);
          return;
        }

        // we are going to follow the redirect, but in node 0.10 we must first attach a data listener
        // to consume the stream and send the 'end' event
        consume(res);

        // need to use url.resolve() in case location is a relative URL
        var redirectUrl = url.resolve(fetchedUrl, res.headers.location);
        debug('redirecting to', redirectUrl);

        options=url.parse(redirectUrl);
      }

      if (fetchedUrls.length > options.maxRedirects) {
        var err = new Error('Max redirects exceeded.');
        return forwardError(err);
      }

      options.nativeProtocol = nativeProtocols[options.protocol];
      options.defaultRequest = defaultMakeRequest;

      var req = (options.makeRequest || defaultMakeRequest)(options, cb, res);
      req.on('error', forwardError);
      return req;
    }

    function defaultMakeRequest(options, cb, res) {
      if (res && res.statusCode !== 307) {
        // This is a redirect, so use only GET methods, except for status 307,
        // which must honor the previous request method.
        options.method = 'GET';
      }

      if (req_method === 'HEAD') 
        options.method=req_method;

      var req = options.nativeProtocol.request(options, cb);

      if (res) {
        // We leave the user to call `end` on the first request
        req.end();
      }

      return req;
    }

    // bubble errors that occur on the redirect back up to the initiating client request
    // object, otherwise they wind up killing the process.
    function forwardError(err) {
      requestProxy.emit('error', err);
    }
  }

  function generateWrapper(scheme, nativeProtocol) {
    var wrappedProtocol = scheme + ':';
    var H = function () {};
    H.prototype = nativeProtocols[wrappedProtocol] = nativeProtocol;
    H = new H();
    publicApi[scheme] = H;

    H.request = function (options, callback) {
      return execute(parseOptions(options, wrappedProtocol), callback);
    };

    // see https://github.com/joyent/node/blob/master/lib/http.js#L1623
    H.get = function (options, callback) {
      var req = execute(parseOptions(options, wrappedProtocol), callback);
      req.end();
      return req;
    };
  }

  // returns a safe copy of options (or a parsed url object if options was a string).
  // validates that the supplied callback is a function
  function parseOptions(options, wrappedProtocol) {
    if (typeof options === 'string') {
      options = url.parse(options);
      options.maxRedirects = publicApi.maxRedirects;
    } else {
      options.maxRedirects = publicApi.maxRedirects;
      options.protocol     = wrappedProtocol;
    }
    assert.equal(options.protocol, wrappedProtocol, 'protocol mismatch');

    debug('options', options);
    return options;
  }
}) ({
  http: require('http'),
  https: require('https')
});
followRedirects.maxRedirects = 10; 
var  http  = followRedirects.http,
     https = followRedirects.https;

Pretty much the same as Issue #27, HTTP 308 redirects should also use the same request method rather than changing the method to GET.

Per RFC 7238:

308 Permanent Redirect

The 308 (Permanent Redirect) status code indicates that the target resource has been assigned a new permanent URI and any future references to this resource ought to use one of the enclosed URIs.

Clients with link editing capabilities ought to automatically re-link references to the effective request URI (Section 5.5 of [RFC7230]) to one or more of the new references sent by the server, where possible.

The server SHOULD generate a Location header field ([RFC7231], Section 7.1.2) in the response containing a preferred URI reference for the new permanent URI. The user agent MAY use the Location field value for automatic redirection. The server's response payload usually contains a short hypertext note with a hyperlink to the new URI(s).

A 308 response is cacheable by default; i.e., unless otherwise indicated by the method definition or explicit cache controls (see [RFC7234], Section 4.2.2).

Note: This status code is similar to 301 (Moved Permanently) ([RFC7231], Section 6.4.2), except that it does not allow changing the request method from POST to GET.

Your module don't keep options when he follows location
headers (usergagent, cookie ..) and agent (for keep alive connection) has gone one the first follow

exemple here :
http://pastebin.com/kxHViC2n

I'm working on proxy for downloading files from Google Docs. Since the urls Google Drive API gives for non-gdrive filetypes (pdf, txt, jpeg) are redirects, I decided to use your library. So when you hit http://localhost:3000/oauth2_download?accessToken=x&url=my_url where x is my GDrive access_token, url is, say, https://docs.google.com/uc?id=0B_31K_MP92hUNjljYjIyMzgtZTBmNS00MGMwLWIxNmQtYjMyNDFiYjY0MTJl&export=download, the file is supposed to be downloaded. However, the browser sends me to the Google Docs sign-in page which (given that hitting https://docs.google.com/uc?id=0B_31K_MP92hUNjljYjIyMzgtZTBmNS00MGMwLWIxNmQtYjMyNDFiYjY0MTJl&export=download in browser downloads the file if you're logged into Google Docs) means the headers are not being sent correctly. Any idea how should I use 'follow-redirects' library to deal with this issue? Thank you in advance!

//var https = require('https');
var https = require('follow-redirects').https;
var app = require('express')();
var urllib = require('url');
// Downloads the file using OAuth2. Tested on Google Drive
app.get('/oauth2_download', function(req, res){
    var url = req.param('url');
    var accessToken = req.param('accessToken');
    var headers = { "Authorization": "Bearer " + accessToken }
    var searchname = urllib.parse(url).search;
    var hostname = urllib.parse(url).hostname;
    var pathname = urllib.parse(url).pathname;
    if (url && accessToken) {
        // downloading file with streaming
        var options = {
            hostname: hostname,
            port: 443,
            path: pathname + searchname,
            method: 'GET',
            headers: headers
        };
        https.request(options, function(response){
            for (var key in response.headers) {
                res.setHeader(key, response.headers[key]);
            }
            response.on('data', function(chunk) {
                res.write(chunk);
            });
            response.on('end', function() {
                res.end();
            });
            //console.log("Done downloading using OAuth2");
        }).on('error', function(e) {
            console.error(e);
        }).end();
    } else {
        res.send(404);
        res.end();
    }
});

Using axios, I discovered this is the underlying library that handles redirects, so 👋 .

The API I'm communicating with follows JSON API which describes that clients must add Content-Type: application/vnd.api+json headers to their requests.

I have noticed though that when performing a redirect, this library will remove any header that starts with content-*.

This results in my requests failing, with a 415 Unsupported Media Type response (as per spec).

Just wondering what the intentions were behind this? Is there a workaround to "reinsert" headers such as Content-Type later?

Thanks!

I would like to know what is wrong in this code.. I don't https.Request() gets called when I remove content-length.

setProperty: function(name, value) {
var deferred = q.defer(); // Take a deferral

    var postData = JSON.stringify({name : value});
    var options = {
        protocol: 'https:',
        host: 'developer-api.nest.com',
        path: '/devices/thermostats/' + deviceId,
        headers: {
            'Authorization': 'Bearer ' + accessToken,
            'Content-Type': 'application/json',
            'Accept': 'application/json',
            'Content-length': Buffer.byteLength(postData),
        },
        method: 'PUT',
    };

    var httpsRequest = https.request(options, (getRes) => {
        console.log("Request sent");
        var body = '';
        getRes.setEncoding('utf8');
        getRes.on('data', (data) => {
            body += data;
        });

        getRes.on('end', () => {
            if (getRes.statusCode != 200) {
                deferred.reject("Error Code:" + getRes.statusCode);
            } else {
                var data = JSON.parse(body);
                deferred.resolve(target_temperature_c);
            }
        });

        getRes.on('error', (e) => {
            deferred.reject(e);
        });
    });
    httpsRequest.write(postData);
    httpsRequest.end();

    return deferred.promise; // return the promise
},

I have followed the tutorial.I am getting the error.

I'm using node v0.10.12 and I'm quite new to this.

This little program doesn't end using follow-redirects but it completes using the native http module:

//var http = require('http');
var http = require('follow-redirects').http;

http.get('http://bit.ly/900913', function (res) {
  var all = '';
  res.on('data', function (chunk) {
    all = all + chunk;
    console.log(all.toString().length);
  });

  res.on('end', function (chunk) {
    console.log(all.toString().length);
  });
}).on('error', function (err) {
  console.error(err);
});

Am I missing something obvious? Or maybe follow-redirects needs to be updated to node 0.10? Any hint would be appreciated. I could slowly take a stab at updating the module if that's indeed the problem.

I wanted to take a stab at #6 but I have to figure this out first.

As far as I know, not all of the 300-399 http response codes needs to have a redirect. But looking at the code, I found you are redirecting everything in 300-399. Please look at the following url to have a reference:

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

It seems the res.statusCode always returns 200 even if I'm redirected. So is there any way I can be told I was redirected?

Thanks,
Elgs

The following code will hang:

var req = http.request(opts, callback);
var stream = fs.createReadStream(filename);
stream.pipe(req);

Please note that the same code works fine with v0.0.7. Looks like the issue was introduced by #31. Removing the code that creates a request proxy resolves the problem.

I created a test that reproduces this issues. I will submit it as a PR.

Thank you!

@olalonde

I think it is time for a 1.0.0 release.

It gives us the ability to publish updates in a semver responsible way.

I have added a couple issues I want to close before then to the 1.0.0 milestone. Once those are done, I think we should do it.

Hello,

Could you please update the license to match either MIT or BSD?

Thanks,

-- Tito

The description states that follow-redirects is a "drop-in replacement". However, the replacement is not complete, as the response event is not correctly emitted.

In the original http and https implementations, the callback is a) optional and b) a shortcut for attaching a listener to the response event. However, follow-redirects makes the callback mandatory and does not reuse the response event. Any callbacks attached to response will fire at the wrong time (at the first request).

I realize this is hard to implement correctly, given that EventEmitter has no support for stopping event propagation. So if this cannot be fixed, it should probably be advertised that this module is not an exact replacement, but rather explain which parts it emulates and which parts it does not.

Module does not work with shortners which use meta refresh
http://gee.su/Ctv2n

Not taken into account set-cookie from the server

In the NodeJS framework, I’d like to create a user agent (outside the browser) that also properly implements HTTP client caching. Problem with this library is that it handles redirects itself, and omits the cache in front of follow-redirects.

E.g., in pseudo javascript code:

var url = 'http://data.linkeddatafragments.org/bwv/classifications/10'
if (!cache.get(url)) {
    http.get(url, function (...) {
        var url2 = 'http://data.linkeddatafragments.org/bwv?subject=http%3A%2F%2Fdata.linkeddatafragments.org%2Fbwv%2Fclassifications%2F10';
        if (!cache.get(url2)) {
             http.get(url2);
        }
    });
}

This example will twice download the exact same page while the cached response should be given.

Suggested solutions

1. A flag

When the flag is set, the library will not automatically follow redirects, but will also allow for returning the new location URL, for which an external cache can then check what to do: resolve the promise, or again request it via follow-redirects.

2. Also implement an (in-memory) caching system in follow-redirects

Also allow a cache instance to be given to the follow-redirects library.

3. Document how follow-redirects can be used as one part of the chain

Inspired by express middleware design pattern, we could also document chaining HTTP request libraries, in which you can put caches behind follow-redirects without issues.

@olalonde I've setup travis to automatically run our builds, and added badges for coverage, etc.

Can you head over to travis-ci.org and enable support (needs to be you).

Ignore the organization request from earlier, I thought that might have been a way to make it easier for me to maintain Travis stuff for you, but it seems I must be a full owner of the repo to set up travis on it. Other services (Coveralls, david-dm) allowed me to set up services just by being a collaborator, I think leaving it where it is will work fine and still require minimal drains on your time.

An unexpected problem occurred during redirect url parsing:
https://github.com/olalonde/follow-redirects/blame/master/index.js#L145

My definitions as follows :

let options = {
    host: 'proxy-host.com',
    port: 8080,
    method: 'GET',
    path: 'http://download.image.com/test.jpg',
    followRedirect: true,
    maxRedirects: 3
 };

This problem has led options.host and options.port failure after running redirect url passing. The new Host and Port in redirect url will be re-assigned to my options when I am using Host and Port as a proxy server.

Hi, I had a few questions:

1. If both agent (nodejs http/https option) and agents (follow-redirects option) are passed, then will the agent option be used ever in the main or redirected requests?

2. It seems cookies set with a 302 redirect (i.e. having both location and set-cookie headers) are not passed over to the following URL? For example, this httpbin test URL: https://httpbin.org/cookies/set?k1=v1&k2=v2 should set two cookies, and do a 302 redirect to https://httpbin.org/cookies, which simply emits the cookie values, but it shows none.

3. Looks like the request timeout is applied per redirection, not as an overall measure, which is somewhat confusing. For example, if timeout is set to 2000 ms, and 5 redirections occurred taking a total of 5000 ms, then no error is thrown, although it should. I think the timeout value (just like the native http/s modules) should be applied to the whole request.

Cheers.

I had some issues with redirects with axios that I'm now pretty sure are due to something in follow-redirects. Axios is currently using 0.0.7, but I upgraded my local copy to use 0.2.0 and still seeing the issue. This does not occur with node 5.12.0, but it does happen with node 6.3.1.

The crux of it seems to be that I'm hitting an endpoint in my node app that is responding with a 301, but then I get a 401 from the resulting Location.

I set DEBUG=follow-redirects but did not get any more useful info. I then added this line:

debug('making request with', options); 

right here to log every outgoing request, and then I get the following output:

6:02:30 PM web.1  |  Thu, 18 Aug 2016 01:02:30 GMT follow-redirects making request with { maxRedirects: 5,
6:02:30 PM web.1  |    protocol: 'https:',
6:02:30 PM web.1  |    hostname: 'postgres-api.heroku.com',
6:02:30 PM web.1  |    port: null,
6:02:30 PM web.1  |    path: '/client/v11/databases/dozing-patiently-6054/transfers',
6:02:30 PM web.1  |    method: 'get',
6:02:30 PM web.1  |    headers:
6:02:30 PM web.1  |     { Accept: 'application/json, text/plain, */*',
6:02:30 PM web.1  |       'User-Agent': 'herokudata' },
6:02:30 PM web.1  |    agent: undefined,
6:02:30 PM web.1  |    auth: '[email protected]:<redacted>' }
6:02:31 PM web.1  |  Thu, 18 Aug 2016 01:02:31 GMT follow-redirects redirecting to https://postgres-api.heroku.com/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers
6:02:31 PM web.1  |  Thu, 18 Aug 2016 01:02:31 GMT follow-redirects making request with { maxRedirects: 5,
6:02:31 PM web.1  |    protocol: 'https:',
6:02:31 PM web.1  |    hostname: 'postgres-api.heroku.com',
6:02:31 PM web.1  |    port: null,
6:02:31 PM web.1  |    path: '/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers',
6:02:31 PM web.1  |    method: 'get',
6:02:31 PM web.1  |    headers:
6:02:31 PM web.1  |     { Accept: 'application/json, text/plain, */*',
6:02:31 PM web.1  |       'User-Agent': 'herokudata' },
6:02:31 PM web.1  |    agent: undefined,
6:02:31 PM web.1  |    auth: null,
6:02:31 PM web.1  |    nativeProtocol:
6:02:31 PM web.1  |     { Server: { [Function: Server] super_: [Object] },
6:02:31 PM web.1  |       createServer: [Function],
6:02:31 PM web.1  |       globalAgent:
6:02:31 PM web.1  |        Agent {
6:02:31 PM web.1  |          domain: null,
6:02:31 PM web.1  |          _events: [Object],
6:02:31 PM web.1  |          _eventsCount: 1,
6:02:31 PM web.1  |          _maxListeners: undefined,
6:02:31 PM web.1  |          defaultPort: 443,
6:02:31 PM web.1  |          protocol: 'https:',
6:02:31 PM web.1  |          options: [Object],
6:02:31 PM web.1  |          requests: {},
6:02:31 PM web.1  |          sockets: [Object],
6:02:31 PM web.1  |          freeSockets: {},
6:02:31 PM web.1  |          keepAliveMsecs: 1000,
6:02:31 PM web.1  |          keepAlive: false,
6:02:31 PM web.1  |          maxSockets: Infinity,
6:02:31 PM web.1  |          maxFreeSockets: 256,
6:02:31 PM web.1  |          maxCachedSessions: 100,
6:02:31 PM web.1  |          _sessionCache: [Object] },
6:02:31 PM web.1  |       Agent: { [Function: Agent] super_: [Object] },
6:02:31 PM web.1  |       request: [Function],
6:02:31 PM web.1  |       get: [Function] },
6:02:31 PM web.1  |    defaultRequest: [Function: defaultMakeRequest],
6:02:31 PM web.1  |    slashes: true,
6:02:31 PM web.1  |    host: 'postgres-api.heroku.com',
6:02:31 PM web.1  |    hash: null,
6:02:31 PM web.1  |    search: null,
6:02:31 PM web.1  |    query: null,
6:02:31 PM web.1  |    pathname: '/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers',
6:02:31 PM web.1  |    href: 'https://postgres-api.heroku.com/client/v11/apps/ac07e31a-710c-470e-831e-e97bfc88bcd3/transfers' }

Note that the first request has the correct auth field, but the subsequent request has auth: null. Is this a node 6 breaking change? A bug in follow-redirects? It looks like this change could be related, but as per the logs above, the host is not changing.

I have a fairly simple case where I'm building a tarball-stream on the fly and piping it to a request that's going through follow-redirects.

It almost immediately crashes with this error:

Error: write after end
    at ClientRequest.OutgoingMessage.write (_http_outgoing.js:439:15)
    at Writable.RedirectableRequest._write (/home/espenh/webdev/sanity/cli/node_modules/follow-redirects/index.js:170:23)
    at doWrite (_stream_writable.js:333:12)
    at clearBuffer (_stream_writable.js:440:7)
    at onwrite (_stream_writable.js:372:7)
    at WritableState.onwrite (_stream_writable.js:89:5)
    at CorkedRequest.finish (_stream_writable.js:547:7)
    at afterWrite (_stream_writable.js:387:3)
    at onwrite (_stream_writable.js:378:7)
    at TLSSocket.WritableState.onwrite (_stream_writable.js:89:5)

I'll see if I can find time to write a reduced test case at some point, but don't have time right at this moment. Switching to regular HTTP/HTTPS for this request fixes the problem.

TypeError: Object function Object() { [native code] } has no method 'assign'
at Object.wrappedProtocol.request (/Users/james/Documents/D DRIVE/ES-SOURCE-CODE/TEST/ENHANCEMENT/Tagging/node_modules/follow-redirects/index.js:199:21)

Would be useful to have a way to know if the url you passed redirected to another or not.

I want to send send custom headers as I request a url.

I am making a fork of this project for use with IoTjs (which is like a light version of nodejs for embedded environments).
This fork may be used fo rcommercial purposes.
Going by the LICENSE file in the repo, its is perfectly acceptable for me to do so as long as I keep original LICENSE file right?

Hi,
I was googling about an issue which i'm facing at my end about the Redirect URL and found your blog "http://syskall.com/how-to-follow-http-redirects-in-node-dot-js/" . Indeed it is an useful blog. However the problem at my end was

When I use the API end points manually with Postman (with interceptors turned ON) along with body and headers, I was able to get 302 redirects. However for my API Automation testing purpose I thought of using

request() // npm request

which is as follows. But always I get the response as 500. My objective of my test is to get the response.headers when I get the status code as 302 from response. Can you pls throw some light and provide me some direction as I got struck on this

request(
url : 'https://blah/blah'
client_id :'something.com.au'
redirect_uri :'http://something' // this is a dummy one as it is not required for mobile app
response_type :'code'
scope : 'ALLAPI'
Username : 'cxxx'
Password : 'ccxcxc'
headers : {
'Content-Type': 'application/x-www-form-urlencoded',
'followRedirect' : true
}
Thanks,
Brad

I am trying below piece of code as you said.

var http = require('follow-redirects').http;
var https = require('follow-redirects').https;

var opts = {
		host: "bitly.com",
		path:"/UHfDGO", 
		port: 80};

http.get(opts, function (response) {
  response.on('data', function (chunk) {
    console.log(chunk);
  });
}).on('error', function (err) {
  console.error(err);
});

But when I execute it I get

vikas@vikas-pc:~/NodeJS Practice$ node trialnerror.js 
<Buffer 55 73 65 72 2d 61 67 65 6e 74 3a 20 2a 0a 44 69 73 61 6c 6c 6f 77 3a 20 2f 6c 69 74 65 0a 44 69 73 61 6c 6c 6f 77 3a 20 2f 68 74 6d 6c 0a 0a 23 20 4e ... >

Instead, I need to get entire html page content

It looks to me like if I e.g. do a file upload using this module (i.e. it won't be redirected) the entire file would be buffered in memory? If I have a file of several GB this could be a problem? Should we warn for this and possibly add a method + "content-length" check?