1
0
Fork 0

v1.1.0: Progress events, response timeouts, actually usable errors, namespaced debug information, documentation fixes

master
Sven Slootweg 9 years ago
parent 1fdaac9099
commit d1d5e4b353

2
.gitignore vendored

@ -2,3 +2,5 @@
# https://help.github.com/articles/ignoring-files # https://help.github.com/articles/ignoring-files
# Example .gitignore files: https://github.com/github/gitignore # Example .gitignore files: https://github.com/github/gitignore
/node_modules/ /node_modules/
/test-exit2.js
/test-exit.coffee

@ -2,11 +2,13 @@
A sane HTTP client library for Node.js with Streams2 support. A sane HTTP client library for Node.js with Streams2 support.
![](//img.shields.io/gratipay/joepie91.svg)
## Why bhttp? ## Why bhttp?
There are already a few commonly used HTTP client libraries for Node.js, but all of them have issues: There are already a few commonly used HTTP client libraries for Node.js, but all of them have issues:
* The core `http` module is rather low-level, and even relatively simple requests take a lot of work to make correctly. It also automatically uses an agent for HTTP requests, which slows down concurrent HTTP requests when you're streaming the responses somewhere. * The core `http` module is rather low-level, and even relatively simple requests take a lot of work to make correctly. It also automatically uses a limited amount of agents for HTTP requests (in Node.js 0.10), which slows down concurrent HTTP requests when you're streaming the responses somewhere.
* `request` is buggy, only supports old-style streams, has the same 'agent' problem as `http`, the documentation is poor, and the API is not very intuitive. * `request` is buggy, only supports old-style streams, has the same 'agent' problem as `http`, the documentation is poor, and the API is not very intuitive.
* `needle` is a lot simpler, but suffers from the same 'agent' problem, and the API can be a bit annoying in some ways. It also doesn't have a proper session API. * `needle` is a lot simpler, but suffers from the same 'agent' problem, and the API can be a bit annoying in some ways. It also doesn't have a proper session API.
* `hyperquest` (mostly) solves the 'agent' problem correctly, but has a very spartan API. Making non-GET requests is more complex than it should be. * `hyperquest` (mostly) solves the 'agent' problem correctly, but has a very spartan API. Making non-GET requests is more complex than it should be.
@ -21,10 +23,13 @@ All these issues (and more) are solved in `bhttp`. It offers the following:
* Easy-to-use session mechanics - a new session will automatically give you a new cookie jar, cookies are kept track of automatically, and 'default options' are deep-merged. * Easy-to-use session mechanics - a new session will automatically give you a new cookie jar, cookies are kept track of automatically, and 'default options' are deep-merged.
* Streaming requests are kept out of the agent pool - ie. no blocking of other requests. * Streaming requests are kept out of the agent pool - ie. no blocking of other requests.
* Optionally, a Promises API (you can also use nodebacks). * Optionally, a Promises API (you can also use nodebacks).
* Progress events! For both uploading and downloading.
## Caveats ## Caveats
* `bhttp` does not yet use a HTTPS-capable agent. This means that all SSL-related options are currently ignored (per Node.js `http` documentation). If you need secure HTTPS requests, make sure to specify a custom agent! `bhttp` does not yet use a HTTPS-capable agent. This means that all SSL-related options are currently ignored by default (per Node.js `http` documentation).
__This does *not* mean that you cannot use `bhttp` for HTTPS requests!__ If you need secure HTTPS requests, just make sure to specify a [custom `https` agent](https://nodejs.org/api/https.html#https_class_https_agent).
## License ## License
@ -106,6 +111,44 @@ bhttp.get("http://somesite.com/bigfile.mp4", {stream: true}, function(err, respo
}) })
``` ```
### Progress events
Upload progress events:
```javascript
var Promise = require("bluebird");
var bhttp = require("bhttp");
Promise.try(function() {
return bhttp.post("http://somehostingservice.com/upload", {
file: fs.createReadStream("./bigfile.mkv")
}, {
onUploadProgress: function(completedBytes, totalBytes) {
console.log("Upload progress:", (completedBytes / totalBytes * 100), "%");
}
});
}).then(function(response) {
console.log("Response from hosting service:", response.body.toString());
});
```
Download progress events:
```javascript
var Promise = require("bluebird");
var bhttp = require("bhttp");
Promise.try(function() {
return bhttp.get("http://somehostingservice.com/bigfile.mkv", {stream: true});
}).then(function(response) {
response.on("progress", function(completedBytes, totalBytes) {
console.log("Download progress:", (completedBytes / totalBytes * 100), "%");
});
response.pipe(fs.createWriteStream("./bigfile.mkv"));
});
```
### Sessions ### Sessions
```javascript ```javascript
@ -125,6 +168,8 @@ Promise.try(function(){
## API ## API
The various error types are documented at the bottom of this README.
### bhttp.head(url, [options, [callback]]) ### bhttp.head(url, [options, [callback]])
### bhttp.get(url, [options, [callback]]) ### bhttp.get(url, [options, [callback]])
### bhttp.delete(url, [options, [callback]]) ### bhttp.delete(url, [options, [callback]])
@ -149,6 +194,8 @@ The `data` payload can be one of the following things:
Makes a request, and returns the response object asynchronously. The response object is a standard `http.IncomingMessages` with a few additional properties (documented below the argument list). Makes a request, and returns the response object asynchronously. The response object is a standard `http.IncomingMessages` with a few additional properties (documented below the argument list).
Note that (progress) event handlers must be specified in the `options` or (in the case of download progress events) as an event listener on the response object - as `bhttp` uses Promises, it is not technically possible to return an EventEmitter.
* __url__: The URL to request, with protocol. When using HTTPS, please be sure to read the 'Caveats' section. * __url__: The URL to request, with protocol. When using HTTPS, please be sure to read the 'Caveats' section.
* __options__: *Optional.* Extra options for the request. Any other options not listed here will be passed on directly to the `http` or `https` module. * __options__: *Optional.* Extra options for the request. Any other options not listed here will be passed on directly to the `http` or `https` module.
* __Basic options__ * __Basic options__
@ -169,10 +216,14 @@ Makes a request, and returns the response object asynchronously. The response ob
* __Advanced options__ * __Advanced options__
* __method__: The request method to use. You don't need this when using the shorthand methods. * __method__: The request method to use. You don't need this when using the shorthand methods.
* __cookieJar__: A custom cookie jar to use. You'll probably want to use `bhttp.session()` instead. * __cookieJar__: A custom cookie jar to use. You'll probably want to use `bhttp.session()` instead.
* __responseTimeout__: The timeout, in milliseconds, after which the request should be considered to have failed if no response is received yet. Note that this measures from the start of the request to the start of the response, and is *not* a connection timeout. If a timeout occurs, a ResponseTimeoutError will be thrown asynchronously (see error documentation below).
* __allowChunkedMultipart__: *Defaults to `false`.* Many servers don't support `multipart/form-data` when it is transmitted with chunked transfer encoding (eg. when the stream length is unknown), and silently fail with an empty request payload - this is why `bhttp` disallows it by default. If you are *absolutely certain* that the endpoint supports this functionality, you can override the behaviour by setting this to `true`. * __allowChunkedMultipart__: *Defaults to `false`.* Many servers don't support `multipart/form-data` when it is transmitted with chunked transfer encoding (eg. when the stream length is unknown), and silently fail with an empty request payload - this is why `bhttp` disallows it by default. If you are *absolutely certain* that the endpoint supports this functionality, you can override the behaviour by setting this to `true`.
* __discardResponse__: *Defaults to `false`.* Whether to throw away the response without reading it. Only really useful for fire-and-forget calls. This is almost never what you want. * __discardResponse__: *Defaults to `false`.* Whether to throw away the response without reading it. Only really useful for fire-and-forget calls. This is almost never what you want.
* __keepRedirectResponses__: *Defaults to `false`.* Whether to keep the response streams of redirects. You probably don't need this. __When enabling this, you must *explicitly* read out every single redirect response, or you will experience memory leaks!__ * __keepRedirectResponses__: *Defaults to `false`.* Whether to keep the response streams of redirects. You probably don't need this. __When enabling this, you must *explicitly* read out every single redirect response, or you will experience memory leaks!__
* __justPrepare__: *Defaults to `false`.* When set to `true`, bhttp just prepares the request, and doesn't actually carry it out; useful if you want to make some manual modifications. Instead of a response, the method will asynchronously return an array with the signature `[request, response, requestState]` that you will need to pass into the `bhttp.makeRequest()` method. * __justPrepare__: *Defaults to `false`.* When set to `true`, bhttp just prepares the request, and doesn't actually carry it out; useful if you want to make some manual modifications. Instead of a response, the method will asynchronously return an array with the signature `[request, response, requestState]` that you will need to pass into the `bhttp.makeRequest()` method.
* __Event handlers__
* __onUploadProgress__: A callback to call for upload progress events (this covers both input streams and form data). The callback signature is `(completedBytes, totalBytes, request)`. If the total size is not known, `totalBytes` will be `undefined`. The `request` variable will hold the request object that the progress event applies to - this is relevant when dealing with automatic redirect following, where multiple requests may occur.
* __onDownloadProgress__: A callback to call for download progress events. The callback signature is `(completedBytes, totalBytes, response)`. If the total size is not known, `totalBytes` will be `undefined`. The `response` variable will hold the response object that the progress event applies to - this is relevant when dealing with automatic redirect following, where multiple responses may occur. *Note that using the `progress` event on a response object is usually a more practical option!*
* __callback__: *Optional.* When using the nodeback API, the callback to use. If not specified, a Promise will be returned. * __callback__: *Optional.* When using the nodeback API, the callback to use. If not specified, a Promise will be returned.
@ -183,6 +234,10 @@ A few extra properties are set on the response object (which is a `http.Incoming
* __request__: The request configuration that was generated by `bhttp`. You probably don't need this. * __request__: The request configuration that was generated by `bhttp`. You probably don't need this.
* __requestState__: The request state that was accumulated by `bhttp`. You probably don't need this. * __requestState__: The request state that was accumulated by `bhttp`. You probably don't need this.
Additionally, there's an extra event on the `response` object:
* __'progress' (completedBytes, totalBytes)__: The 'download progress' for the response body. This works the same as the `onDownloadProgress` option, except the event will be specific to this response, and it allows for somewhat nicer syntax. Make sure to attach this handler *before* you start reading the response stream!
`bhttp` can automatically parse the metadata for the following types of streams: `bhttp` can automatically parse the metadata for the following types of streams:
* `fs` streams * `fs` streams
@ -190,7 +245,7 @@ A few extra properties are set on the response object (which is a `http.Incoming
* `request` requests * `request` requests
* `combined-stream` streams (assuming all the underlying streams are of one of the types listed here) * `combined-stream` streams (assuming all the underlying streams are of one of the types listed here)
If you are using a different type of streams, you can wrap the stream using `bhttp.wrapStream` to manually specify the needed metadata. If you are using a different type of stream, you can wrap the stream using `bhttp.wrapStream` to manually specify the needed metadata.
### bhttp.session([defaultOptions]) ### bhttp.session([defaultOptions])
@ -213,3 +268,51 @@ The resulting wrapper can be passed on to the `bhttp` methods as if it were a re
### bhttp.makeRequest(request, response, requestState) ### bhttp.makeRequest(request, response, requestState)
When using the `justPrepare` option, you can use this method to proceed with the request after manual modifications. The function signature is identical to the signature of the array returned when using `justPrepare`. `response` will usually be `null`, but must be passed on as is, to account for future API changes. When using the `justPrepare` option, you can use this method to proceed with the request after manual modifications. The function signature is identical to the signature of the array returned when using `justPrepare`. `response` will usually be `null`, but must be passed on as is, to account for future API changes.
## Error types
All these correctly extend the `Error` class - this means that you can use them as a `.catch` predicate when using Promises, and that you can use `instanceof` on them when using the nodeback API.
### bhttp.bhttpError
The base class for all errors generated by `bhttp`. You usually don't need this.
### bhttp.ConflictingOptionsError
You have specified two or more request options that cannot be used together.
The error message will contain more details.
### bhttp.UnsupportedProtocolError
You tried to load a URL that isn't using either the HTTP or HTTPS protocol. Only HTTP and HTTPS are currently supported.
### bhttp.RedirectError
A redirect was encountered that could not be followed.
This could be because the redirect limit was reached, or because the HTTP specification doesn't allow automatic following of the redirect that was encountered.
The error message will contain more details.
### bhttp.MultipartError
Something went wrong while generating the multipart/form-data stream.
Currently, this will only be thrown if you try to use chunked transfer encoding for a multipart stream - a common situation where this can occur, is when you pass in streams with an unknown length.
To resolve this error, you must either explicitly specify the length of the streams using `bhttp.wrapStream` or, if the target server supports it, enable the `allowChunkedMultipart` option.
### bhttp.ConnectionTimeoutError
The connection timed out.
The connection timeout is defined by the operating system, and cannot currently be overridden.
### bhttp.ResponseTimeoutError
The response timed out.
The response timeout can be specified using the `responseTimeout` option, and it is measured from the start of the request to the start of the response. If no response is received within the `responseTimeout`, a `ResponseTimeoutError` will be thrown asynchronously, and the request will be aborted.
__You should not set a `responseTimeout` for requests that involve large file uploads!__ Because a response can only be received *after* the request has completed, any file/stream upload that takes longer than the `responseTimeout`, will result in a `ResponseTimeoutError`.

@ -17,37 +17,59 @@ _ = require "lodash"
S = require "string" S = require "string"
formFixArray = require "form-fix-array" formFixArray = require "form-fix-array"
errors = require "errors" errors = require "errors"
debug = require("debug")("bhttp") debug = require("debug")
debugRequest = debug("bhttp:request")
debugResponse = debug("bhttp:response")
extend = require "extend"
# Other third-party modules # Other third-party modules
formData = require "form-data2" formData = require "form-data2"
concatStream = require "concat-stream" concatStream = require "concat-stream"
toughCookie = require "tough-cookie" toughCookie = require "tough-cookie"
streamLength = require "stream-length" streamLength = require "stream-length"
sink = require "through2-sink"
spy = require "through2-spy"
# For the version in the user agent, etc. # For the version in the user agent, etc.
packageConfig = require "../package.json" packageConfig = require "../package.json"
bhttpErrors = {}
# Error types # Error types
errors.create errors.create
name: "bhttpError" name: "bhttpError"
scope: bhttpErrors
errors.create errors.create
name: "ConflictingOptionsError" name: "ConflictingOptionsError"
parents: errors.bhttpError parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create errors.create
name: "UnsupportedProtocolError" name: "UnsupportedProtocolError"
parents: errors.bhttpError parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create errors.create
name: "RedirectError" name: "RedirectError"
parents: errors.bhttpError parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create errors.create
name: "MultipartError" name: "MultipartError"
parents: errors.bhttpError parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create
name: "ConnectionTimeoutError"
parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create
name: "ResponseTimeoutError"
parents: bhttpErrors.bhttpError
scope: bhttpErrors
# Utility functions # Utility functions
@ -69,7 +91,7 @@ isStream = (obj) ->
# Middleware # Middleware
# NOTE: requestState is an object that signifies the current state of the overall request; eg. for a response involving one or more redirects, it will hold a 'redirect history'. # NOTE: requestState is an object that signifies the current state of the overall request; eg. for a response involving one or more redirects, it will hold a 'redirect history'.
prepareSession = (request, response, requestState) -> prepareSession = (request, response, requestState) ->
debug "preparing session" debugRequest "preparing session"
Promise.try -> Promise.try ->
if requestState.sessionOptions? if requestState.sessionOptions?
# Request options take priority over session options # Request options take priority over session options
@ -92,14 +114,14 @@ prepareSession = (request, response, requestState) ->
# Get the current cookie string for the URL # Get the current cookie string for the URL
request.cookieJar.get request.url request.cookieJar.get request.url
.then (cookieString) -> .then (cookieString) ->
debug "sending cookie string: %s", cookieString debugRequest "sending cookie string: %s", cookieString
request.options.headers["cookie"] = cookieString request.options.headers["cookie"] = cookieString
Promise.resolve [request, response, requestState] Promise.resolve [request, response, requestState]
else else
Promise.resolve [request, response, requestState] Promise.resolve [request, response, requestState]
prepareDefaults = (request, response, requestState) -> prepareDefaults = (request, response, requestState) ->
debug "preparing defaults" debugRequest "preparing defaults"
Promise.try -> Promise.try ->
# These are the options that we need for response processing, but don't need to be passed on to the http/https module. # These are the options that we need for response processing, but don't need to be passed on to the http/https module.
request.responseOptions = request.responseOptions =
@ -111,6 +133,8 @@ prepareDefaults = (request, response, requestState) ->
stream: request.options.stream ? false stream: request.options.stream ? false
justPrepare: request.options.justPrepare ? false justPrepare: request.options.justPrepare ? false
redirectLimit: request.options.redirectLimit ? 10 redirectLimit: request.options.redirectLimit ? 10
onDownloadProgress: request.options.onDownloadProgress
responseTimeout: request.options.responseTimeout
# Whether chunked transfer encoding for multipart/form-data payloads is acceptable. This is likely to break quietly on a lot of servers. # Whether chunked transfer encoding for multipart/form-data payloads is acceptable. This is likely to break quietly on a lot of servers.
request.options.allowChunkedMultipart ?= false request.options.allowChunkedMultipart ?= false
@ -127,7 +151,7 @@ prepareDefaults = (request, response, requestState) ->
Promise.resolve [request, response, requestState] Promise.resolve [request, response, requestState]
prepareUrl = (request, response, requestState) -> prepareUrl = (request, response, requestState) ->
debug "preparing URL" debugRequest "preparing URL"
Promise.try -> Promise.try ->
# Parse the specified URL, and use the resulting information to build a complete `options` object # Parse the specified URL, and use the resulting information to build a complete `options` object
urlOptions = urlUtil.parse request.url, true urlOptions = urlUtil.parse request.url, true
@ -139,7 +163,7 @@ prepareUrl = (request, response, requestState) ->
Promise.resolve [request, response, requestState] Promise.resolve [request, response, requestState]
prepareProtocol = (request, response, requestState) -> prepareProtocol = (request, response, requestState) ->
debug "preparing protocol" debugRequest "preparing protocol"
Promise.try -> Promise.try ->
request.protocolModule = switch request.protocol request.protocolModule = switch request.protocol
when "http" then http when "http" then http
@ -147,7 +171,7 @@ prepareProtocol = (request, response, requestState) ->
else null else null
if not request.protocolModule? if not request.protocolModule?
return Promise.reject() new errors.UnsupportedProtocolError "The protocol specified (#{protocol}) is not currently supported by this module." return Promise.reject() new bhttpErrors.UnsupportedProtocolError "The protocol specified (#{protocol}) is not currently supported by this module."
request.options.port ?= switch request.protocol request.options.port ?= switch request.protocol
when "http" then 80 when "http" then 80
@ -156,14 +180,14 @@ prepareProtocol = (request, response, requestState) ->
Promise.resolve [request, response, requestState] Promise.resolve [request, response, requestState]
prepareOptions = (request, response, requestState) -> prepareOptions = (request, response, requestState) ->
debug "preparing options" debugRequest "preparing options"
Promise.try -> Promise.try ->
# Do some sanity checks - there are a number of options that cannot be used together # Do some sanity checks - there are a number of options that cannot be used together
if (request.options.formFields? or request.options.files?) and (request.options.inputStream? or request.options.inputBuffer?) if (request.options.formFields? or request.options.files?) and (request.options.inputStream? or request.options.inputBuffer?)
return Promise.reject addErrorData(new errors.ConflictingOptionsError("You cannot define both formFields/files and a raw inputStream or inputBuffer."), request, response, requestState) return Promise.reject addErrorData(new bhttpErrors.ConflictingOptionsError("You cannot define both formFields/files and a raw inputStream or inputBuffer."), request, response, requestState)
if request.options.encodeJSON and (request.options.inputStream? or request.options.inputBuffer?) if request.options.encodeJSON and (request.options.inputStream? or request.options.inputBuffer?)
return Promise.reject addErrorData(new errors.ConflictingOptionsError("You cannot use both encodeJSON and a raw inputStream or inputBuffer.", undefined, "If you meant to JSON-encode the stream, you will currently have to do so manually."), request, response, requestState) return Promise.reject addErrorData(new bhttpErrors.ConflictingOptionsError("You cannot use both encodeJSON and a raw inputStream or inputBuffer.", undefined, "If you meant to JSON-encode the stream, you will currently have to do so manually."), request, response, requestState)
# If the user plans on streaming the response, we need to disable the agent entirely - otherwise the streams will block the pool. # If the user plans on streaming the response, we need to disable the agent entirely - otherwise the streams will block the pool.
if request.responseOptions.stream if request.responseOptions.stream
@ -172,8 +196,11 @@ prepareOptions = (request, response, requestState) ->
Promise.resolve [request, response, requestState] Promise.resolve [request, response, requestState]
preparePayload = (request, response, requestState) -> preparePayload = (request, response, requestState) ->
debug "preparing payload" debugRequest "preparing payload"
Promise.try -> Promise.try ->
# Persist the download progress event handler on the request object, if there is one.
request.onUploadProgress = request.options.onUploadProgress
# If a 'files' parameter is present, then we will send the form data as multipart data - it's most likely binary data. # If a 'files' parameter is present, then we will send the form data as multipart data - it's most likely binary data.
multipart = request.options.forceMultipart or request.options.files? multipart = request.options.forceMultipart or request.options.files?
@ -188,13 +215,13 @@ preparePayload = (request, response, requestState) ->
containsStreams = _.any request.options.formFields, (item) -> isStream(item) containsStreams = _.any request.options.formFields, (item) -> isStream(item)
if request.options.encodeJSON and containsStreams if request.options.encodeJSON and containsStreams
return Promise.reject() new errors.ConflictingOptionsError "Sending a JSON-encoded payload containing data from a stream is not currently supported.", undefined, "Either don't use encodeJSON, or read your stream into a string or Buffer." return Promise.reject() new bhttpErrors.ConflictingOptionsError "Sending a JSON-encoded payload containing data from a stream is not currently supported.", undefined, "Either don't use encodeJSON, or read your stream into a string or Buffer."
if request.options.method in ["post", "put", "patch"] if request.options.method in ["post", "put", "patch"]
# Prepare the payload, and set the appropriate headers. # Prepare the payload, and set the appropriate headers.
if (request.options.encodeJSON or request.options.formFields?) and not multipart if (request.options.encodeJSON or request.options.formFields?) and not multipart
# We know the payload and its size in advance. # We know the payload and its size in advance.
debug "got url-encodable form-data" debugRequest "got url-encodable form-data"
request.options.headers["content-type"] = "application/x-www-form-urlencoded" request.options.headers["content-type"] = "application/x-www-form-urlencoded"
if request.options.encodeJSON if request.options.encodeJSON
@ -210,7 +237,7 @@ preparePayload = (request, response, requestState) ->
return Promise.resolve() return Promise.resolve()
else if request.options.formFields? and multipart else if request.options.formFields? and multipart
# This is going to be multipart data, and we'll let `form-data` set the headers for us. # This is going to be multipart data, and we'll let `form-data` set the headers for us.
debug "got multipart form-data" debugRequest "got multipart form-data"
formDataObject = new formData() formDataObject = new formData()
for fieldName, fieldValue of formFixArray(request.options.formFields) for fieldName, fieldValue of formFixArray(request.options.formFields)
@ -238,7 +265,7 @@ preparePayload = (request, response, requestState) ->
Promise.resolve() Promise.resolve()
else if request.options.inputStream? else if request.options.inputStream?
# A raw inputStream was provided, just leave it be. # A raw inputStream was provided, just leave it be.
debug "got inputStream" debugRequest "got inputStream"
Promise.try -> Promise.try ->
request.payloadStream = request.options.inputStream request.payloadStream = request.options.inputStream
@ -247,20 +274,20 @@ preparePayload = (request, response, requestState) ->
else else
streamLength request.options.inputStream streamLength request.options.inputStream
.then (length) -> .then (length) ->
debug "length for inputStream is %s", length debugRequest "length for inputStream is %s", length
request.options.headers["content-length"] = length request.options.headers["content-length"] = length
.catch (err) -> .catch (err) ->
debug "unable to determine inputStream length, switching to chunked transfer encoding" debugRequest "unable to determine inputStream length, switching to chunked transfer encoding"
request.options.headers["content-transfer-encoding"] = "chunked" request.options.headers["content-transfer-encoding"] = "chunked"
else if request.options.inputBuffer? else if request.options.inputBuffer?
# A raw inputBuffer was provided, just leave it be (but make sure it's an actual Buffer). # A raw inputBuffer was provided, just leave it be (but make sure it's an actual Buffer).
debug "got inputBuffer" debugRequest "got inputBuffer"
if typeof request.options.inputBuffer == "string" if typeof request.options.inputBuffer == "string"
request.payload = new Buffer(request.options.inputBuffer) # Input string should be utf-8! request.payload = new Buffer(request.options.inputBuffer) # Input string should be utf-8!
else else
request.payload = request.options.inputBuffer request.payload = request.options.inputBuffer
debug "length for inputBuffer is %s", request.payload.length debugRequest "length for inputBuffer is %s", request.payload.length
request.options.headers["content-length"] = request.payload.length request.options.headers["content-length"] = request.payload.length
return Promise.resolve() return Promise.resolve()
@ -272,10 +299,10 @@ preparePayload = (request, response, requestState) ->
Promise.resolve [request, response, requestState] Promise.resolve [request, response, requestState]
prepareCleanup = (request, response, requestState) -> prepareCleanup = (request, response, requestState) ->
debug "preparing cleanup" debugRequest "preparing cleanup"
Promise.try -> Promise.try ->
# Remove the options that we're not going to pass on to the actual http/https library. # Remove the options that we're not going to pass on to the actual http/https library.
delete request.options[key] for key in ["query", "formFields", "files", "encodeJSON", "inputStream", "inputBuffer", "discardResponse", "keepRedirectResponses", "followRedirects", "noDecode", "decodeJSON", "allowChunkedMultipart", "forceMultipart"] delete request.options[key] for key in ["query", "formFields", "files", "encodeJSON", "inputStream", "inputBuffer", "discardResponse", "keepRedirectResponses", "followRedirects", "noDecode", "decodeJSON", "allowChunkedMultipart", "forceMultipart", "onUploadProgress", "onDownloadProgress"]
# Lo-Dash apparently has no `map` equivalent for object keys...? # Lo-Dash apparently has no `map` equivalent for object keys...?
fixedHeaders = {} fixedHeaders = {}
@ -288,7 +315,7 @@ prepareCleanup = (request, response, requestState) ->
# The guts of the module # The guts of the module
prepareRequest = (request, response, requestState) -> prepareRequest = (request, response, requestState) ->
debug "preparing request" debugRequest "preparing request"
# FIXME: Mock httpd for testing functionality. # FIXME: Mock httpd for testing functionality.
Promise.try -> Promise.try ->
middlewareFunctions = [ middlewareFunctions = [
@ -310,41 +337,78 @@ prepareRequest = (request, response, requestState) ->
return promiseChain return promiseChain
makeRequest = (request, response, requestState) -> makeRequest = (request, response, requestState) ->
debug "making %s request to %s", request.options.method.toUpperCase(), request.url debugRequest "making %s request to %s", request.options.method.toUpperCase(), request.url
Promise.try -> Promise.try ->
# Instantiate a regular HTTP/HTTPS request # Instantiate a regular HTTP/HTTPS request
req = request.protocolModule.request request.options req = request.protocolModule.request request.options
# This is where we write our payload or stream to the request, and the actual request is made. timeoutTimer = null
if request.payload?
# The entire payload is a single Buffer.
debug "sending payload"
req.write request.payload
req.end()
else if request.payloadStream?
# The payload is a stream.
debug "piping payloadStream"
if request.payloadStream._bhttpStreamWrapper?
request.payloadStream.stream.pipe req
else
request.payloadStream.pipe req
else
# For GET, HEAD, DELETE, etc. there is no payload, but we still need to call end() to complete the request.
debug "closing request without payload"
req.end()
new Promise (resolve, reject) -> new Promise (resolve, reject) ->
# Connection timeout handling, if one is set.
if request.responseOptions.responseTimeout?
debugRequest "setting response timeout timer to #{request.responseOptions.responseTimeout}ms..."
req.on "socket", (socket) ->
timeoutHandler = ->
debugRequest "a response timeout occurred!"
req.abort()
reject addErrorData(new bhttpErrors.ResponseTimeoutError("The response timed out."))
timeoutTimer = setTimeout(timeoutHandler, request.responseOptions.responseTimeout)
# Set up the upload progress monitoring.
totalBytes = request.options.headers["content-length"]
completedBytes = 0
progressStream = spy (chunk) ->
completedBytes += chunk.length
req.emit "progress", completedBytes, totalBytes
if request.onUploadProgress?
req.on "progress", (completedBytes, totalBytes) ->
request.onUploadProgress(completedBytes, totalBytes, req)
# This is where we write our payload or stream to the request, and the actual request is made.
if request.payload?
# The entire payload is a single Buffer. We'll still pretend that it's a stream for our progress events, though, to provide a consistent API.
debugRequest "sending payload"
req.emit "progress", request.payload.length, request.payload.length
req.write request.payload
req.end()
else if request.payloadStream?
# The payload is a stream.
debugRequest "piping payloadStream"
if request.payloadStream._bhttpStreamWrapper?
request.payloadStream.stream
.pipe progressStream
.pipe req
else
request.payloadStream
.pipe progressStream
.pipe req
else
# For GET, HEAD, DELETE, etc. there is no payload, but we still need to call end() to complete the request.
debugRequest "closing request without payload"
req.end()
# In case something goes wrong during this process, somehow... # In case something goes wrong during this process, somehow...
req.on "error", (err) -> req.on "error", (err) ->
reject err if err.code == "ETIMEDOUT"
debugRequest "a connection timeout occurred!"
reject addErrorData(new bhttpErrors.ConnectionTimeoutError("The connection timed out."))
else
reject err
req.on "response", (res) -> req.on "response", (res) ->
if timeoutTimer?
debugResponse "got response in time, clearing response timeout timer"
clearTimeout(timeoutTimer)
resolve res resolve res
.then (response) -> .then (response) ->
Promise.resolve [request, response, requestState] Promise.resolve [request, response, requestState]
processResponse = (request, response, requestState) -> processResponse = (request, response, requestState) ->
debug "processing response, got status code %s", response.statusCode debugResponse "processing response, got status code %s", response.statusCode
# When we receive the response, we'll buffer it up and/or decode it, depending on what the user specified, and resolve the returned Promise. If the user just wants the raw stream, we resolve immediately after receiving a response. # When we receive the response, we'll buffer it up and/or decode it, depending on what the user specified, and resolve the returned Promise. If the user just wants the raw stream, we resolve immediately after receiving a response.
@ -352,7 +416,7 @@ processResponse = (request, response, requestState) ->
# First, if a cookie jar is set and we received one or more cookies from the server, we should store them in our cookieJar. # First, if a cookie jar is set and we received one or more cookies from the server, we should store them in our cookieJar.
if request.cookieJar? and response.headers["set-cookie"]? if request.cookieJar? and response.headers["set-cookie"]?
promises = for cookieHeader in response.headers["set-cookie"] promises = for cookieHeader in response.headers["set-cookie"]
debug "storing cookie: %s", cookieHeader debugResponse "storing cookie: %s", cookieHeader
request.cookieJar.set cookieHeader, request.url request.cookieJar.set cookieHeader, request.url
Promise.all promises Promise.all promises
else else
@ -365,7 +429,7 @@ processResponse = (request, response, requestState) ->
if response.statusCode in [301, 302, 303, 307] and request.responseOptions.followRedirects if response.statusCode in [301, 302, 303, 307] and request.responseOptions.followRedirects
if requestState.redirectHistory.length >= (request.responseOptions.redirectLimit - 1) if requestState.redirectHistory.length >= (request.responseOptions.redirectLimit - 1)
return Promise.reject addErrorData(new errors.RedirectError("The maximum amount of redirects ({request.responseOptions.redirectLimit}) was reached.")) return Promise.reject addErrorData(new bhttpErrors.RedirectError("The maximum amount of redirects ({request.responseOptions.redirectLimit}) was reached."))
# 301: For GET and HEAD, redirect unchanged. For POST, PUT, PATCH, DELETE, "ask user" (in our case: throw an error.) # 301: For GET and HEAD, redirect unchanged. For POST, PUT, PATCH, DELETE, "ask user" (in our case: throw an error.)
# 302: Redirect, change method to GET. # 302: Redirect, change method to GET.
@ -377,21 +441,38 @@ processResponse = (request, response, requestState) ->
when "get", "head" when "get", "head"
return redirectUnchanged request, response, requestState return redirectUnchanged request, response, requestState
when "post", "put", "patch", "delete" when "post", "put", "patch", "delete"
return Promise.reject addErrorData(new errors.RedirectError("Encountered a 301 redirect for POST, PUT, PATCH or DELETE. RFC says we can't automatically continue."), request, response, requestState) return Promise.reject addErrorData(new bhttpErrors.RedirectError("Encountered a 301 redirect for POST, PUT, PATCH or DELETE. RFC says we can't automatically continue."), request, response, requestState)
else else
return Promise.reject addErrorData(new errors.RedirectError("Encountered a 301 redirect, but not sure how to proceed for the #{request.options.method.toUpperCase()} method.")) return Promise.reject addErrorData(new bhttpErrors.RedirectError("Encountered a 301 redirect, but not sure how to proceed for the #{request.options.method.toUpperCase()} method."))
when 302, 303 when 302, 303
return redirectGet request, response, requestState return redirectGet request, response, requestState
when 307 when 307
if request.containsStreams and request.options.method not in ["get", "head"] if request.containsStreams and request.options.method not in ["get", "head"]
return Promise.reject addErrorData(new errors.RedirectError("Encountered a 307 redirect for POST, PUT or DELETE, but your payload contained (single-use) streams. We therefore can't automatically follow the redirect."), request, response, requestState) return Promise.reject addErrorData(new bhttpErrors.RedirectError("Encountered a 307 redirect for POST, PUT or DELETE, but your payload contained (single-use) streams. We therefore can't automatically follow the redirect."), request, response, requestState)
else else
return redirectUnchanged request, response, requestState return redirectUnchanged request, response, requestState
else if request.responseOptions.discardResponse else if request.responseOptions.discardResponse
response.resume() # Drain the response stream response.resume() # Drain the response stream
Promise.resolve response Promise.resolve response
else else
totalBytes = response.headers["content-length"]
if totalBytes? # Otherwise `undefined` will turn into `NaN`, and we don't want that.
totalBytes = parseInt(totalBytes)
completedBytes = 0
progressStream = sink (chunk) ->
completedBytes += chunk.length
response.emit "progress", completedBytes, totalBytes
if request.responseOptions.onDownloadProgress?
response.on "progress", (completedBytes, totalBytes) ->
request.responseOptions.onDownloadProgress(completedBytes, totalBytes, response)
new Promise (resolve, reject) -> new Promise (resolve, reject) ->
# This is slightly hacky, but returning a .pipe'd stream would make all the response object attributes unavailable to the end user. Therefore, 'branching' the stream into our progress monitoring stream is a much better option. We use .pause() to ensure the stream doesn't start flowing until the end user (or our library) has actually piped it into something.
response.pipe(progressStream)
response.pause()
if request.responseOptions.stream if request.responseOptions.stream
resolve response resolve response
else else
@ -428,7 +509,7 @@ doPayloadRequest = (url, data, options, callback) ->
@request url, options, callback @request url, options, callback
redirectGet = (request, response, requestState) -> redirectGet = (request, response, requestState) ->
debug "following forced-GET redirect to %s", response.headers["location"] debugResponse "following forced-GET redirect to %s", response.headers["location"]
Promise.try -> Promise.try ->
options = _.clone(requestState.originalOptions) options = _.clone(requestState.originalOptions)
options.method = "get" options.method = "get"
@ -438,7 +519,7 @@ redirectGet = (request, response, requestState) ->
doRedirect request, response, requestState, options doRedirect request, response, requestState, options
redirectUnchanged = (request, response, requestState) -> redirectUnchanged = (request, response, requestState) ->
debug "following same-method redirect to %s", response.headers["location"] debugResponse "following same-method redirect to %s", response.headers["location"]
Promise.try -> Promise.try ->
options = _.clone(requestState.originalOptions) options = _.clone(requestState.originalOptions)
doRedirect request, response, requestState, options doRedirect request, response, requestState, options
@ -541,6 +622,8 @@ bhttpAPI =
options: options options: options
} }
extend(bhttpAPI, bhttpErrors)
module.exports = bhttpAPI module.exports = bhttpAPI
# That's all, folks! # That's all, folks!

@ -1,4 +1,4 @@
var Promise, S, addErrorData, bhttpAPI, concatStream, createCookieJar, debug, doPayloadRequest, doRedirect, errors, formData, formFixArray, http, https, isStream, makeRequest, ofTypes, packageConfig, prepareCleanup, prepareDefaults, prepareOptions, preparePayload, prepareProtocol, prepareRequest, prepareSession, prepareUrl, processResponse, querystring, redirectGet, redirectUnchanged, stream, streamLength, toughCookie, urlUtil, util, _; var Promise, S, addErrorData, bhttpAPI, bhttpErrors, concatStream, createCookieJar, debug, debugRequest, debugResponse, doPayloadRequest, doRedirect, errors, extend, formData, formFixArray, http, https, isStream, makeRequest, ofTypes, packageConfig, prepareCleanup, prepareDefaults, prepareOptions, preparePayload, prepareProtocol, prepareRequest, prepareSession, prepareUrl, processResponse, querystring, redirectGet, redirectUnchanged, sink, spy, stream, streamLength, toughCookie, urlUtil, util, _;
urlUtil = require("url"); urlUtil = require("url");
@ -22,7 +22,13 @@ formFixArray = require("form-fix-array");
errors = require("errors"); errors = require("errors");
debug = require("debug")("bhttp"); debug = require("debug");
debugRequest = debug("bhttp:request");
debugResponse = debug("bhttp:response");
extend = require("extend");
formData = require("form-data2"); formData = require("form-data2");
@ -32,30 +38,53 @@ toughCookie = require("tough-cookie");
streamLength = require("stream-length"); streamLength = require("stream-length");
sink = require("through2-sink");
spy = require("through2-spy");
packageConfig = require("../package.json"); packageConfig = require("../package.json");
bhttpErrors = {};
errors.create({ errors.create({
name: "bhttpError" name: "bhttpError",
scope: bhttpErrors
}); });
errors.create({ errors.create({
name: "ConflictingOptionsError", name: "ConflictingOptionsError",
parents: errors.bhttpError parents: bhttpErrors.bhttpError,
scope: bhttpErrors
}); });
errors.create({ errors.create({
name: "UnsupportedProtocolError", name: "UnsupportedProtocolError",
parents: errors.bhttpError parents: bhttpErrors.bhttpError,
scope: bhttpErrors
}); });
errors.create({ errors.create({
name: "RedirectError", name: "RedirectError",
parents: errors.bhttpError parents: bhttpErrors.bhttpError,
scope: bhttpErrors
}); });
errors.create({ errors.create({
name: "MultipartError", name: "MultipartError",
parents: errors.bhttpError parents: bhttpErrors.bhttpError,
scope: bhttpErrors
});
errors.create({
name: "ConnectionTimeoutError",
parents: bhttpErrors.bhttpError,
scope: bhttpErrors
});
errors.create({
name: "ResponseTimeoutError",
parents: bhttpErrors.bhttpError,
scope: bhttpErrors
}); });
ofTypes = function(obj, types) { ofTypes = function(obj, types) {
@ -80,7 +109,7 @@ isStream = function(obj) {
}; };
prepareSession = function(request, response, requestState) { prepareSession = function(request, response, requestState) {
debug("preparing session"); debugRequest("preparing session");
return Promise["try"](function() { return Promise["try"](function() {
if (requestState.sessionOptions != null) { if (requestState.sessionOptions != null) {
request.options = _.merge(_.clone(requestState.sessionOptions), request.options); request.options = _.merge(_.clone(requestState.sessionOptions), request.options);
@ -96,7 +125,7 @@ prepareSession = function(request, response, requestState) {
delete request.options.cookieJar; delete request.options.cookieJar;
return request.cookieJar.get(request.url); return request.cookieJar.get(request.url);
}).then(function(cookieString) { }).then(function(cookieString) {
debug("sending cookie string: %s", cookieString); debugRequest("sending cookie string: %s", cookieString);
request.options.headers["cookie"] = cookieString; request.options.headers["cookie"] = cookieString;
return Promise.resolve([request, response, requestState]); return Promise.resolve([request, response, requestState]);
}); });
@ -107,7 +136,7 @@ prepareSession = function(request, response, requestState) {
}; };
prepareDefaults = function(request, response, requestState) { prepareDefaults = function(request, response, requestState) {
debug("preparing defaults"); debugRequest("preparing defaults");
return Promise["try"](function() { return Promise["try"](function() {
var _base, _base1, _base2, _ref, _ref1, _ref2, _ref3, _ref4, _ref5, _ref6, _ref7; var _base, _base1, _base2, _ref, _ref1, _ref2, _ref3, _ref4, _ref5, _ref6, _ref7;
request.responseOptions = { request.responseOptions = {
@ -118,7 +147,9 @@ prepareDefaults = function(request, response, requestState) {
decodeJSON: (_ref4 = request.options.decodeJSON) != null ? _ref4 : false, decodeJSON: (_ref4 = request.options.decodeJSON) != null ? _ref4 : false,
stream: (_ref5 = request.options.stream) != null ? _ref5 : false, stream: (_ref5 = request.options.stream) != null ? _ref5 : false,
justPrepare: (_ref6 = request.options.justPrepare) != null ? _ref6 : false, justPrepare: (_ref6 = request.options.justPrepare) != null ? _ref6 : false,
redirectLimit: (_ref7 = request.options.redirectLimit) != null ? _ref7 : 10 redirectLimit: (_ref7 = request.options.redirectLimit) != null ? _ref7 : 10,
onDownloadProgress: request.options.onDownloadProgress,
responseTimeout: request.options.responseTimeout
}; };
if ((_base = request.options).allowChunkedMultipart == null) { if ((_base = request.options).allowChunkedMultipart == null) {
_base.allowChunkedMultipart = false; _base.allowChunkedMultipart = false;
@ -135,7 +166,7 @@ prepareDefaults = function(request, response, requestState) {
}; };
prepareUrl = function(request, response, requestState) { prepareUrl = function(request, response, requestState) {
debug("preparing URL"); debugRequest("preparing URL");
return Promise["try"](function() { return Promise["try"](function() {
var urlOptions, _ref; var urlOptions, _ref;
urlOptions = urlUtil.parse(request.url, true); urlOptions = urlUtil.parse(request.url, true);
@ -153,7 +184,7 @@ prepareUrl = function(request, response, requestState) {
}; };
prepareProtocol = function(request, response, requestState) { prepareProtocol = function(request, response, requestState) {
debug("preparing protocol"); debugRequest("preparing protocol");
return Promise["try"](function() { return Promise["try"](function() {
var _base; var _base;
request.protocolModule = (function() { request.protocolModule = (function() {
@ -167,7 +198,7 @@ prepareProtocol = function(request, response, requestState) {
} }
})(); })();
if (request.protocolModule == null) { if (request.protocolModule == null) {
return Promise.reject()(new errors.UnsupportedProtocolError("The protocol specified (" + protocol + ") is not currently supported by this module.")); return Promise.reject()(new bhttpErrors.UnsupportedProtocolError("The protocol specified (" + protocol + ") is not currently supported by this module."));
} }
if ((_base = request.options).port == null) { if ((_base = request.options).port == null) {
_base.port = (function() { _base.port = (function() {
@ -184,14 +215,14 @@ prepareProtocol = function(request, response, requestState) {
}; };
prepareOptions = function(request, response, requestState) { prepareOptions = function(request, response, requestState) {
debug("preparing options"); debugRequest("preparing options");
return Promise["try"](function() { return Promise["try"](function() {
var _base; var _base;
if (((request.options.formFields != null) || (request.options.files != null)) && ((request.options.inputStream != null) || (request.options.inputBuffer != null))) { if (((request.options.formFields != null) || (request.options.files != null)) && ((request.options.inputStream != null) || (request.options.inputBuffer != null))) {
return Promise.reject(addErrorData(new errors.ConflictingOptionsError("You cannot define both formFields/files and a raw inputStream or inputBuffer."), request, response, requestState)); return Promise.reject(addErrorData(new bhttpErrors.ConflictingOptionsError("You cannot define both formFields/files and a raw inputStream or inputBuffer."), request, response, requestState));
} }
if (request.options.encodeJSON && ((request.options.inputStream != null) || (request.options.inputBuffer != null))) { if (request.options.encodeJSON && ((request.options.inputStream != null) || (request.options.inputBuffer != null))) {
return Promise.reject(addErrorData(new errors.ConflictingOptionsError("You cannot use both encodeJSON and a raw inputStream or inputBuffer.", void 0, "If you meant to JSON-encode the stream, you will currently have to do so manually."), request, response, requestState)); return Promise.reject(addErrorData(new bhttpErrors.ConflictingOptionsError("You cannot use both encodeJSON and a raw inputStream or inputBuffer.", void 0, "If you meant to JSON-encode the stream, you will currently have to do so manually."), request, response, requestState));
} }
if (request.responseOptions.stream) { if (request.responseOptions.stream) {
if ((_base = request.options).agent == null) { if ((_base = request.options).agent == null) {
@ -203,9 +234,10 @@ prepareOptions = function(request, response, requestState) {
}; };
preparePayload = function(request, response, requestState) { preparePayload = function(request, response, requestState) {
debug("preparing payload"); debugRequest("preparing payload");
return Promise["try"](function() { return Promise["try"](function() {
var containsStreams, fieldName, fieldValue, formDataObject, multipart, streamOptions, valueElement, _i, _len, _ref, _ref1, _ref2; var containsStreams, fieldName, fieldValue, formDataObject, multipart, streamOptions, valueElement, _i, _len, _ref, _ref1, _ref2;
request.onUploadProgress = request.options.onUploadProgress;
multipart = request.options.forceMultipart || (request.options.files != null); multipart = request.options.forceMultipart || (request.options.files != null);
multipart = multipart || _.any(request.options.formFields, function(item) { multipart = multipart || _.any(request.options.formFields, function(item) {
return item instanceof Buffer || isStream(item); return item instanceof Buffer || isStream(item);
@ -215,11 +247,11 @@ preparePayload = function(request, response, requestState) {
return isStream(item); return isStream(item);
}); });
if (request.options.encodeJSON && containsStreams) { if (request.options.encodeJSON && containsStreams) {
return Promise.reject()(new errors.ConflictingOptionsError("Sending a JSON-encoded payload containing data from a stream is not currently supported.", void 0, "Either don't use encodeJSON, or read your stream into a string or Buffer.")); return Promise.reject()(new bhttpErrors.ConflictingOptionsError("Sending a JSON-encoded payload containing data from a stream is not currently supported.", void 0, "Either don't use encodeJSON, or read your stream into a string or Buffer."));
} }
if ((_ref = request.options.method) === "post" || _ref === "put" || _ref === "patch") { if ((_ref = request.options.method) === "post" || _ref === "put" || _ref === "patch") {
if ((request.options.encodeJSON || (request.options.formFields != null)) && !multipart) { if ((request.options.encodeJSON || (request.options.formFields != null)) && !multipart) {
debug("got url-encodable form-data"); debugRequest("got url-encodable form-data");
request.options.headers["content-type"] = "application/x-www-form-urlencoded"; request.options.headers["content-type"] = "application/x-www-form-urlencoded";
if (request.options.encodeJSON) { if (request.options.encodeJSON) {
request.payload = JSON.stringify((_ref1 = request.options.formFields) != null ? _ref1 : null); request.payload = JSON.stringify((_ref1 = request.options.formFields) != null ? _ref1 : null);
@ -231,7 +263,7 @@ preparePayload = function(request, response, requestState) {
request.options.headers["content-length"] = request.payload.length; request.options.headers["content-length"] = request.payload.length;
return Promise.resolve(); return Promise.resolve();
} else if ((request.options.formFields != null) && multipart) { } else if ((request.options.formFields != null) && multipart) {
debug("got multipart form-data"); debugRequest("got multipart form-data");
formDataObject = new formData(); formDataObject = new formData();
_ref2 = formFixArray(request.options.formFields); _ref2 = formFixArray(request.options.formFields);
for (fieldName in _ref2) { for (fieldName in _ref2) {
@ -262,7 +294,7 @@ preparePayload = function(request, response, requestState) {
} }
}); });
} else if (request.options.inputStream != null) { } else if (request.options.inputStream != null) {
debug("got inputStream"); debugRequest("got inputStream");
return Promise["try"](function() { return Promise["try"](function() {
var _ref3; var _ref3;
request.payloadStream = request.options.inputStream; request.payloadStream = request.options.inputStream;
@ -272,20 +304,20 @@ preparePayload = function(request, response, requestState) {
return streamLength(request.options.inputStream); return streamLength(request.options.inputStream);
} }
}).then(function(length) { }).then(function(length) {
debug("length for inputStream is %s", length); debugRequest("length for inputStream is %s", length);
return request.options.headers["content-length"] = length; return request.options.headers["content-length"] = length;
})["catch"](function(err) { })["catch"](function(err) {
debug("unable to determine inputStream length, switching to chunked transfer encoding"); debugRequest("unable to determine inputStream length, switching to chunked transfer encoding");
return request.options.headers["content-transfer-encoding"] = "chunked"; return request.options.headers["content-transfer-encoding"] = "chunked";
}); });
} else if (request.options.inputBuffer != null) { } else if (request.options.inputBuffer != null) {
debug("got inputBuffer"); debugRequest("got inputBuffer");
if (typeof request.options.inputBuffer === "string") { if (typeof request.options.inputBuffer === "string") {
request.payload = new Buffer(request.options.inputBuffer); request.payload = new Buffer(request.options.inputBuffer);
} else { } else {
request.payload = request.options.inputBuffer; request.payload = request.options.inputBuffer;
} }
debug("length for inputBuffer is %s", request.payload.length); debugRequest("length for inputBuffer is %s", request.payload.length);
request.options.headers["content-length"] = request.payload.length; request.options.headers["content-length"] = request.payload.length;
return Promise.resolve(); return Promise.resolve();
} }
@ -298,10 +330,10 @@ preparePayload = function(request, response, requestState) {
}; };
prepareCleanup = function(request, response, requestState) { prepareCleanup = function(request, response, requestState) {
debug("preparing cleanup"); debugRequest("preparing cleanup");
return Promise["try"](function() { return Promise["try"](function() {
var fixedHeaders, key, value, _i, _len, _ref, _ref1; var fixedHeaders, key, value, _i, _len, _ref, _ref1;
_ref = ["query", "formFields", "files", "encodeJSON", "inputStream", "inputBuffer", "discardResponse", "keepRedirectResponses", "followRedirects", "noDecode", "decodeJSON", "allowChunkedMultipart", "forceMultipart"]; _ref = ["query", "formFields", "files", "encodeJSON", "inputStream", "inputBuffer", "discardResponse", "keepRedirectResponses", "followRedirects", "noDecode", "decodeJSON", "allowChunkedMultipart", "forceMultipart", "onUploadProgress", "onDownloadProgress"];
for (_i = 0, _len = _ref.length; _i < _len; _i++) { for (_i = 0, _len = _ref.length; _i < _len; _i++) {
key = _ref[_i]; key = _ref[_i];
delete request.options[key]; delete request.options[key];
@ -318,7 +350,7 @@ prepareCleanup = function(request, response, requestState) {
}; };
prepareRequest = function(request, response, requestState) { prepareRequest = function(request, response, requestState) {
debug("preparing request"); debugRequest("preparing request");
return Promise["try"](function() { return Promise["try"](function() {
var middlewareFunctions, promiseChain; var middlewareFunctions, promiseChain;
middlewareFunctions = [prepareSession, prepareDefaults, prepareUrl, prepareProtocol, prepareOptions, preparePayload, prepareCleanup]; middlewareFunctions = [prepareSession, prepareDefaults, prepareUrl, prepareProtocol, prepareOptions, preparePayload, prepareCleanup];
@ -333,30 +365,65 @@ prepareRequest = function(request, response, requestState) {
}; };
makeRequest = function(request, response, requestState) { makeRequest = function(request, response, requestState) {
debug("making %s request to %s", request.options.method.toUpperCase(), request.url); debugRequest("making %s request to %s", request.options.method.toUpperCase(), request.url);
return Promise["try"](function() { return Promise["try"](function() {
var req; var req, timeoutTimer;
req = request.protocolModule.request(request.options); req = request.protocolModule.request(request.options);
if (request.payload != null) { timeoutTimer = null;
debug("sending payload"); return new Promise(function(resolve, reject) {
req.write(request.payload); var completedBytes, progressStream, totalBytes;
req.end(); if (request.responseOptions.responseTimeout != null) {
} else if (request.payloadStream != null) { debugRequest("setting response timeout timer to " + request.responseOptions.responseTimeout + "ms...");
debug("piping payloadStream"); req.on("socket", function(socket) {
if (request.payloadStream._bhttpStreamWrapper != null) { var timeoutHandler;
request.payloadStream.stream.pipe(req); timeoutHandler = function() {
debugRequest("a response timeout occurred!");
req.abort();
return reject(addErrorData(new bhttpErrors.ResponseTimeoutError("The response timed out.")));
};
return timeoutTimer = setTimeout(timeoutHandler, request.responseOptions.responseTimeout);
});
}
totalBytes = request.options.headers["content-length"];
completedBytes = 0;
progressStream = spy(function(chunk) {
completedBytes += chunk.length;
return req.emit("progress", completedBytes, totalBytes);
});
if (request.onUploadProgress != null) {
req.on("progress", function(completedBytes, totalBytes) {
return request.onUploadProgress(completedBytes, totalBytes, req);
});
}
if (request.payload != null) {
debugRequest("sending payload");
req.emit("progress", request.payload.length, request.payload.length);
req.write(request.payload);
req.end();
} else if (request.payloadStream != null) {
debugRequest("piping payloadStream");
if (request.payloadStream._bhttpStreamWrapper != null) {
request.payloadStream.stream.pipe(progressStream).pipe(req);
} else {
request.payloadStream.pipe(progressStream).pipe(req);
}
} else { } else {
request.payloadStream.pipe(req); debugRequest("closing request without payload");
req.end();
} }
} else {
debug("closing request without payload");
req.end();
}
return new Promise(function(resolve, reject) {
req.on("error", function(err) { req.on("error", function(err) {
return reject(err); if (err.code === "ETIMEDOUT") {
debugRequest("a connection timeout occurred!");
return reject(addErrorData(new bhttpErrors.ConnectionTimeoutError("The connection timed out.")));
} else {
return reject(err);
}
}); });
return req.on("response", function(res) { return req.on("response", function(res) {
if (timeoutTimer != null) {
debugResponse("got response in time, clearing response timeout timer");
clearTimeout(timeoutTimer);
}
return resolve(res); return resolve(res);
}); });
}); });
@ -366,7 +433,7 @@ makeRequest = function(request, response, requestState) {
}; };
processResponse = function(request, response, requestState) { processResponse = function(request, response, requestState) {
debug("processing response, got status code %s", response.statusCode); debugResponse("processing response, got status code %s", response.statusCode);
return Promise["try"](function() { return Promise["try"](function() {
var cookieHeader, promises; var cookieHeader, promises;
if ((request.cookieJar != null) && (response.headers["set-cookie"] != null)) { if ((request.cookieJar != null) && (response.headers["set-cookie"] != null)) {
@ -376,7 +443,7 @@ processResponse = function(request, response, requestState) {
_results = []; _results = [];
for (_i = 0, _len = _ref.length; _i < _len; _i++) { for (_i = 0, _len = _ref.length; _i < _len; _i++) {
cookieHeader = _ref[_i]; cookieHeader = _ref[_i];
debug("storing cookie: %s", cookieHeader); debugResponse("storing cookie: %s", cookieHeader);
_results.push(request.cookieJar.set(cookieHeader, request.url)); _results.push(request.cookieJar.set(cookieHeader, request.url));
} }
return _results; return _results;
@ -386,13 +453,13 @@ processResponse = function(request, response, requestState) {
return Promise.resolve(); return Promise.resolve();
} }
}).then(function() { }).then(function() {
var _ref, _ref1; var completedBytes, progressStream, totalBytes, _ref, _ref1;
response.request = request; response.request = request;
response.requestState = requestState; response.requestState = requestState;
response.redirectHistory = requestState.redirectHistory; response.redirectHistory = requestState.redirectHistory;
if (((_ref = response.statusCode) === 301 || _ref === 302 || _ref === 303 || _ref === 307) && request.responseOptions.followRedirects) { if (((_ref = response.statusCode) === 301 || _ref === 302 || _ref === 303 || _ref === 307) && request.responseOptions.followRedirects) {
if (requestState.redirectHistory.length >= (request.responseOptions.redirectLimit - 1)) { if (requestState.redirectHistory.length >= (request.responseOptions.redirectLimit - 1)) {
return Promise.reject(addErrorData(new errors.RedirectError("The maximum amount of redirects ({request.responseOptions.redirectLimit}) was reached."))); return Promise.reject(addErrorData(new bhttpErrors.RedirectError("The maximum amount of redirects ({request.responseOptions.redirectLimit}) was reached.")));
} }
switch (response.statusCode) { switch (response.statusCode) {
case 301: case 301:
@ -404,9 +471,9 @@ processResponse = function(request, response, requestState) {
case "put": case "put":
case "patch": case "patch":
case "delete": case "delete":
return Promise.reject(addErrorData(new errors.RedirectError("Encountered a 301 redirect for POST, PUT, PATCH or DELETE. RFC says we can't automatically continue."), request, response, requestState)); return Promise.reject(addErrorData(new bhttpErrors.RedirectError("Encountered a 301 redirect for POST, PUT, PATCH or DELETE. RFC says we can't automatically continue."), request, response, requestState));
default: default:
return Promise.reject(addErrorData(new errors.RedirectError("Encountered a 301 redirect, but not sure how to proceed for the " + (request.options.method.toUpperCase()) + " method."))); return Promise.reject(addErrorData(new bhttpErrors.RedirectError("Encountered a 301 redirect, but not sure how to proceed for the " + (request.options.method.toUpperCase()) + " method.")));
} }
break; break;
case 302: case 302:
@ -414,7 +481,7 @@ processResponse = function(request, response, requestState) {
return redirectGet(request, response, requestState); return redirectGet(request, response, requestState);
case 307: case 307:
if (request.containsStreams && ((_ref1 = request.options.method) !== "get" && _ref1 !== "head")) { if (request.containsStreams && ((_ref1 = request.options.method) !== "get" && _ref1 !== "head")) {
return Promise.reject(addErrorData(new errors.RedirectError("Encountered a 307 redirect for POST, PUT or DELETE, but your payload contained (single-use) streams. We therefore can't automatically follow the redirect."), request, response, requestState)); return Promise.reject(addErrorData(new bhttpErrors.RedirectError("Encountered a 307 redirect for POST, PUT or DELETE, but your payload contained (single-use) streams. We therefore can't automatically follow the redirect."), request, response, requestState));
} else { } else {
return redirectUnchanged(request, response, requestState); return redirectUnchanged(request, response, requestState);
} }
@ -423,7 +490,23 @@ processResponse = function(request, response, requestState) {
response.resume(); response.resume();
return Promise.resolve(response); return Promise.resolve(response);
} else { } else {
totalBytes = response.headers["content-length"];
if (totalBytes != null) {
totalBytes = parseInt(totalBytes);
}
completedBytes = 0;
progressStream = sink(function(chunk) {
completedBytes += chunk.length;
return response.emit("progress", completedBytes, totalBytes);
});
if (request.responseOptions.onDownloadProgress != null) {
response.on("progress", function(completedBytes, totalBytes) {
return request.responseOptions.onDownloadProgress(completedBytes, totalBytes, response);
});
}
return new Promise(function(resolve, reject) { return new Promise(function(resolve, reject) {
response.pipe(progressStream);
response.pause();
if (request.responseOptions.stream) { if (request.responseOptions.stream) {
return resolve(response); return resolve(response);
} else { } else {
@ -464,7 +547,7 @@ doPayloadRequest = function(url, data, options, callback) {
}; };
redirectGet = function(request, response, requestState) { redirectGet = function(request, response, requestState) {
debug("following forced-GET redirect to %s", response.headers["location"]); debugResponse("following forced-GET redirect to %s", response.headers["location"]);
return Promise["try"](function() { return Promise["try"](function() {
var key, options, _i, _len, _ref; var key, options, _i, _len, _ref;
options = _.clone(requestState.originalOptions); options = _.clone(requestState.originalOptions);
@ -479,7 +562,7 @@ redirectGet = function(request, response, requestState) {
}; };
redirectUnchanged = function(request, response, requestState) { redirectUnchanged = function(request, response, requestState) {
debug("following same-method redirect to %s", response.headers["location"]); debugResponse("following same-method redirect to %s", response.headers["location"]);
return Promise["try"](function() { return Promise["try"](function() {
var options; var options;
options = _.clone(requestState.originalOptions); options = _.clone(requestState.originalOptions);
@ -652,4 +735,6 @@ bhttpAPI = {
} }
}; };
extend(bhttpAPI, bhttpErrors);
module.exports = bhttpAPI; module.exports = bhttpAPI;

@ -1,6 +1,6 @@
{ {
"name": "bhttp", "name": "bhttp",
"version": "1.0.4", "version": "1.1.0",
"description": "A sane HTTP client library for Node.js with Streams2 support.", "description": "A sane HTTP client library for Node.js with Streams2 support.",
"main": "index.js", "main": "index.js",
"scripts": { "scripts": {
@ -26,11 +26,14 @@
"concat-stream": "^1.4.7", "concat-stream": "^1.4.7",
"debug": "^2.1.1", "debug": "^2.1.1",
"errors": "^0.2.0", "errors": "^0.2.0",
"extend": "^2.0.0",
"form-data2": "^1.0.0", "form-data2": "^1.0.0",
"form-fix-array": "^1.0.0", "form-fix-array": "^1.0.0",
"lodash": "^2.4.1", "lodash": "^2.4.1",
"stream-length": "^1.0.2", "stream-length": "^1.0.2",
"string": "^3.0.0", "string": "^3.0.0",
"through2-sink": "^1.0.0",
"through2-spy": "^1.2.0",
"tough-cookie": "^0.12.1" "tough-cookie": "^0.12.1"
}, },
"devDependencies": { "devDependencies": {

@ -0,0 +1,5 @@
firstCode = global.code
require "./"
secondCode = global.code
console.log global.name
console.log "Leaking global.code:", (firstCode != secondCode)

@ -0,0 +1,37 @@
Promise = require "bluebird"
bhttp = require "./"
fs = require "fs"
formatLine = (line) -> line.toString().replace(/\n/g, "\\n").replace(/\r/g, "\\r")
Promise.try ->
# multipart POST upload
console.log "Starting upload..."
bhttp.post "http://posttestserver.com/post.php",
fieldOne: "value 1"
fieldTwo: "value 2"
fieldThree: ["value 3a", "value 3b"]
fieldFour: new Buffer "value 4"
testFile: fs.createReadStream("./lower.txt")
,
headers: {"user-agent": "bhttp/test POST multipart"}
onUploadProgress: (completedBytes, totalBytes, request) ->
console.log "#{completedBytes / totalBytes * 100}%", completedBytes, totalBytes
.then (response) ->
console.log "POST multipart", formatLine(response.body)
.then ->
# GET large file
console.log "Starting download..."
bhttp.get "http://posttestserver.com/files/2015/04/06/f_00.16.102133822615",
headers: {"user-agent": "bhttp/test GET large file"}
stream: true
.then (response) ->
#setTimeout (->), 10000
console.log "Got response"
response.on "progress", (completedBytes, totalBytes, request) ->
console.log "#{completedBytes / totalBytes * 100}%", completedBytes, totalBytes
response.resume()
response.on "end", ->
console.log "Completed response download"

@ -0,0 +1,6 @@
bhttp = require "./"
console.log "make request"
bhttp.get("http://dead.dns.entry/asdfasdgasdg", responseTimeout: 4000)
bhttp.get("http://cryto.net", responseTimeout: 4000)
bhttp.get("http://dead.dns.entry/asdfasdgasdg")
Loading…
Cancel
Save