1
0
Fork 0

v1.1.0: Progress events, response timeouts, actually usable errors, namespaced debug information, documentation fixes

master
Sven Slootweg 9 years ago
parent 1fdaac9099
commit d1d5e4b353

2
.gitignore vendored

@ -2,3 +2,5 @@
# https://help.github.com/articles/ignoring-files
# Example .gitignore files: https://github.com/github/gitignore
/node_modules/
/test-exit2.js
/test-exit.coffee

@ -2,11 +2,13 @@
A sane HTTP client library for Node.js with Streams2 support.
![](//img.shields.io/gratipay/joepie91.svg)
## Why bhttp?
There are already a few commonly used HTTP client libraries for Node.js, but all of them have issues:
* The core `http` module is rather low-level, and even relatively simple requests take a lot of work to make correctly. It also automatically uses an agent for HTTP requests, which slows down concurrent HTTP requests when you're streaming the responses somewhere.
* The core `http` module is rather low-level, and even relatively simple requests take a lot of work to make correctly. It also automatically uses a limited amount of agents for HTTP requests (in Node.js 0.10), which slows down concurrent HTTP requests when you're streaming the responses somewhere.
* `request` is buggy, only supports old-style streams, has the same 'agent' problem as `http`, the documentation is poor, and the API is not very intuitive.
* `needle` is a lot simpler, but suffers from the same 'agent' problem, and the API can be a bit annoying in some ways. It also doesn't have a proper session API.
* `hyperquest` (mostly) solves the 'agent' problem correctly, but has a very spartan API. Making non-GET requests is more complex than it should be.
@ -21,10 +23,13 @@ All these issues (and more) are solved in `bhttp`. It offers the following:
* Easy-to-use session mechanics - a new session will automatically give you a new cookie jar, cookies are kept track of automatically, and 'default options' are deep-merged.
* Streaming requests are kept out of the agent pool - ie. no blocking of other requests.
* Optionally, a Promises API (you can also use nodebacks).
* Progress events! For both uploading and downloading.
## Caveats
* `bhttp` does not yet use a HTTPS-capable agent. This means that all SSL-related options are currently ignored (per Node.js `http` documentation). If you need secure HTTPS requests, make sure to specify a custom agent!
`bhttp` does not yet use a HTTPS-capable agent. This means that all SSL-related options are currently ignored by default (per Node.js `http` documentation).
__This does *not* mean that you cannot use `bhttp` for HTTPS requests!__ If you need secure HTTPS requests, just make sure to specify a [custom `https` agent](https://nodejs.org/api/https.html#https_class_https_agent).
## License
@ -106,6 +111,44 @@ bhttp.get("http://somesite.com/bigfile.mp4", {stream: true}, function(err, respo
})
```
### Progress events
Upload progress events:
```javascript
var Promise = require("bluebird");
var bhttp = require("bhttp");
Promise.try(function() {
return bhttp.post("http://somehostingservice.com/upload", {
file: fs.createReadStream("./bigfile.mkv")
}, {
onUploadProgress: function(completedBytes, totalBytes) {
console.log("Upload progress:", (completedBytes / totalBytes * 100), "%");
}
});
}).then(function(response) {
console.log("Response from hosting service:", response.body.toString());
});
```
Download progress events:
```javascript
var Promise = require("bluebird");
var bhttp = require("bhttp");
Promise.try(function() {
return bhttp.get("http://somehostingservice.com/bigfile.mkv", {stream: true});
}).then(function(response) {
response.on("progress", function(completedBytes, totalBytes) {
console.log("Download progress:", (completedBytes / totalBytes * 100), "%");
});
response.pipe(fs.createWriteStream("./bigfile.mkv"));
});
```
### Sessions
```javascript
@ -125,6 +168,8 @@ Promise.try(function(){
## API
The various error types are documented at the bottom of this README.
### bhttp.head(url, [options, [callback]])
### bhttp.get(url, [options, [callback]])
### bhttp.delete(url, [options, [callback]])
@ -149,6 +194,8 @@ The `data` payload can be one of the following things:
Makes a request, and returns the response object asynchronously. The response object is a standard `http.IncomingMessages` with a few additional properties (documented below the argument list).
Note that (progress) event handlers must be specified in the `options` or (in the case of download progress events) as an event listener on the response object - as `bhttp` uses Promises, it is not technically possible to return an EventEmitter.
* __url__: The URL to request, with protocol. When using HTTPS, please be sure to read the 'Caveats' section.
* __options__: *Optional.* Extra options for the request. Any other options not listed here will be passed on directly to the `http` or `https` module.
* __Basic options__
@ -169,10 +216,14 @@ Makes a request, and returns the response object asynchronously. The response ob
* __Advanced options__
* __method__: The request method to use. You don't need this when using the shorthand methods.
* __cookieJar__: A custom cookie jar to use. You'll probably want to use `bhttp.session()` instead.
* __responseTimeout__: The timeout, in milliseconds, after which the request should be considered to have failed if no response is received yet. Note that this measures from the start of the request to the start of the response, and is *not* a connection timeout. If a timeout occurs, a ResponseTimeoutError will be thrown asynchronously (see error documentation below).
* __allowChunkedMultipart__: *Defaults to `false`.* Many servers don't support `multipart/form-data` when it is transmitted with chunked transfer encoding (eg. when the stream length is unknown), and silently fail with an empty request payload - this is why `bhttp` disallows it by default. If you are *absolutely certain* that the endpoint supports this functionality, you can override the behaviour by setting this to `true`.
* __discardResponse__: *Defaults to `false`.* Whether to throw away the response without reading it. Only really useful for fire-and-forget calls. This is almost never what you want.
* __keepRedirectResponses__: *Defaults to `false`.* Whether to keep the response streams of redirects. You probably don't need this. __When enabling this, you must *explicitly* read out every single redirect response, or you will experience memory leaks!__
* __justPrepare__: *Defaults to `false`.* When set to `true`, bhttp just prepares the request, and doesn't actually carry it out; useful if you want to make some manual modifications. Instead of a response, the method will asynchronously return an array with the signature `[request, response, requestState]` that you will need to pass into the `bhttp.makeRequest()` method.
* __Event handlers__
* __onUploadProgress__: A callback to call for upload progress events (this covers both input streams and form data). The callback signature is `(completedBytes, totalBytes, request)`. If the total size is not known, `totalBytes` will be `undefined`. The `request` variable will hold the request object that the progress event applies to - this is relevant when dealing with automatic redirect following, where multiple requests may occur.
* __onDownloadProgress__: A callback to call for download progress events. The callback signature is `(completedBytes, totalBytes, response)`. If the total size is not known, `totalBytes` will be `undefined`. The `response` variable will hold the response object that the progress event applies to - this is relevant when dealing with automatic redirect following, where multiple responses may occur. *Note that using the `progress` event on a response object is usually a more practical option!*
* __callback__: *Optional.* When using the nodeback API, the callback to use. If not specified, a Promise will be returned.
@ -183,6 +234,10 @@ A few extra properties are set on the response object (which is a `http.Incoming
* __request__: The request configuration that was generated by `bhttp`. You probably don't need this.
* __requestState__: The request state that was accumulated by `bhttp`. You probably don't need this.
Additionally, there's an extra event on the `response` object:
* __'progress' (completedBytes, totalBytes)__: The 'download progress' for the response body. This works the same as the `onDownloadProgress` option, except the event will be specific to this response, and it allows for somewhat nicer syntax. Make sure to attach this handler *before* you start reading the response stream!
`bhttp` can automatically parse the metadata for the following types of streams:
* `fs` streams
@ -190,7 +245,7 @@ A few extra properties are set on the response object (which is a `http.Incoming
* `request` requests
* `combined-stream` streams (assuming all the underlying streams are of one of the types listed here)
If you are using a different type of streams, you can wrap the stream using `bhttp.wrapStream` to manually specify the needed metadata.
If you are using a different type of stream, you can wrap the stream using `bhttp.wrapStream` to manually specify the needed metadata.
### bhttp.session([defaultOptions])
@ -213,3 +268,51 @@ The resulting wrapper can be passed on to the `bhttp` methods as if it were a re
### bhttp.makeRequest(request, response, requestState)
When using the `justPrepare` option, you can use this method to proceed with the request after manual modifications. The function signature is identical to the signature of the array returned when using `justPrepare`. `response` will usually be `null`, but must be passed on as is, to account for future API changes.
## Error types
All these correctly extend the `Error` class - this means that you can use them as a `.catch` predicate when using Promises, and that you can use `instanceof` on them when using the nodeback API.
### bhttp.bhttpError
The base class for all errors generated by `bhttp`. You usually don't need this.
### bhttp.ConflictingOptionsError
You have specified two or more request options that cannot be used together.
The error message will contain more details.
### bhttp.UnsupportedProtocolError
You tried to load a URL that isn't using either the HTTP or HTTPS protocol. Only HTTP and HTTPS are currently supported.
### bhttp.RedirectError
A redirect was encountered that could not be followed.
This could be because the redirect limit was reached, or because the HTTP specification doesn't allow automatic following of the redirect that was encountered.
The error message will contain more details.
### bhttp.MultipartError
Something went wrong while generating the multipart/form-data stream.
Currently, this will only be thrown if you try to use chunked transfer encoding for a multipart stream - a common situation where this can occur, is when you pass in streams with an unknown length.
To resolve this error, you must either explicitly specify the length of the streams using `bhttp.wrapStream` or, if the target server supports it, enable the `allowChunkedMultipart` option.
### bhttp.ConnectionTimeoutError
The connection timed out.
The connection timeout is defined by the operating system, and cannot currently be overridden.
### bhttp.ResponseTimeoutError
The response timed out.
The response timeout can be specified using the `responseTimeout` option, and it is measured from the start of the request to the start of the response. If no response is received within the `responseTimeout`, a `ResponseTimeoutError` will be thrown asynchronously, and the request will be aborted.
__You should not set a `responseTimeout` for requests that involve large file uploads!__ Because a response can only be received *after* the request has completed, any file/stream upload that takes longer than the `responseTimeout`, will result in a `ResponseTimeoutError`.

@ -17,37 +17,59 @@ _ = require "lodash"
S = require "string"
formFixArray = require "form-fix-array"
errors = require "errors"
debug = require("debug")("bhttp")
debug = require("debug")
debugRequest = debug("bhttp:request")
debugResponse = debug("bhttp:response")
extend = require "extend"
# Other third-party modules
formData = require "form-data2"
concatStream = require "concat-stream"
toughCookie = require "tough-cookie"
streamLength = require "stream-length"
sink = require "through2-sink"
spy = require "through2-spy"
# For the version in the user agent, etc.
packageConfig = require "../package.json"
bhttpErrors = {}
# Error types
errors.create
name: "bhttpError"
scope: bhttpErrors
errors.create
name: "ConflictingOptionsError"
parents: errors.bhttpError
parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create
name: "UnsupportedProtocolError"
parents: errors.bhttpError
parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create
name: "RedirectError"
parents: errors.bhttpError
parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create
name: "MultipartError"
parents: errors.bhttpError
parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create
name: "ConnectionTimeoutError"
parents: bhttpErrors.bhttpError
scope: bhttpErrors
errors.create
name: "ResponseTimeoutError"
parents: bhttpErrors.bhttpError
scope: bhttpErrors
# Utility functions
@ -69,7 +91,7 @@ isStream = (obj) ->
# Middleware
# NOTE: requestState is an object that signifies the current state of the overall request; eg. for a response involving one or more redirects, it will hold a 'redirect history'.
prepareSession = (request, response, requestState) ->
debug "preparing session"
debugRequest "preparing session"
Promise.try ->
if requestState.sessionOptions?
# Request options take priority over session options
@ -92,14 +114,14 @@ prepareSession = (request, response, requestState) ->
# Get the current cookie string for the URL
request.cookieJar.get request.url
.then (cookieString) ->
debug "sending cookie string: %s", cookieString
debugRequest "sending cookie string: %s", cookieString
request.options.headers["cookie"] = cookieString
Promise.resolve [request, response, requestState]
else
Promise.resolve [request, response, requestState]
prepareDefaults = (request, response, requestState) ->
debug "preparing defaults"
debugRequest "preparing defaults"
Promise.try ->
# These are the options that we need for response processing, but don't need to be passed on to the http/https module.
request.responseOptions =
@ -111,6 +133,8 @@ prepareDefaults = (request, response, requestState) ->
stream: request.options.stream ? false
justPrepare: request.options.justPrepare ? false
redirectLimit: request.options.redirectLimit ? 10
onDownloadProgress: request.options.onDownloadProgress
responseTimeout: request.options.responseTimeout
# Whether chunked transfer encoding for multipart/form-data payloads is acceptable. This is likely to break quietly on a lot of servers.
request.options.allowChunkedMultipart ?= false
@ -127,7 +151,7 @@ prepareDefaults = (request, response, requestState) ->
Promise.resolve [request, response, requestState]
prepareUrl = (request, response, requestState) ->
debug "preparing URL"
debugRequest "preparing URL"
Promise.try ->
# Parse the specified URL, and use the resulting information to build a complete `options` object
urlOptions = urlUtil.parse request.url, true
@ -139,7 +163,7 @@ prepareUrl = (request, response, requestState) ->
Promise.resolve [request, response, requestState]
prepareProtocol = (request, response, requestState) ->
debug "preparing protocol"
debugRequest "preparing protocol"
Promise.try ->
request.protocolModule = switch request.protocol
when "http" then http
@ -147,7 +171,7 @@ prepareProtocol = (request, response, requestState) ->
else null
if not request.protocolModule?
return Promise.reject() new errors.UnsupportedProtocolError "The protocol specified (#{protocol}) is not currently supported by this module."
return Promise.reject() new bhttpErrors.UnsupportedProtocolError "The protocol specified (#{protocol}) is not currently supported by this module."
request.options.port ?= switch request.protocol
when "http" then 80
@ -156,14 +180,14 @@ prepareProtocol = (request, response, requestState) ->
Promise.resolve [request, response, requestState]
prepareOptions = (request, response, requestState) ->
debug "preparing options"
debugRequest "preparing options"
Promise.try ->
# Do some sanity checks - there are a number of options that cannot be used together
if (request.options.formFields? or request.options.files?) and (request.options.inputStream? or request.options.inputBuffer?)
return Promise.reject addErrorData(new errors.ConflictingOptionsError("You cannot define both formFields/files and a raw inputStream or inputBuffer."), request, response, requestState)
return Promise.reject addErrorData(new bhttpErrors.ConflictingOptionsError("You cannot define both formFields/files and a raw inputStream or inputBuffer."), request, response, requestState)
if request.options.encodeJSON and (request.options.inputStream? or request.options.inputBuffer?)
return Promise.reject addErrorData(new errors.ConflictingOptionsError("You cannot use both encodeJSON and a raw inputStream or inputBuffer.", undefined, "If you meant to JSON-encode the stream, you will currently have to do so manually."), request, response, requestState)
return Promise.reject addErrorData(new bhttpErrors.ConflictingOptionsError("You cannot use both encodeJSON and a raw inputStream or inputBuffer.", undefined, "If you meant to JSON-encode the stream, you will currently have to do so manually."), request, response, requestState)
# If the user plans on streaming the response, we need to disable the agent entirely - otherwise the streams will block the pool.
if request.responseOptions.stream
@ -172,8 +196,11 @@ prepareOptions = (request, response, requestState) ->
Promise.resolve [request, response, requestState]
preparePayload = (request, response, requestState) ->
debug "preparing payload"
debugRequest "preparing payload"
Promise.try ->
# Persist the download progress event handler on the request object, if there is one.
request.onUploadProgress = request.options.onUploadProgress
# If a 'files' parameter is present, then we will send the form data as multipart data - it's most likely binary data.
multipart = request.options.forceMultipart or request.options.files?
@ -188,13 +215,13 @@ preparePayload = (request, response, requestState) ->
containsStreams = _.any request.options.formFields, (item) -> isStream(item)
if request.options.encodeJSON and containsStreams
return Promise.reject() new errors.ConflictingOptionsError "Sending a JSON-encoded payload containing data from a stream is not currently supported.", undefined, "Either don't use encodeJSON, or read your stream into a string or Buffer."
return Promise.reject() new bhttpErrors.ConflictingOptionsError "Sending a JSON-encoded payload containing data from a stream is not currently supported.", undefined, "Either don't use encodeJSON, or read your stream into a string or Buffer."
if request.options.method in ["post", "put", "patch"]
# Prepare the payload, and set the appropriate headers.
if (request.options.encodeJSON or request.options.formFields?) and not multipart
# We know the payload and its size in advance.
debug "got url-encodable form-data"
debugRequest "got url-encodable form-data"
request.options.headers["content-type"] = "application/x-www-form-urlencoded"
if request.options.encodeJSON
@ -210,7 +237,7 @@ preparePayload = (request, response, requestState) ->
return Promise.resolve()
else if request.options.formFields? and multipart
# This is going to be multipart data, and we'll let `form-data` set the headers for us.
debug "got multipart form-data"
debugRequest "got multipart form-data"
formDataObject = new formData()
for fieldName, fieldValue of formFixArray(request.options.formFields)
@ -238,7 +265,7 @@ preparePayload = (request, response, requestState) ->
Promise.resolve()
else if request.options.inputStream?
# A raw inputStream was provided, just leave it be.
debug "got inputStream"
debugRequest "got inputStream"
Promise.try ->
request.payloadStream = request.options.inputStream
@ -247,20 +274,20 @@ preparePayload = (request, response, requestState) ->
else
streamLength request.options.inputStream
.then (length) ->
debug "length for inputStream is %s", length
debugRequest "length for inputStream is %s", length
request.options.headers["content-length"] = length
.catch (err) ->
debug "unable to determine inputStream length, switching to chunked transfer encoding"
debugRequest "unable to determine inputStream length, switching to chunked transfer encoding"
request.options.headers["content-transfer-encoding"] = "chunked"
else if request.options.inputBuffer?
# A raw inputBuffer was provided, just leave it be (but make sure it's an actual Buffer).
debug "got inputBuffer"
debugRequest "got inputBuffer"
if typeof request.options.inputBuffer == "string"
request.payload = new Buffer(request.options.inputBuffer) # Input string should be utf-8!
else
request.payload = request.options.inputBuffer
debug "length for inputBuffer is %s", request.payload.length
debugRequest "length for inputBuffer is %s", request.payload.length
request.options.headers["content-length"] = request.payload.length
return Promise.resolve()
@ -272,10 +299,10 @@ preparePayload = (request, response, requestState) ->
Promise.resolve [request, response, requestState]
prepareCleanup = (request, response, requestState) ->
debug "preparing cleanup"
debugRequest "preparing cleanup"
Promise.try ->
# Remove the options that we're not going to pass on to the actual http/https library.
delete request.options[key] for key in ["query", "formFields", "files", "encodeJSON", "inputStream", "inputBuffer", "discardResponse", "keepRedirectResponses", "followRedirects", "noDecode", "decodeJSON", "allowChunkedMultipart", "forceMultipart"]
delete request.options[key] for key in ["query", "formFields", "files", "encodeJSON", "inputStream", "inputBuffer", "discardResponse", "keepRedirectResponses", "followRedirects", "noDecode", "decodeJSON", "allowChunkedMultipart", "forceMultipart", "onUploadProgress", "onDownloadProgress"]
# Lo-Dash apparently has no `map` equivalent for object keys...?
fixedHeaders = {}
@ -288,7 +315,7 @@ prepareCleanup = (request, response, requestState) ->
# The guts of the module
prepareRequest = (request, response, requestState) ->
debug "preparing request"
debugRequest "preparing request"
# FIXME: Mock httpd for testing functionality.
Promise.try ->
middlewareFunctions = [
@ -310,41 +337,78 @@ prepareRequest = (request, response, requestState) ->
return promiseChain
makeRequest = (request, response, requestState) ->
debug "making %s request to %s", request.options.method.toUpperCase(), request.url
debugRequest "making %s request to %s", request.options.method.toUpperCase(), request.url
Promise.try ->
# Instantiate a regular HTTP/HTTPS request
req = request.protocolModule.request request.options
# This is where we write our payload or stream to the request, and the actual request is made.
if request.payload?
# The entire payload is a single Buffer.
debug "sending payload"
req.write request.payload
req.end()
else if request.payloadStream?
# The payload is a stream.
debug "piping payloadStream"
if request.payloadStream._bhttpStreamWrapper?
request.payloadStream.stream.pipe req
else
request.payloadStream.pipe req
else
# For GET, HEAD, DELETE, etc. there is no payload, but we still need to call end() to complete the request.
debug "closing request without payload"
req.end()
timeoutTimer = null
new Promise (resolve, reject) ->
# Connection timeout handling, if one is set.
if request.responseOptions.responseTimeout?
debugRequest "setting response timeout timer to #{request.responseOptions.responseTimeout}ms..."
req.on "socket", (socket) ->
timeoutHandler = ->
debugRequest "a response timeout occurred!"
req.abort()
reject addErrorData(new bhttpErrors.ResponseTimeoutError("The response timed out."))
timeoutTimer = setTimeout(timeoutHandler, request.responseOptions.responseTimeout)
# Set up the upload progress monitoring.
totalBytes = request.options.headers["content-length"]
completedBytes = 0
progressStream = spy (chunk) ->
completedBytes += chunk.length
req.emit "progress", completedBytes, totalBytes
if request.onUploadProgress?
req.on "progress", (completedBytes, totalBytes) ->
request.onUploadProgress(completedBytes, totalBytes, req)
# This is where we write our payload or stream to the request, and the actual request is made.
if request.payload?
# The entire payload is a single Buffer. We'll still pretend that it's a stream for our progress events, though, to provide a consistent API.
debugRequest "sending payload"
req.emit "progress", request.payload.length, request.payload.length
req.write request.payload
req.end()
else if request.payloadStream?
# The payload is a stream.
debugRequest "piping payloadStream"
if request.payloadStream._bhttpStreamWrapper?
request.payloadStream.stream
.pipe progressStream
.pipe req
else
request.payloadStream
.pipe progressStream
.pipe req
else
# For GET, HEAD, DELETE, etc. there is no payload, but we still need to call end() to complete the request.
debugRequest "closing request without payload"
req.end()
# In case something goes wrong during this process, somehow...
req.on "error", (err) ->
reject err
if err.code == "ETIMEDOUT"
debugRequest "a connection timeout occurred!"
reject addErrorData(new bhttpErrors.ConnectionTimeoutError("The connection timed out."))
else
reject err
req.on "response", (res) ->
if timeoutTimer?
debugResponse "got response in time, clearing response timeout timer"
clearTimeout(timeoutTimer)
resolve res
.then (response) ->
Promise.resolve [request, response, requestState]
processResponse = (request, response, requestState) ->
debug "processing response, got status code %s", response.statusCode
debugResponse "processing response, got status code %s", response.statusCode
# When we receive the response, we'll buffer it up and/or decode it, depending on what the user specified, and resolve the returned Promise. If the user just wants the raw stream, we resolve immediately after receiving a response.
@ -352,7 +416,7 @@ processResponse = (request, response, requestState) ->
# First, if a cookie jar is set and we received one or more cookies from the server, we should store them in our cookieJar.
if request.cookieJar? and response.headers["set-cookie"]?
promises = for cookieHeader in response.headers["set-cookie"]
debug "storing cookie: %s", cookieHeader
debugResponse "storing cookie: %s", cookieHeader
request.cookieJar.set cookieHeader, request.url
Promise.all promises
else
@ -365,7 +429,7 @@ processResponse = (request, response, requestState) ->
if response.statusCode in [301, 302, 303, 307] and request.responseOptions.followRedirects
if requestState.redirectHistory.length >= (request.responseOptions.redirectLimit - 1)
return Promise.reject addErrorData(new errors.RedirectError("The maximum amount of redirects ({request.responseOptions.redirectLimit}) was reached."))
return Promise.reject addErrorData(new bhttpErrors.RedirectError("The maximum amount of redirects ({request.responseOptions.redirectLimit}) was reached."))
# 301: For GET and HEAD, redirect unchanged. For POST, PUT, PATCH, DELETE, "ask user" (in our case: throw an error.)
# 302: Redirect, change method to GET.
@ -377,21 +441,38 @@ processResponse = (request, response, requestState) ->
when "get", "head"
return redirectUnchanged request, response, requestState
when "post", "put", "patch", "delete"
return Promise.reject addErrorData(new errors.RedirectError("Encountered a 301 redirect for POST, PUT, PATCH or DELETE. RFC says we can't automatically continue."), request, response, requestState)
return Promise.reject addErrorData(new bhttpErrors.RedirectError("Encountered a 301 redirect for POST, PUT, PATCH or DELETE. RFC says we can't automatically continue."), request, response, requestState)
else
return Promise.reject addErrorData(new errors.RedirectError("Encountered a 301 redirect, but not sure how to proceed for the #{request.options.method.toUpperCase()} method."))
return Promise.reject addErrorData(new bhttpErrors.RedirectError("Encountered a 301 redirect, but not sure how to proceed for the #{request.options.method.toUpperCase()} method."))
when 302, 303
return redirectGet request, response, requestState
when 307
if request.containsStreams and request.options.method not in ["get", "head"]
return Promise.reject addErrorData(new errors.RedirectError("Encountered a 307 redirect for POST, PUT or DELETE, but your payload contained (single-use) streams. We therefore can't automatically follow the redirect."), request, response, requestState)
return Promise.reject addErrorData(new bhttpErrors.RedirectError("Encountered a 307 redirect for POST, PUT or DELETE, but your payload contained (single-use) streams. We therefore can't automatically follow the redirect."), request, response, requestState)
else
return redirectUnchanged request, response, requestState
else if request.responseOptions.discardResponse
response.resume() # Drain the response stream
Promise.resolve response
else
totalBytes = response.headers["content-length"]
if totalBytes? # Otherwise `undefined` will turn into `NaN`, and we don't want that.
totalBytes = parseInt(totalBytes)
completedBytes = 0
progressStream = sink (chunk) ->
completedBytes += chunk.length
response.emit "progress", completedBytes, totalBytes
if request.responseOptions.onDownloadProgress?
response.on "progress", (completedBytes, totalBytes) ->
request.responseOptions.onDownloadProgress(completedBytes, totalBytes, response)
new Promise (resolve, reject) ->
# This is slightly hacky, but returning a .pipe'd stream would make all the response object attributes unavailable to the end user. Therefore, 'branching' the stream into our progress monitoring stream is a much better option. We use .pause() to ensure the stream doesn't start flowing until the end user (or our library) has actually piped it into something.
response.pipe(progressStream)
response.pause()
if request.responseOptions.stream
resolve response
else
@ -428,7 +509,7 @@ doPayloadRequest = (url, data, options, callback) ->
@request url, options, callback
redirectGet = (request, response, requestState) ->
debug "following forced-GET redirect to %s", response.headers["location"]
debugResponse "following forced-GET redirect to %s", response.headers["location"]
Promise.try ->
options = _.clone(requestState.originalOptions)
options.method = "get"
@ -438,7 +519,7 @@ redirectGet = (request, response, requestState) ->
doRedirect request, response, requestState, options
redirectUnchanged = (request, response, requestState) ->
debug "following same-method redirect to %s", response.headers["location"]
debugResponse "following same-method redirect to %s", response.headers["location"]
Promise.try ->
options = _.clone(requestState.originalOptions)
doRedirect request, response, requestState, options
@ -541,6 +622,8 @@ bhttpAPI =
options: options
}
extend(bhttpAPI, bhttpErrors)
module.exports = bhttpAPI
# That's all, folks!

@ -1,4 +1,4 @@
var Promise, S, addErrorData, bhttpAPI, concatStream, createCookieJar, debug, doPayloadRequest, doRedirect, errors, formData, formFixArray, http, https, isStream, makeRequest, ofTypes, packageConfig, prepareCleanup, prepareDefaults, prepareOptions, preparePayload, prepareProtocol, prepareRequest, prepareSession, prepareUrl, processResponse, querystring, redirectGet, redirectUnchanged, stream, streamLength, toughCookie, urlUtil, util, _;
var Promise, S, addErrorData, bhttpAPI, bhttpErrors, concatStream, createCookieJar, debug, debugRequest, debugResponse, doPayloadRequest, doRedirect, errors, extend, formData, formFixArray, http, https, isStream, makeRequest, ofTypes, packageConfig, prepareCleanup, prepareDefaults, prepareOptions, preparePayload, prepareProtocol, prepareRequest, prepareSession, prepareUrl, processResponse, querystring, redirectGet, redirectUnchanged, sink, spy, stream, streamLength, toughCookie, urlUtil, util, _;
urlUtil = require("url");
@ -22,7 +22,13 @@ formFixArray = require("form-fix-array");
errors = require("errors");
debug = require("debug")("bhttp");
debug = require("debug");
debugRequest = debug("bhttp:request");
debugResponse = debug("bhttp:response");
extend = require("extend");
formData = require("form-data2");
@ -32,30 +38,53 @@ toughCookie = require("tough-cookie");
streamLength = require("stream-length");
sink = require("through2-sink");
spy = require("through2-spy");
packageConfig = require("../package.json");
bhttpErrors = {};
errors.create({
name: "bhttpError"
name: "bhttpError",
scope: bhttpErrors
});
errors.create({
name: "ConflictingOptionsError",
parents: errors.bhttpError
parents: bhttpErrors.bhttpError,
scope: bhttpErrors
});
errors.create({
name: "UnsupportedProtocolError",
parents: errors.bhttpError
parents: bhttpErrors.bhttpError,
scope: bhttpErrors
});
errors.create({
name: "RedirectError",
parents: errors.bhttpError
parents: bhttpErrors.bhttpError,
scope: bhttpErrors
});
errors.create({
name: "MultipartError",
parents: errors.bhttpError
parents: bhttpErrors.bhttpError,
scope: bhttpErrors
});
errors.create({
name: "ConnectionTimeoutError",
parents: bhttpErrors.bhttpError,
scope: bhttpErrors
});
errors.create({
name: "ResponseTimeoutError",
parents: bhttpErrors.bhttpError,
scope: bhttpErrors
});
ofTypes = function(obj, types) {
@ -80,7 +109,7 @@ isStream = function(obj) {
};
prepareSession = function(request, response, requestState) {
debug("preparing session");
debugRequest("preparing session");
return Promise["try"](function() {
if (requestState.sessionOptions != null) {
request.options = _.merge(_.clone(requestState.sessionOptions), request.options);
@ -96,7 +125,7 @@ prepareSession = function(request, response, requestState) {
delete request.options.cookieJar;
return request.cookieJar.get(request.url);
}).then(function(cookieString) {
debug("sending cookie string: %s", cookieString);
debugRequest("sending cookie string: %s", cookieString);
request.options.headers["cookie"] = cookieString;
return Promise.resolve([request, response, requestState]);
});
@ -107,7 +136,7 @@ prepareSession = function(request, response, requestState) {
};
prepareDefaults = function(request, response, requestState) {
debug("preparing defaults");
debugRequest("preparing defaults");
return Promise["try"](function() {
var _base, _base1, _base2, _ref, _ref1, _ref2, _ref3, _ref4, _ref5, _ref6, _ref7;
request.responseOptions = {
@ -118,7 +147,9 @@ prepareDefaults = function(request, response, requestState) {
decodeJSON: (_ref4 = request.options.decodeJSON) != null ? _ref4 : false,
stream: (_ref5 = request.options.stream) != null ? _ref5 : false,
justPrepare: (_ref6 = request.options.justPrepare) != null ? _ref6 : false,
redirectLimit: (_ref7 = request.options.redirectLimit) != null ? _ref7 : 10
redirectLimit: (_ref7 = request.options.redirectLimit) != null ? _ref7 : 10,
onDownloadProgress: request.options.onDownloadProgress,
responseTimeout: request.options.responseTimeout
};
if ((_base = request.options).allowChunkedMultipart == null) {
_base.allowChunkedMultipart = false;
@ -135,7 +166,7 @@ prepareDefaults = function(request, response, requestState) {
};
prepareUrl = function(request, response, requestState) {
debug("preparing URL");
debugRequest("preparing URL");
return Promise["try"](function() {
var urlOptions, _ref;
urlOptions = urlUtil.parse(request.url, true);
@ -153,7 +184,7 @@ prepareUrl = function(request, response, requestState) {
};
prepareProtocol = function(request, response, requestState) {
debug("preparing protocol");
debugRequest("preparing protocol");
return Promise["try"](function() {
var _base;
request.protocolModule = (function() {
@ -167,7 +198,7 @@ prepareProtocol = function(request, response, requestState) {
}
})();
if (request.protocolModule == null) {
return Promise.reject()(new errors.UnsupportedProtocolError("The protocol specified (" + protocol + ") is not currently supported by this module."));
return Promise.reject()(new bhttpErrors.UnsupportedProtocolError("The protocol specified (" + protocol + ") is not currently supported by this module."));
}
if ((_base = request.options).port == null) {
_base.port = (function() {
@ -184,14 +215,14 @@ prepareProtocol = function(request, response, requestState) {
};
prepareOptions = function(request, response, requestState) {
debug("preparing options");
debugRequest("preparing options");
return Promise["try"](function() {
var _base;
if (((request.options.formFields != null) || (request.options.files != null)) && ((request.options.inputStream != null) || (request.options.inputBuffer != null))) {
return Promise.reject(addErrorData(new errors.ConflictingOptionsError("You cannot define both formFields/files and a raw inputStream or inputBuffer."), request, response, requestState));
return Promise.reject(addErrorData(new bhttpErrors.ConflictingOptionsError("You cannot define both formFields/files and a raw inputStream or inputBuffer."), request, response, requestState));
}
if (request.options.encodeJSON && ((request.options.inputStream != null) || (request.options.inputBuffer != null))) {
return Promise.reject(addErrorData(new errors.ConflictingOptionsError("You cannot use both encodeJSON and a raw inputStream or inputBuffer.", void 0, "If you meant to JSON-encode the stream, you will currently have to do so manually."), request, response, requestState));
return Promise.reject(addErrorData(new bhttpErrors.ConflictingOptionsError("You cannot use both encodeJSON and a raw inputStream or inputBuffer.", void 0, "If you meant to JSON-encode the stream, you will currently have to do so manually."), request, response, requestState));
}
if (request.responseOptions.stream) {
if ((_base = request.options).agent == null) {
@ -203,9 +234,10 @@ prepareOptions = function(request, response, requestState) {
};
preparePayload = function(request, response, requestState) {
debug("preparing payload");
debugRequest("preparing payload");
return Promise["try"](function() {
var containsStreams, fieldName, fieldValue, formDataObject, multipart, streamOptions, valueElement, _i, _len, _ref, _ref1, _ref2;
request.onUploadProgress = request.options.onUploadProgress;
multipart = request.options.forceMultipart || (request.options.files != null);
multipart = multipart || _.any(request.options.formFields, function(item) {
return item instanceof Buffer || isStream(item);
@ -215,11 +247,11 @@ preparePayload = function(request, response, requestState) {
return isStream(item);
});
if (request.options.encodeJSON && containsStreams) {
return Promise.reject()(new errors.ConflictingOptionsError("Sending a JSON-encoded payload containing data from a stream is not currently supported.", void 0, "Either don't use encodeJSON, or read your stream into a string or Buffer."));
return Promise.reject()(new bhttpErrors.ConflictingOptionsError("Sending a JSON-encoded payload containing data from a stream is not currently supported.", void 0, "Either don't use encodeJSON, or read your stream into a string or Buffer."));
}
if ((_ref = request.options.method) === "post" || _ref === "put" || _ref === "patch") {
if ((request.options.encodeJSON || (request.options.formFields != null)) && !multipart) {
debug("got url-encodable form-data");
debugRequest("got url-encodable form-data");
request.options.headers["content-type"] = "application/x-www-form-urlencoded";
if (request.options.encodeJSON) {
request.payload = JSON.stringify((_ref1 = request.options.formFields) != null ? _ref1 : null);
@ -231,7 +263,7 @@ preparePayload = function(request, response, requestState) {
request.options.headers["content-length"] = request.payload.length;
return Promise.resolve();
} else if ((request.options.formFields != null) && multipart) {
debug("got multipart form-data");
debugRequest("got multipart form-data");
formDataObject = new formData();
_ref2 = formFixArray(request.options.formFields);
for (fieldName in _ref2) {
@ -262,7 +294,7 @@ preparePayload = function(request, response, requestState) {
}
});
} else if (request.options.inputStream != null) {
debug("got inputStream");
debugRequest("got inputStream");
return Promise["try"](function() {
var _ref3;
request.payloadStream = request.options.inputStream;
@ -272,20 +304,20 @@ preparePayload = function(request, response, requestState) {
return streamLength(request.options.inputStream);
}
}).then(function(length) {
debug("length for inputStream is %s", length);
debugRequest("length for inputStream is %s", length);
return request.options.headers["content-length"] = length;
})["catch"](function(err) {
debug("unable to determine inputStream length, switching to chunked transfer encoding");
debugRequest("unable to determine inputStream length, switching to chunked transfer encoding");
return request.options.headers["content-transfer-encoding"] = "chunked";
});
} else if (request.options.inputBuffer != null) {
debug("got inputBuffer");
debugRequest("got inputBuffer");
if (typeof request.options.inputBuffer === "string") {
request.payload = new Buffer(request.options.inputBuffer);
} else {
request.payload = request.options.inputBuffer;
}
debug("length for inputBuffer is %s", request.payload.length);
debugRequest("length for inputBuffer is %s", request.payload.length);
request.options.headers["content-length"] = request.payload.length;
return Promise.resolve();
}
@ -298,10 +330,10 @@ preparePayload = function(request, response, requestState) {
};
prepareCleanup = function(request, response, requestState) {
debug("preparing cleanup");
debugRequest("preparing cleanup");
return Promise["try"](function() {
var fixedHeaders, key, value, _i, _len, _ref, _ref1;
_ref = ["query", "formFields", "files", "encodeJSON", "inputStream", "inputBuffer", "discardResponse", "keepRedirectResponses", "followRedirects", "noDecode", "decodeJSON", "allowChunkedMultipart", "forceMultipart"];
_ref = ["query", "formFields", "files", "encodeJSON", "inputStream", "inputBuffer", "discardResponse", "keepRedirectResponses", "followRedirects", "noDecode", "decodeJSON", "allowChunkedMultipart", "forceMultipart", "onUploadProgress", "onDownloadProgress"];
for (_i = 0, _len = _ref.length; _i < _len; _i++) {
key = _ref[_i];
delete request.options[key];
@ -318,7 +350,7 @@ prepareCleanup = function(request, response, requestState) {
};
prepareRequest = function(request, response, requestState) {
debug("preparing request");
debugRequest("preparing request");
return Promise["try"](function() {
var middlewareFunctions, promiseChain;
middlewareFunctions = [prepareSession, prepareDefaults, prepareUrl, prepareProtocol, prepareOptions, preparePayload, prepareCleanup];
@ -333,30 +365,65 @@ prepareRequest = function(request, response, requestState) {
};
makeRequest = function(request, response, requestState) {
debug("making %s request to %s", request.options.method.toUpperCase(), request.url);
debugRequest("making %s request to %s", request.options.method.toUpperCase(), request.url);
return Promise["try"](function() {
var req;
var req, timeoutTimer;
req = request.protocolModule.request(request.options);
if (request.payload != null) {
debug("sending payload");
req.write(request.payload);
req.end();
} else if (request.payloadStream != null) {
debug("piping payloadStream");
if (request.payloadStream._bhttpStreamWrapper != null) {
request.payloadStream.stream.pipe(req);
timeoutTimer = null;
return new Promise(function(resolve, reject) {
var completedBytes, progressStream, totalBytes;
if (request.responseOptions.responseTimeout != null) {
debugRequest("setting response timeout timer to " + request.responseOptions.responseTimeout + "ms...");
req.on("socket", function(socket) {
var timeoutHandler;
timeoutHandler = function() {
debugRequest("a response timeout occurred!");
req.abort();
return reject(addErrorData(new bhttpErrors.ResponseTimeoutError("The response timed out.")));
};
return timeoutTimer = setTimeout(timeoutHandler, request.responseOptions.responseTimeout);
});
}
totalBytes = request.options.headers["content-length"];
completedBytes = 0;
progressStream = spy(function(chunk) {
completedBytes += chunk.length;
return req.emit("progress", completedBytes, totalBytes);
});
if (request.onUploadProgress != null) {
req.on("progress", function(completedBytes, totalBytes) {
return request.onUploadProgress(completedBytes, totalBytes, req);
});
}
if (request.payload != null) {
debugRequest("sending payload");
req.emit("progress", request.payload.length, request.payload.length);
req.write(request.payload);
req.end();
} else if (request.payloadStream != null) {
debugRequest("piping payloadStream");
if (request.payloadStream._bhttpStreamWrapper != null) {
request.payloadStream.stream.pipe(progressStream).pipe(req);
} else {
request.payloadStream.pipe(progressStream).pipe(req);
}
} else {
request.payloadStream.pipe(req);
debugRequest("closing request without payload");
req.end();
}
} else {
debug("closing request without payload");
req.end();
}
return new Promise(function(resolve, reject) {
req.on("error", function(err) {
return reject(err);
if (err.code === "ETIMEDOUT") {
debugRequest("a connection timeout occurred!");
return reject(addErrorData(new bhttpErrors.ConnectionTimeoutError("The connection timed out.")));
} else {
return reject(err);
}
});
return req.on("response", function(res) {
if (timeoutTimer != null) {
debugResponse("got response in time, clearing response timeout timer");
clearTimeout(timeoutTimer);
}
return resolve(res);
});
});
@ -366,7 +433,7 @@ makeRequest = function(request, response, requestState) {
};
processResponse = function(request, response, requestState) {
debug("processing response, got status code %s", response.statusCode);
debugResponse("processing response, got status code %s", response.statusCode);
return Promise["try"](function() {
var cookieHeader, promises;
if ((request.cookieJar != null) && (response.headers["set-cookie"] != null)) {
@ -376,7 +443,7 @@ processResponse = function(request, response, requestState) {
_results = [];
for (_i = 0, _len = _ref.length; _i < _len; _i++) {
cookieHeader = _ref[_i];
debug("storing cookie: %s", cookieHeader);
debugResponse("storing cookie: %s", cookieHeader);
_results.push(request.cookieJar.set(cookieHeader, request.url));
}
return _results;
@ -386,13 +453,13 @@ processResponse = function(request, response, requestState) {
return Promise.resolve();
}
}).then(function() {
var _ref, _ref1;
var completedBytes, progressStream, totalBytes, _ref, _ref1;
response.request = request;
response.requestState = requestState;
response.redirectHistory = requestState.redirectHistory;
if (((_ref = response.statusCode) === 301 || _ref === 302 || _ref === 303 || _ref === 307) && request.responseOptions.followRedirects) {
if (requestState.redirectHistory.length >= (request.responseOptions.redirectLimit - 1)) {
return Promise.reject(addErrorData(new errors.RedirectError("The maximum amount of redirects ({request.responseOptions.redirectLimit}) was reached.")));
return Promise.reject(addErrorData(new bhttpErrors.RedirectError("The maximum amount of redirects ({request.responseOptions.redirectLimit}) was reached.")));
}
switch (response.statusCode) {
case 301:
@ -404,9 +471,9 @@ processResponse = function(request, response, requestState) {
case "put":
case "patch":
case "delete":
return Promise.reject(addErrorData(new errors.RedirectError("Encountered a 301 redirect for POST, PUT, PATCH or DELETE. RFC says we can't automatically continue."), request, response, requestState));
return Promise.reject(addErrorData(new bhttpErrors.RedirectError("Encountered a 301 redirect for POST, PUT, PATCH or DELETE. RFC says we can't automatically continue."), request, response, requestState));
default:
return Promise.reject(addErrorData(new errors.RedirectError("Encountered a 301 redirect, but not sure how to proceed for the " + (request.options.method.toUpperCase()) + " method.")));
return Promise.reject(addErrorData(new bhttpErrors.RedirectError("Encountered a 301 redirect, but not sure how to proceed for the " + (request.options.method.toUpperCase()) + " method.")));
}
break;
case 302:
@ -414,7 +481,7 @@ processResponse = function(request, response, requestState) {
return redirectGet(request, response, requestState);
case 307:
if (request.containsStreams && ((_ref1 = request.options.method) !== "get" && _ref1 !== "head")) {
return Promise.reject(addErrorData(new errors.RedirectError("Encountered a 307 redirect for POST, PUT or DELETE, but your payload contained (single-use) streams. We therefore can't automatically follow the redirect."), request, response, requestState));
return Promise.reject(addErrorData(new bhttpErrors.RedirectError("Encountered a 307 redirect for POST, PUT or DELETE, but your payload contained (single-use) streams. We therefore can't automatically follow the redirect."), request, response, requestState));
} else {
return redirectUnchanged(request, response, requestState);
}
@ -423,7 +490,23 @@ processResponse = function(request, response, requestState) {
response.resume();
return Promise.resolve(response);
} else {
totalBytes = response.headers["content-length"];
if (totalBytes != null) {
totalBytes = parseInt(totalBytes);
}
completedBytes = 0;
progressStream = sink(function(chunk) {
completedBytes += chunk.length;
return response.emit("progress", completedBytes, totalBytes);
});
if (request.responseOptions.onDownloadProgress != null) {
response.on("progress", function(completedBytes, totalBytes) {
return request.responseOptions.onDownloadProgress(completedBytes, totalBytes, response);
});
}
return new Promise(function(resolve, reject) {
response.pipe(progressStream);
response.pause();
if (request.responseOptions.stream) {
return resolve(response);
} else {
@ -464,7 +547,7 @@ doPayloadRequest = function(url, data, options, callback) {
};
redirectGet = function(request, response, requestState) {
debug("following forced-GET redirect to %s", response.headers["location"]);
debugResponse("following forced-GET redirect to %s", response.headers["location"]);
return Promise["try"](function() {
var key, options, _i, _len, _ref;
options = _.clone(requestState.originalOptions);
@ -479,7 +562,7 @@ redirectGet = function(request, response, requestState) {
};
redirectUnchanged = function(request, response, requestState) {
debug("following same-method redirect to %s", response.headers["location"]);
debugResponse("following same-method redirect to %s", response.headers["location"]);
return Promise["try"](function() {
var options;
options = _.clone(requestState.originalOptions);
@ -652,4 +735,6 @@ bhttpAPI = {
}
};
extend(bhttpAPI, bhttpErrors);
module.exports = bhttpAPI;

@ -1,6 +1,6 @@
{
"name": "bhttp",
"version": "1.0.4",
"version": "1.1.0",
"description": "A sane HTTP client library for Node.js with Streams2 support.",
"main": "index.js",
"scripts": {
@ -26,11 +26,14 @@
"concat-stream": "^1.4.7",
"debug": "^2.1.1",
"errors": "^0.2.0",
"extend": "^2.0.0",
"form-data2": "^1.0.0",
"form-fix-array": "^1.0.0",
"lodash": "^2.4.1",
"stream-length": "^1.0.2",
"string": "^3.0.0",
"through2-sink": "^1.0.0",
"through2-spy": "^1.2.0",
"tough-cookie": "^0.12.1"
},
"devDependencies": {

@ -0,0 +1,5 @@
firstCode = global.code
require "./"
secondCode = global.code
console.log global.name
console.log "Leaking global.code:", (firstCode != secondCode)

@ -0,0 +1,37 @@
Promise = require "bluebird"
bhttp = require "./"
fs = require "fs"
formatLine = (line) -> line.toString().replace(/\n/g, "\\n").replace(/\r/g, "\\r")
Promise.try ->
# multipart POST upload
console.log "Starting upload..."
bhttp.post "http://posttestserver.com/post.php",
fieldOne: "value 1"
fieldTwo: "value 2"
fieldThree: ["value 3a", "value 3b"]
fieldFour: new Buffer "value 4"
testFile: fs.createReadStream("./lower.txt")
,
headers: {"user-agent": "bhttp/test POST multipart"}
onUploadProgress: (completedBytes, totalBytes, request) ->
console.log "#{completedBytes / totalBytes * 100}%", completedBytes, totalBytes
.then (response) ->
console.log "POST multipart", formatLine(response.body)
.then ->
# GET large file
console.log "Starting download..."
bhttp.get "http://posttestserver.com/files/2015/04/06/f_00.16.102133822615",
headers: {"user-agent": "bhttp/test GET large file"}
stream: true
.then (response) ->
#setTimeout (->), 10000
console.log "Got response"
response.on "progress", (completedBytes, totalBytes, request) ->
console.log "#{completedBytes / totalBytes * 100}%", completedBytes, totalBytes
response.resume()
response.on "end", ->
console.log "Completed response download"

@ -0,0 +1,6 @@
bhttp = require "./"
console.log "make request"
bhttp.get("http://dead.dns.entry/asdfasdgasdg", responseTimeout: 4000)
bhttp.get("http://cryto.net", responseTimeout: 4000)
bhttp.get("http://dead.dns.entry/asdfasdgasdg")
Loading…
Cancel
Save