3.6 KiB
cdx
A simple streaming CDX file parser. Parses CDX files (in particular, those corresponding to WARC files) that correspond to the format as specified by the Internet Archive. All items in the CDX field legend are supported, plus the S
field.
Scope and development status
cdx
currently only reads (compliant) CDX streams. In the future, it will likely be expanded to also be able to write CDX streams, but this is not currently supported. Error handling is currently nearly non-existent - you are expected to provide a compliant CDX stream.
Installation
npm install --save cdx
Usage
cdx
is a streaming parser. It takes a CDX byte stream as input (regardless of the source), and outputs an object stream of CDXRecord
objects with the named attributes set to the corresponding values from the CDX stream. Additionally, a plain object containing these attributes is available as the data
attribute, for easy (JSON) serialization.
The signature is automatically parsed from the first line of the CDX data. Specifying a custom signature is not currently supported. If a field is not specified (that is, it is either not listed in the signature or its value is -
), it will simply not be set on the CDXRecord
.
var cdx = require("cdx"),
fs = require("fs");
fs.createReadStream("sample.cdx")
.pipe(cdx())
.pipe(...);
An example that parses the sample CDX file, 'picks out' the serializable data, and then outputs it to stdout
as serialized JSON, can be found in sample.cdx
(you'll need to install devDependencies first to actually run that file, though).
Fields
All fields are self-explanatory, hopefully. These are just adapted from the legend provided by the Internet Archive, so I really have no idea what most of these do.
compressedRecordSize
(for.warc.gz
, this is the gzipped size of the record)compressedDATFileOffset
compressedARCFileOffset
(for.warc.gz
, this is the gzipped starting position of the record, combine with size to get the ending position)uncompressedDATFileOffset
uncompressedARCFileOffset
ARCDocumentLength
oldStyleChecksum
newStyleChecksum
canonicalizedUrl
canonicalizedFrame
canonicalizedHost
canonicalizedImage
canonicalizedJumpPoint
canonicalizedLink
canonicalizedPath
canonicalizedRedirect
canonicalizedHrefURL
canonicalizedSrcURL
canonicalizedScriptURL
originalMimeType
(for.warc.gz
, this is the original mimetype of the document as specified by the origin webserver)originalURL
(for.warc.gz
, this is the original URL that the document was retrieved from)originalFrame
originalHost
originalImage
originalJumpPoint
originalLink
originalPath
originalRedirect
originalHrefURL
originalSrcURL
originalScriptURL
date
(for.warc.gz
, this is the retrieval date of the record)IP
fileName
(for.warc.gz
, this is the path to the WARC file that this record lives in - may not be useful, as it may refer to a path on a different filesystem)port
responseCode
(for.warc.gz
, this is the HTTP status code encountered when retrieving the document)title
metaTags
massagedURL
languageString
uniqueness
newsGroup
rulespaceCategory
multiColumnLanguageDescription
someWeirdFBISWhatsChangedKindaThing
(don't ask...)comment
Contributing
Contributions welcome! Please file bugs on GitHub, and target pull requests at the develop
branch. Thank you!