You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
80 lines
1.5 KiB
Plaintext
80 lines
1.5 KiB
Plaintext
# TYPES
|
|
|
|
- event daemon(?) for eg. nodemon
|
|
|
|
- task
|
|
- pipeline
|
|
- step (source, transform, sink)
|
|
- file
|
|
- failure
|
|
|
|
## Task
|
|
|
|
Represents a task/job.
|
|
|
|
Contains:
|
|
- Pipeline or event daemon
|
|
|
|
## Pipeline
|
|
|
|
Represents a sequence of operations, from source to sink.
|
|
|
|
Contains:
|
|
- Array of steps
|
|
|
|
## Step
|
|
|
|
Can be either source, transform, or sink -- a transform conceptually counts as both source and sink, sort of.
|
|
|
|
Contains:
|
|
- __buildStep (type: source, transform, sink)
|
|
- displayName
|
|
- Source only:
|
|
- supportsTeardown
|
|
- basePath
|
|
- initialize (logic)
|
|
- teardown (logic)
|
|
- Transform only:
|
|
- supportsStreams
|
|
- supportsVirtualFiles
|
|
- supportsVirtualFilesystem
|
|
- transform (logic)
|
|
- Sink only:
|
|
- supportsStreams
|
|
- supportsAcknowledgment
|
|
- sink (logic)
|
|
|
|
## File
|
|
|
|
A file, real or virtual; equates to a successful event on a source/transform.
|
|
|
|
Contains:
|
|
- Metadata
|
|
- Contents or Stream
|
|
|
|
## Metadata
|
|
|
|
All the data that describes the file; nests to refer to parent input file's metadata.
|
|
|
|
Contains:
|
|
- inputFileMetadata (only when originating from transform or sink)
|
|
- step: The step that this file originates from
|
|
- stepDuration: How long the originating step took
|
|
- sourcedDate (only when originating from source)
|
|
- lastModified (optional)
|
|
- targetPath (save-to-directory sink only?)
|
|
- relativePath
|
|
- basePath
|
|
|
|
|
|
## Failure
|
|
|
|
Signifies that something went wrong.
|
|
|
|
Contains:
|
|
- error: Error object/data
|
|
- step: Step that it originated from
|
|
- file: Input file (only for sinks/transforms)
|
|
- isFatal
|
|
- duration (only for transforms)
|