joepie91
/
dlayer-knex
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Initial commit

Browse Source
master
Sven Slootweg 10 months ago
commit 42dc99e17b
7 changed files with 626 additions and 0 deletions
Show Stats Download Patch File Download Diff File

1
.gitignore vendored
Unescape Escape View File

@ -0,0 +1 @@
node_modules

187
README.md
Unescape Escape View File

@ -0,0 +1,187 @@
# dlayer-knex
Small utility for adding database support to your dlayer schema, using Knex, with minimum viable functionality that doesn't get in your way.
- Currently supports SQLite and PostgreSQL only (as this library needs to do a level of introspection that Knex does not have implementations for)
- Deliberately very limited featureset and API surface; you are expected to manually specify *complex* queries in your graph (and dlayer's extension system makes this possible)
- Current status: __beta__. Should work fine, but if it breaks, you get to keep both halves.
- Documentation may be incomplete.
## Realistic-ish example
```js
let db = knex(require("./knexfile").development);
let knexUsers = await dlayerKnex.generateModule(db, "users");
let knexThreads = await dlayerKnex.generateModule(db, "threads", {
user_id: [ "user", "threads" ]
});
let knexPosts = await dlayerKnex.generateModule(db, "posts", {
user_id: [ "user", "posts" ],
thread_id: [ "thread", "posts" ]
});
let api = dlayer({
makeContext: () => ({}),
modules: [
knexUsers.module,
knexPosts.module,
knexThreads.module
],
schema: {
users: knexUsers.rootQueries,
threads: knexThreads.rootQueries,
posts: knexPosts.rootQueries
}
});
let { moreThan, lessThan, not } = dlayerKnex;
let testResult = await api.query({
users: {
list: {
$arguments: {
filter: { id: not(lessThan(2)) }
},
id: true,
username: true,
threads: {
id: true,
subject: true,
user: {
id: true,
username: true
}
}
}
}
});
/*
testResult = {
users: {
list: [{
id: 2,
username: 'joepie91',
threads: [{
id: 2,
subject: 'Test 2',
user: {
id: 2,
username: 'joepie91'
}
}, {
id: 3,
subject: 'Test 3',
user: {
id: 2,
username: 'joepie91'
}
}]
}, {
id: 3,
username: 'bar',
threads: [{
id: 1,
subject: 'Test 1',
user: {
id: 3,
username: 'bar'
}
}]
}, {
id: 4,
username: null,
threads: []
}]
}
}
*/
```
## API
### generateModule(knexInstance, tableName, relations)
Relations are always defined on the table which stores the referenced ID in a column; the inverse relationship will be automatically created on the foreign table.
The foreign key already needs to be set up in the database schema - `dlayer-knex` reads out the table schema to set up the relations! The values you specify here are only to indicate on which fields of the `dlayer-knex`-generated objects you want the relations to be available.
So assuming that in the schema, `threads.user_id` points to `users.id`, then with this definition:
```js
let knexThreads = await dlayerKnex.generateModule(db, "threads", {
user_id: [ "user", "threads" ]
});
```
... that means that the following fields are created:
- A `user` field on any `threads` objects created by `dlayer-knex`, referencing the corresponding object from `users` where `users.id = threads.user_id`
- A `threads` field on any `users` objects created by `dlayer-knex`, with a list of all records from the `threads` table that have `threads.user_id` set to the `users` object's `id`
(This is equivalent to a set of `belongsTo` and `hasMany` relations in many ORMs, you just only specify the destination field names in `dlayer-knex`'s config.)
Through-relations, ie. using many-to-many tables, are not *explicitly* supported in `dlayer-knex`, but they don't need to be; you can simply treat the intermediate table as its own type with its own definition and relations, and do two steps of relation-resolving in your `dlayer` query.
For example, if you have a `communities` and `users` table, then the many-to-many table that links them together might be called `memberships`, and you might end up with a query like `users[0].memberships.community` to obtain that user's communities. (This is pseudocode, dlayer does not currently use that sort of syntax, but you hopefully get the idea!)
### lessThan(value), moreThan(value)
Pretty much what it says on the tin; can be used in query arguments in place of exact values.
### anyOf(arrayOfValues)
Can be used in query arguments in place of exact values. Will filter for the value being *one of* the specified values, rather than just a single one. Cannot currently be wrapped around `lessThan`, `moreThan`, or `not` - that might be added in the future, if people have a good reason for it.
### not(value)
Negates the match. Can be wrapped around a `lessThan`/`moreThan`/`anyOf` wrapper, but not around *another* `not` wrapper. Can also be used with exact values.
## dlayer module schema
The `generateModule` method generates a dlayer module and a number of root queries, that you can use in your dlayer schema. The module defines the following things:
- A type representing records from the specified table; it will be named `dlayer-knex.TABLE_NAME`, where `TABLE_NAME` is the name of the table in the database. This is the name you should use when extending the type, eg. to add complex queries.
- Extensions for any other `dlayer-knex` types, that are necessary to generate inverse relations. You don't generally have to care about these.
- A `dlayerKnexTable` value in the module's context, which is a DataLoader that fetches items from the table by its primary key. You don't generally have to care about this, *unless* you need to work around an API limitation; in that case, you can use dlayer's `$getModuleContext` utility function to access it, but keep in mind that this context value is not included in the semver guarantees for this library, and may break between releases.
Separately, a set of root queries is also generated, which you can insert into your root schema where it fits your usecase. Often this will be in a top-level property with the same name as the table, but it doesn't *need* to be there. These root queries are:
### list
Equivalent to a `SELECT` query, retrieves existing records from a table. Possible arguments:
- __filter:__ an object of predicates to match records against, optionally using `moreThan`/`lessThan`/`anyOf`/`not`.
- __orderBy:__ a column/field name to order the results by; ascending by default, prefix it with a `-` to sort descendingly.
- __skip:__ the amount of records to skip at the start; also known as an 'offset' or 'start'.
- __limit:__ the amount of records to retrieve, counting from the first non-skipped record.
- __first:__ when set to `true`, only return the first result (or `undefined`), rather than a list of results.
This query produces a list of records according to your criteria (or a single one, if `first` has been used).
### delete
Equivalent to a `DELETE` query, deletes records from a table. Possible arguments:
- __filter:__ an object of predicates to match records against, same as for `list`. Only these records will be deleted.
Produces an object `{ count: Number }` to tell you how many records were deleted.
### change
Equivalent to an `UPDATE` query, changes existing records in a table, based on some predicate. Possible arguments:
- __filter:__ an object of predicates to match records against, same as for `list`. Only these records will be changed.
- __values:__ an object of new values to set for the matched records. Relation fields are ignored; you need to set their underlying ID references instead. Only literal values can be specified here.
This query produces a list of the matched records, *after* having been updated with the specified new values.
### create
Equivalent to an `INSERT` query, creates new records in a table. Possible arguments:
- __values:__ an array of objects, that should be created in the table. Here, again, relation fields are ignored, and you need to set their underlying ID references. Only literal values accepted.
Produces a list of the newly created records, *after* having been inserted into the database; ie. including their new automatically-assigned ID.

13
package.json
Unescape Escape View File

@ -0,0 +1,13 @@
{
"name": "dlayer-knex",
"version": "0.1.0",
"main": "src/index.js",
"repository": "https://git.cryto.net/joepie91/dlayer-knex.git",
"author": "Sven Slootweg <admin@cryto.net>",
"license": "WTFPL OR CC0-1.0",
"dependencies": {
"dataloader": "^2.2.2",
"match-value": "^1.1.0",
"syncpipe": "^1.0.0"
}
}

28
src/dataloader-from-knex-results.js
Unescape Escape View File

@ -0,0 +1,28 @@
"use strict";
const DataLoader = require("dataloader");
const syncpipe = require("syncpipe");
module.exports = function dataloaderFromKnexResults({ fetch, selectKey }) {
/* A `whereIn` query would return any results that were found, but *not* produce empty
* results for those that weren't. But DataLoader expects for exactly one item to be
* provided for each supplied ID, even if it doesn't exist.
*
* This abstraction deals with that by collecting the results from Knex, and then
* creating a DataLoader-compatible sequence that pulls from these results, leaving
* gaps for non-existent items.
*/
let selectKeyCallback = selectKey ?? ((item) => item.id);
return new DataLoader(async (keys) => {
let matches = await fetch(keys);
let byKey = syncpipe(matches, [
_ => _.map((record) => [ selectKeyCallback(record), record ]),
_ => Object.fromEntries(_)
]);
return keys.map((key) => byKey[key] ?? null);
});
};

280
src/index.js
Unescape Escape View File

@ -0,0 +1,280 @@
"use strict";
const assert = require("assert");
const matchValue = require("match-value");
const knexIntrospect = require("./introspect");
const dataloaderFromKnexResults = require("./dataloader-from-knex-results");
// TODO: Full verification that Knex doesn't allow for SQL injection anywhere dynamic values are specified below
/* Future ideas:
- Update record: give each type an update method for that individual record, by ID? For "get previous value, then set new value" style usecases
*/
function applyFilterPredicate(query, column, predicate, negate = false) {
if (predicate.__comparator === "not") {
return applyFilterPredicate(query, column, predicate.comparator, true);
} else if (predicate.__comparator === "anyOf") {
if (negate) {
return query.whereNotIn(column, predicate.value);
} else {
return query.whereIn(column, predicate.value);
}
} else {
let operator = matchValue(predicate.__comparator, {
equals: "=",
moreThan: ">",
lessThan: "<"
});
if (predicate.value === null) {
assert(operator === "=");
// Only for explicit nulls!
if (negate) {
return query.whereNotNull(column);
} else {
return query.whereNull(column);
}
} else {
if (negate) {
return query.whereNot(column, operator, predicate.value);
} else {
return query.where(column, operator, predicate.value);
}
}
}
}
function applyFilterArgument(query, filter) {
for (let [ column, predicate ] of Object.entries(filter)) {
if (predicate == null || predicate.__comparator == null) {
query = applyFilterPredicate(query, column, { __comparator: "equals", value: predicate });
} else {
query = applyFilterPredicate(query, column, predicate);
}
}
return query;
}
module.exports = {
lessThan: function (value) {
return {
__comparator: "lessThan",
value: value
};
},
moreThan: function (value) {
return {
__comparator: "moreThan",
value: value
};
},
anyOf: function (values) {
return {
__comparator: "anyOf",
value: values
};
},
not: function (comparator) {
return {
__comparator: "not",
comparator: comparator
};
},
generateModule: async function generate(globalKnex, tableName, relations = {}) {
// relations: { columnName: [ belongsTo, hasMany ] }, eg. on a posts table, it might have { thread_id: [ "thread", "posts" ] }
let tableLayout = await knexIntrospect(globalKnex, tableName);
let fieldNames = Object.keys(tableLayout);
let primaryKeyColumn = Object.values(tableLayout).find((props) => props.isPrimaryKey === true);
let primaryKey = primaryKeyColumn?.name;
assert(primaryKey != null);
// note that we cannot namespace by database name, because eg. sqlite databases don't *have* a guaranteed-unique name. using multiple databases is an uncommon case anyway, and if that is ever needed, we can add a namespace config option for those specific cases
let typeName = `dlayer-knex.${tableName}`;
let remoteTypes = {};
Object.keys(relations).map((idFieldName) => {
let remoteField = relations[idFieldName][1]; // eg. "posts"
let foreignKey = tableLayout[idFieldName].foreignKey;
let remoteTypeName = `dlayer-knex.${foreignKey.table}`;
if (remoteTypes[remoteTypeName] == null) {
remoteTypes[remoteTypeName] = {};
}
remoteTypes[remoteTypeName][remoteField] = async function(_, { $make, $getProperty, knex }) {
let tx = (knex ?? globalKnex);
let result = await tx(tableName).columns("id").where({
[idFieldName]: await $getProperty(this, primaryKey)
});
return result.map((item) => {
return $make(typeName, { id: item.id });
});
};
});
return {
typeName: typeName,
module: {
name: `knex.${tableName}`,
makeContext: (globalContext) => {
let { knex } = globalContext;
let tx = (knex ?? globalKnex);
return {
dlayerKnexTable: dataloaderFromKnexResults({
fetch: (ids) => tx(tableName).whereIn(primaryKey, ids),
selectKey: (record) => record[primaryKey]
})
};
},
types: {
[typeName]: async function({ id }) {
let normalFields = fieldNames.map((name) => {
return [ name, async (_, { dlayerKnexTable }) => {
let record = await dlayerKnexTable.load(id);
return record[name];
}];
});
let relationFields = [];
for (let name of Object.keys(relations)) {
let localField = relations[name][0]; // eg. "thread"
let foreignKey = tableLayout[name].foreignKey;
let remoteTableLayout = await knexIntrospect(globalKnex, foreignKey.table);
if (remoteTableLayout[foreignKey.column].isPrimaryKey === true) {
let field = [ localField, async function (_, { $make, $getProperty }) {
return $make(`dlayer-knex.${foreignKey.table}`, { id: await $getProperty(this, name) })
}];
relationFields.push(field);
} else {
// TODO: Throw error instead?
console.warn(`Foreign key points at column ${foreignKey.table}.${foreignKey.column}, but that column is not a primary key, and this is not supported in dlayer-knex yet`);
}
}
return Object.fromEntries([
... normalFields,
... relationFields
]);
// determine primary key field, use that as the ID attribute? then use dataloader for fetching different items, and use the same dataloader instance as a cache
}
},
extensions: remoteTypes,
root: {
// NOTE: We don't specify any root stuff here; instead we return rootQueries as a property from the call, and let the caller worry about where in the tree to put these queries (eg. it might be as part of another module)
}
},
rootQueries: {
// top-level stuff? but expect the developer to place that in a specific location
list: async function ({ filter, orderBy, skip, limit, first }, { $make, knex }) {
let tx = (knex ?? globalKnex);
let query = tx(tableName).column(primaryKey);
if (filter != null) {
query = applyFilterArgument(query, filter);
}
if (orderBy != null) {
let ascending = orderBy.startsWith("-");
let orderField = (ascending === true)
? orderBy
: orderBy.slice(1);
query = query.orderBy(orderField, (ascending === true) ? "asc" : "desc");
}
if (skip != null) {
query = query.offset(skip);
}
if (limit != null) {
query = query.limit(limit);
}
let result = await query;
// TODO: Map result(s) to types
if (first === true) {
if (result.length > 0) {
return $make(typeName, { id: result[0].id });
} else {
return null;
}
} else {
return result.map((item) => {
return $make(typeName, { id: item.id });
});
}
},
delete: async function ({ filter }, { knex }) {
assert(filter != null);
let tx = (knex ?? globalKnex);
let query = tx(tableName).delete();
query = applyFilterArgument(query, filter);
let deletedRowCount = await query;
return { count: deletedRowCount };
},
change: async function ({ filter, values }, { $make, $getModuleContext, knex }) {
assert(values != null);
let tx = (knex ?? globalKnex);
// NOTE: This is a workaround; since we pass these root queries to the developer to place in the tree themselves, they technically don't execute from within the generate module, and therefore would not normally have access to its context... so we access that context by module name here
let { dlayerKnexTable } = $getModuleContext(`knex.${tableName}`);
let query = tx(tableName)
.update(values)
.returning("id");
if (filter != null) {
applyFilterArgument(query, filter);
}
let affected = await query;
return affected.map((item) => {
dlayerKnexTable.clear(item.id); // yeet from dataloader cache, so that we fetch the updated version instead
return $make(typeName, { id: item.id });
});
},
create: async function ({ values }, { $make, knex }) {
assert(values != null);
let tx = (knex ?? globalKnex);
let isArray = Array.isArray(values);
let query = tx(tableName)
.insert(values)
.returning("id");
let result = await query;
if (isArray) {
return result.map((item) => {
return $make(typeName, { id: item.id });
});
} else {
return $make(typeName, { id: result[0].id })
}
}
}
};
}
};

92
src/introspect.js
Unescape Escape View File

@ -0,0 +1,92 @@
"use strict";
// TODO: Make this its own independent package once the last bits have been polished (eg. consistent type names and onUpdate/onDelete values)
const assert = require("assert");
module.exports = async function knexIntrospect(knex, tableName) {
let clientType = knex.client.config.client;
assert(clientType === "sqlite3" || clientType === "pg");
assert(/^[a-zA_Z_]+$/.test(tableName));
let safeTableName = tableName; // FIXME: Sanitize instead of assert?
let tableLayout = {};
if (clientType === "sqlite3") {
let [ columns, foreignKeys ] = await Promise.all([
knex.raw(`PRAGMA table_info('${safeTableName}')`),
knex.raw(`PRAGMA foreign_key_list('${safeTableName}')`)
]);
for (let column of columns) {
tableLayout[column.name] = {
name: column.name,
type: column.type,
notNull: (column.notnull === 1),
isPrimaryKey: (column.pk === 1),
foreignKey: null
};
}
for (let fk of foreignKeys) {
tableLayout[fk.from].foreignKey = {
table: fk.table,
column: fk.to,
onUpdate: fk.on_update,
onDelete: fk.on_delete
};
}
} else if (clientType === "pg") {
// Yes, this is a lot of queries. Yes, I "should" have used a single query with JOINs. If I did that, I would now have 1) a headache and 2) no working code. If this is something you care about, feel free to submit a PR.
let tables = await knex("pg_class").limit(1).where({ relname: tableName });
assert(tables.length === 1);
let table = tables[0];
let columns = await knex("pg_attribute")
.where({ attrelid: table.oid })
.where("attnum", ">", 0);
for (let column of columns) {
let constraints = await knex("pg_constraint")
.where({ conrelid: table.oid })
.whereRaw(`conkey = ARRAY[?::SMALLINT]`, [ column.attnum ]);
let primaryKeyConstraint, foreignKey;
for (let constraint of constraints) {
if (constraint.contype === "p") {
primaryKeyConstraint = constraint;
} else if (constraint.contype === "f") {
// TODO: Can there be multiple foreign key constraints?
let foreignTable = await knex("pg_class").limit(1).where({ oid: constraint.confrelid });
// TODO: Support composite keys?
assert(constraint.confkey.length === 1);
let foreignColumn = await knex("pg_attribute")
.limit(1)
.where({ attrelid: constraint.confrelid })
.whereRaw(`attnum = ?::SMALLINT`, [ constraint.confkey[0] ]);
foreignKey = {
table: foreignTable[0].relname,
column: foreignColumn[0].attname,
// TODO: Convert possible values for onUpdate/onDelete into something consistent with SQLite, using match-value
onUpdate: (constraint.confupdtype === "a") ? null : constraint.confupdtype,
onDelete: (constraint.confdeltype === "a") ? null : constraint.confdeltype
};
}
}
tableLayout[column.attname] = {
name: column.attname,
type: (await knex("pg_type").limit(1).where({ oid: column.atttypid }))[0].typname,
notNull: column.attnotnull,
isPrimaryKey: (primaryKeyConstraint != null),
foreignKey: foreignKey
};
}
}
return tableLayout;
};

25
yarn.lock
Unescape Escape View File

@ -0,0 +1,25 @@
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1
assure-array@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/assure-array/-/assure-array-1.0.0.tgz#4f4ad16a87659d6200a4fb7103462033d216ec1f"
integrity sha512-igvOvGYidAcJKr6YQIHzLivUpAdqUfi7MN0QfrEnFtifQvuw6D0W4oInrIVgTaefJ+QBVWAj8ZYuUGNnwq6Ydw==
dataloader@^2.2.2:
version "2.2.2"
resolved "https://registry.yarnpkg.com/dataloader/-/dataloader-2.2.2.tgz#216dc509b5abe39d43a9b9d97e6e5e473dfbe3e0"
integrity sha512-8YnDaaf7N3k/q5HnTJVuzSyLETjoZjVmHc4AeKAzOvKHEFQKcn64OKBfzHYtE9zGjctNM7V9I0MfnUVLpi7M5g==
match-value@^1.1.0:
version "1.1.0"
resolved "https://registry.yarnpkg.com/match-value/-/match-value-1.1.0.tgz#ad311ef8bbe2d344a53ec3104e28fe221984b98e"
integrity sha512-NOvpobcmkX+l9Eb6r2s3BkR1g1ZwzExDFdXA9d6p1r1O1olLbo88KuzMiBmg43xSpodfm7I6Hqlx2OoySquEgg==
syncpipe@^1.0.0:
version "1.0.0"
resolved "https://registry.yarnpkg.com/syncpipe/-/syncpipe-1.0.0.tgz#170340f813150bc8fcb8878b1b9c71ea0ccd3727"
integrity sha512-cdiAFTnFJRvUaNPDc2n9CqoFvtIL3+JUMJZrC3kA3FzpugHOqu0TvkgNwmnxPZ5/WjAzMcfMS3xm+AO7rg/j/w==
dependencies:
assure-array "^1.0.0"
Write Preview
Loading…
Cancel
Save