Research questions: - GraphQL schema composability - GraphQL DB integration - Existing GraphQL -> DB adapters tend to require a network (HTTP API) layer, instead of providing a direct GraphQL integration. May be more useful to hand-roll the adapter. - Ergonomic reusable filters for GraphQL nodes, potentially using custom directives - How to define non-global custom types in GraphQL - How to combine literal text-based schema definitions with programmatic schema construction, and whether this is necessary - How to represent progress for long-running tasks in a GraphQL mutation, eg. for an "add ISO from HTTP URL" operation Todo list: - UI: Convert existing Pug templates to JSX/GraphQL - Finalize the conversion of the physical drives page (totals etc.) - GraphQL API: Add database support - GraphQL API: Memory resources (usage, available, etc.) - Wrappers: Add error handling to smartctl wrapper - Wrappers: Complete LVM wrapper - Internals: Storage pool implementations - GraphQL API: Storage pool support - Internals: ISO/template pool implementation - GraphQL API: ISO/template pool support ####################################### qemu-img create -f qcow2 test.qcow 5G qemu-kvm -cdrom ~/Downloads/debian-8.5.0-amd64-netinst.iso -hda test.qcow -boot d -netdev user,id=user.0 -device e1000,netdev=user.0 -m 196 -localtime # Getting Started https://fedoraproject.org/wiki/How_to_use_qemu https://en.wikibooks.org/wiki/QEMU/Images https://nixos.org/wiki/QEMU_guest_with_networking_and_virtfs https://mbharatkumar.wordpress.com/2010/10/09/qemu-getting-started/ http://www.nico.schottelius.org/blog/control-and-shutdown-qemu-kvm-vm-via-unix-socket/ http://man.cx/qemu-system-x86_64(1) # Flags https://wiki.gentoo.org/wiki/QEMU/Options # Monitor https://en.wikibooks.org/wiki/QEMU/Monitor # QMP https://raw.githubusercontent.com/qemu/qemu/master/docs/qmp-intro.txt https://raw.githubusercontent.com/qemu/qemu/master/docs/qmp-spec.txt https://raw.githubusercontent.com/qemu/qemu/master/qmp-commands.hx https://raw.githubusercontent.com/qemu/qemu/master/docs/qmp-events.txt http://wiki.qemu.org/QMP https://kashyapc.com/2013/03/31/multiple-ways-to-access-qemu-machine-protocol-qmp/ https://www.npmjs.com/package/qemu-qmp # QMP (old) https://qemu.weilnetz.de/w64/2012/2012-06-28/qmp-commands.txt https://lxr.missinglinkelectronics.com/qemu/qmp-commands.hx # Networking http://www.linux-kvm.org/page/Networking http://wiki.qemu.org/Documentation/Networking https://pve.proxmox.com/wiki/Network_Model http://hyperlogos.org/page/HOWTO-kvm-vde-networking-Ubuntu-Debian-et-cetera # Disk Images https://alexeytorkhov.blogspot.nl/2009/09/mounting-raw-and-qcow2-vm-disk-images.html https://edoceo.com/cli/qemu http://libguestfs.org/ http://www.linux-kvm.org/page/Change_cdrom # Storage Pools https://libvirt.org/storage.html # Virtio http://www.linux-kvm.org/page/Boot_from_virtio_block_device # Resource throttling https://vpsboard.com/topic/4601-kvm-anti-abuse-how-do-you-counter-abuse-with-kvm-users/ - "cgroups cpuacct not a workable solution?" https://vpsboard.com/topic/4601-kvm-anti-abuse-how-do-you-counter-abuse-with-kvm-users/?do=findComment&comment=66078 - "aka as each vm is a prossess use nice to limit cpu." http://wiki.qemu.org/Features/AutoconvergeLiveMigration - dynamic CPU throttling lead... https://lists.gnu.org/archive/html/qemu-devel/2015-06/msg06737.html # References http://www.linux-kvm.org/page/HOWTO http://www.linux-kvm.org/page/Management_Tools https://github.com/ChoHag/nbsvm/blob/master/nbsvm https://github.com/digitalocean/go-qemu/ # Full documentation http://wiki.qemu.org/download/qemu-doc.html { "return": [{ "io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "inserted": { "iops_rd": 0, "detect_zeroes": "off", "image": { "virtual-size": 258998272, "filename": "/home/sven/Downloads/debian-8.5.0-amd64-netinst.iso", "format": "raw", "actual-size": 259006464, "dirty-flag": false }, "iops_wr": 0, "ro": true, "node-name": "#block112", "backing_file_depth": 0, "drv": "raw", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": { "no-flush": false, "direct": false, "writeback": true }, "file": "/home/sven/Downloads/debian-8.5.0-amd64-netinst.iso", "encryption_key_missing": false }, "tray_open": false, "type": "unknown" }, { "io-status": "ok", "device": "ide0-hd0", "locked": false, "removable": false, "inserted": { "iops_rd": 0, "detect_zeroes": "off", "image": { "virtual-size": 5368709120, "filename": "test.qcow", "cluster-size": 65536, "format": "qcow2", "actual-size": 1939349504, "format-specific": { "type": "qcow2", "data": { "compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false } }, "dirty-flag": false }, "iops_wr": 0, "ro": false, "node-name": "#block321", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": { "no-flush": false, "direct": false, "writeback": true }, "file": "test.qcow", "encryption_key_missing": false }, "type": "unknown" }, { "device": "floppy0", "locked": false, "removable": true, "tray_open": true, "type": "unknown" }, { "device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown" }] } { "return": [{ "io-status": "ok", "device": "ide0-hd0", "locked": false, "removable": false, "inserted": { "iops_rd": 0, "detect_zeroes": "off", "image": { "virtual-size": 5368709120, "filename": "test.qcow", "cluster-size": 65536, "format": "qcow2", "actual-size": 1939349504, "format-specific": { "type": "qcow2", "data": { "compat": "1.1", "lazy-refcounts": false, "refcount-bits": 16, "corrupt": false } }, "dirty-flag": false }, "iops_wr": 0, "ro": false, "node-name": "#block121", "backing_file_depth": 0, "drv": "qcow2", "iops": 0, "bps_wr": 0, "write_threshold": 0, "encrypted": false, "bps": 0, "bps_rd": 0, "cache": { "no-flush": false, "direct": false, "writeback": true }, "file": "test.qcow", "encryption_key_missing": false }, "type": "unknown" }, { "io-status": "ok", "device": "ide1-cd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown" }, { "device": "floppy0", "locked": false, "removable": true, "tray_open": true, "type": "unknown" }, { "device": "sd0", "locked": false, "removable": true, "tray_open": false, "type": "unknown" }] } { "return": [{ "name": "query-rocker-of-dpa-groups" }, { "name": "query-rocker-of-dpa-flows" }, { "name": "query-rocker-ports" }, { "name": "query-rocker" }, { "name": "block-set-write-threshold" }, { "name": "x-input-send-event" }, { "name": "trace-event-set-state" }, { "name": "trace-event-get-state" }, { "name": "rtc-reset-reinjection" }, { "name": "query-acpi-ospm-status" }, { "name": "query-memory-devices" }, { "name": "query-memdev" }, { "name": "blockdev-change-medium" }, { "name": "query-named-block-nodes" }, { "name": "x-blockdev-insert-medium" }, { "name": "x-blockdev-remove-medium" }, { "name": "blockdev-close-tray" }, { "name": "blockdev-open-tray" }, { "name": "x-blockdev-del" }, { "name": "blockdev-add" }, { "name": "query-rx-filter" }, { "name": "chardev-remove" }, { "name": "chardev-add" }, { "name": "query-tpm-types" }, { "name": "query-tpm-models" }, { "name": "query-tpm" }, { "name": "query-target" }, { "name": "query-cpu-definitions" }, { "name": "query-machines" }, { "name": "device-list-properties" }, { "name": "qom-list-types" }, { "name": "change-vnc-password" }, { "name": "nbd-server-stop" }, { "name": "nbd-server-add" }, { "name": "nbd-server-start" }, { "name": "qom-get" }, { "name": "qom-set" }, { "name": "qom-list" }, { "name": "query-block-jobs" }, { "name": "query-balloon" }, { "name": "query-migrate-parameters" }, { "name": "migrate-set-parameters" }, { "name": "query-migrate-capabilities" }, { "name": "migrate-set-capabilities" }, { "name": "query-migrate" }, { "name": "query-command-line-options" }, { "name": "query-uuid" }, { "name": "query-name" }, { "name": "query-spice" }, { "name": "query-vnc-servers" }, { "name": "query-vnc" }, { "name": "query-mice" }, { "name": "query-status" }, { "name": "query-kvm" }, { "name": "query-pci" }, { "name": "query-iothreads" }, { "name": "query-cpus" }, { "name": "query-blockstats" }, { "name": "query-block" }, { "name": "query-chardev-backends" }, { "name": "query-chardev" }, { "name": "query-qmp-schema" }, { "name": "query-events" }, { "name": "query-commands" }, { "name": "query-version" }, { "name": "human-monitor-command" }, { "name": "qmp_capabilities" }, { "name": "add_client" }, { "name": "expire_password" }, { "name": "set_password" }, { "name": "block_set_io_throttle" }, { "name": "block_passwd" }, { "name": "query-fdsets" }, { "name": "remove-fd" }, { "name": "add-fd" }, { "name": "closefd" }, { "name": "getfd" }, { "name": "set_link" }, { "name": "balloon" }, { "name": "change-backing-file" }, { "name": "drive-mirror" }, { "name": "blockdev-snapshot-delete-internal-sync" }, { "name": "blockdev-snapshot-internal-sync" }, { "name": "blockdev-snapshot" }, { "name": "blockdev-snapshot-sync" }, { "name": "block-dirty-bitmap-clear" }, { "name": "block-dirty-bitmap-remove" }, { "name": "block-dirty-bitmap-add" }, { "name": "transaction" }, { "name": "block-job-complete" }, { "name": "block-job-resume" }, { "name": "block-job-pause" }, { "name": "block-job-cancel" }, { "name": "block-job-set-speed" }, { "name": "blockdev-backup" }, { "name": "drive-backup" }, { "name": "block-commit" }, { "name": "block-stream" }, { "name": "block_resize" }, { "name": "object-del" }, { "name": "object-add" }, { "name": "netdev_del" }, { "name": "netdev_add" }, { "name": "query-dump-guest-memory-capability" }, { "name": "dump-guest-memory" }, { "name": "client_migrate_info" }, { "name": "migrate_set_downtime" }, { "name": "migrate_set_speed" }, { "name": "query-migrate-cache-size" }, { "name": "migrate-start-postcopy" }, { "name": "migrate-set-cache-size" }, { "name": "migrate-incoming" }, { "name": "migrate_cancel" }, { "name": "migrate" }, { "name": "xen-set-global-dirty-log" }, { "name": "xen-save-devices-state" }, { "name": "ringbuf-read" }, { "name": "ringbuf-write" }, { "name": "inject-nmi" }, { "name": "pmemsave" }, { "name": "memsave" }, { "name": "cpu-add" }, { "name": "cpu" }, { "name": "send-key" }, { "name": "device_del" }, { "name": "device_add" }, { "name": "system_powerdown" }, { "name": "system_reset" }, { "name": "system_wakeup" }, { "name": "cont" }, { "name": "stop" }, { "name": "screendump" }, { "name": "change" }, { "name": "eject" }, { "name": "quit" }] } ----------------- # Storage pools - LVM - Preallocated - Sparse ("thin provisioning") - File - Preallocated (raw) - Sparse (qcow2) # Image pools Make synchronization settable per image pool? # Network pools? -------------- Consider how to check disk space (total, used, allocated) and how to display overcommitting to the user ------ Reading filesystems: /proc/self/mountinfo https://github.com/coreutils/gnulib/blob/master/lib/mountlist.c https://github.com/coreutils/coreutils/blob/master/src/df.c Reading system info in general: https://github.com/uutils/coreutils (Rust) ------ Schema stuff Storage pools: - node ID - path (for file) - volume group name (for LVM) Storage space/volume: - pool ID - VM ID ------- Tools - resize2fs: resizing filesystems ------ Safety/reliability - Lock a storage pool when a pvmove is in progress? How to detect that when it isn't done through the CVM interface? ------ MARKER: - LVM tools abstraction (CLI wrapper) - Storage pool implementation for LVM and file-based (test LVM via loopback devices) --------- LVM stuff To display available report fields: -o help To return data in JSON format: --reportformat json --------- error-chain - exposed predicate function for "has error type somewhere in chain", for Bluebird .catch ---------- smartctl flags PO--CK 0x0033 51 0 0 1 1 0 0 1 1 -O--CK 0x0032 50 0 0 1 1 0 0 1 0 ----CK 0x0030 48 0 0 1 1 0 0 0 0 POSR-K 0x002f 47 0 0 1 0 1 1 1 1 -OSR-K 0x002e 46 0 0 1 0 1 1 1 0 POS--K 0x0027 39 0 0 1 0 0 1 1 1 -O---K 0x0022 34 0 0 1 0 0 0 1 0 ---R-- 0x0008 8 0 0 0 0 1 0 0 0 K C R S O P | | | | | |_ P prefailure warning | | | | |__ O updated online | | | |___ S speed/performance | | |____ R error rate | |_____ C event count |______ K auto-keep ------ FIXME: Do a self-test after installation, to verify that all the parsing code works on the user's system (wrt possibly different versions of tools), and write an UnexpectedOutput/ExpectedOutputMissing handler that lets the user report any failures. ------ Security stuff: - CSRF? - Helmet? - Zeroing volumes after/before use ------ # Stuff to add Hardware -> Storage Devices - Disk space used - Disk space allocated (can be over 100% with thin provisioning!) - Allocation:usage ratio - IOPS - Read/Write traffic - Read/Write latency ------ lsblk - Use `bytes` flag to get sizes in JSON output in bytes, rather than as a unit string!