joepie91
/
pegjs
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
redux
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '86769a6c5c'
${ noResults }
pegjs/spec/compiler/passes/generate-bytecode.spec.js

691 lines
24 KiB
JavaScript
Raw Normal View History Unescape Escape

Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
describe("compiler pass |generateBytecode|", function() {
Plugin API: Split compiler passes into stages The compiler passes are now split into three stages: * check -- passes that check for various error conditions * transform -- passes that transform the AST (e.g. to perform optimizations) * generate -- passes that are related to code generation Splitting the passes into stages is important for plugins. For example, if a plugin wants to add a new optimization pass, it can add it at the end of the "transform" stage without any knowledge about other passes it contains. Similarly, if it wants to generate something else than the default code generator does from the AST, it can just replace all passes in the "generate" stage by its own one(s). More generally, the stages make it possible to write plugins that do not depend on names and actions of specific passes (which I consider internal and subject of change), just on the definition of stages (which I consider a public API with to which semver rules apply). Implements part of GH-106.
11 years ago
var pass = PEG.compiler.passes.generate.generateBytecode;
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
function bytecodeDetails(bytecode) {
return {
rules: [{ bytecode: bytecode }]
};
}
function constsDetails(consts) { return { consts: consts }; }
describe("for grammar", function() {
it("generates correct bytecode", function() {
expect(pass).toChangeAST([
'a = "a"',
'b = "b"',
'c = "c"'
].join("\n"), {
rules: [
{ bytecode: [15, 0, 2, 2, 19, 0, 20, 1] },
{ bytecode: [15, 2, 2, 2, 19, 2, 20, 3] },
{ bytecode: [15, 4, 2, 2, 19, 4, 20, 5] }
]
});
});
it("defines correct constants", function() {
expect(pass).toChangeAST([
'a = "a"',
'b = "b"',
'c = "c"'
].join("\n"), constsDetails([
'"a"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "a", description: "\\"a\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"b"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "b", description: "\\"b\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"c"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "c", description: "\\"c\\"" }'
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
]));
});
});
describe("for rule", function() {
it("generates correct bytecode", function() {
expect(pass).toChangeAST('start = "a"', bytecodeDetails([
15, 0, 2, 2, 19, 0, 20, 1 // <expression>
]));
});
});
describe("for named", function() {
var grammar = 'start "start" = "a"';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
25, // SILENT_FAILS_ON
15, 1, 2, 2, 19, 1, 20, 2, // <expression>
26, // SILENT_FAILS_OFF
11, 2, 0, // IF_ERROR
20, 0 // FAIL
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
'{ type: "other", description: "start" }',
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("for choice", function() {
it("generates correct bytecode", function() {
expect(pass).toChangeAST('start = "a" / "b" / "c"', bytecodeDetails([
15, 0, 2, 2, 19, 0, 20, 1, // <alternatives[0]>
11, 21, 0, // IF_ERROR
2, // * POP
15, 2, 2, 2, 19, 2, 20, 3, // <alternatives[1]>
11, 9, 0, // IF_ERROR
2, // * POP
15, 4, 2, 2, 19, 4, 20, 5 // <alternatives[2]>
]));
});
});
describe("for action", function() {
describe("without labels", function() {
var grammar = 'start = { code }';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
1, // PUSH_CURR_POS
0, 0, // PUSH
12, 6, 0, // IF_NOT_ERROR
21, 1, // * REPORT_SAVED_POS
23, 1, 1, 0, // CALL
11, 1, 1, // IF_ERROR
6, // * NIP_CURR_POS
5 // * NIP
]));
});
it("defines correct constants", function() {
expect(pass).toChangeAST(
grammar,
constsDetails(['[]', 'function() { code }'])
);
});
});
describe("with one label", function() {
var grammar = 'start = a:"a" { code }';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
1, // PUSH_CURR_POS
15, 0, 2, 2, 19, 0, 20, 1, // <expression>
12, 7, 0, // IF_NOT_ERROR
21, 1, // * REPORT_SAVED_POS
23, 2, 1, 1, 0, // CALL
11, 1, 1, // IF_ERROR
6, // * NIP_CURR_POS
5 // * NIP
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }',
'function(a) { code }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("with multiple labels", function() {
var grammar = 'start = a:"a" b:"b" c:"c" { code }';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
1, // PUSH_CURR_POS
15, 1, 2, 2, 19, 1, 20, 2, // <elements[0]>
12, 46, 4, // IF_NOT_ERROR
15, 3, 2, 2, 19, 3, 20, 4, // * <elements[1]>
12, 30, 5, // IF_NOT_ERROR
15, 5, 2, 2, 19, 5, 20, 6, // * <elements[2]>
12, 14, 5, // IF_NOT_ERROR
21, 3, // * REPORT_SAVED_POS
23, 7, 3, 3, 2, 1, 0, // CALL
11, 1, 1, // IF_ERROR
6, // * NIP_CURR_POS
5, // * NIP
4, 3, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
4, 2, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
2, // * POP
3, // POP_CURR_POS
0, 0 // PUSH
]));
});
it("defines correct constants", function() {
expect(pass).toChangeAST(grammar, constsDetails([
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'peg$FAILED',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"a"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "a", description: "\\"a\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"b"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "b", description: "\\"b\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"c"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "c", description: "\\"c\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'function(a, b, c) { code }'
]));
});
});
});
describe("for sequence", function() {
describe("empty", function() {
var grammar = 'start = ';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
0, 0 // PUSH
]));
});
it("defines correct constants", function() {
expect(pass).toChangeAST(grammar, constsDetails(['[]']));
});
});
describe("non-empty", function() {
var grammar = 'start = "a" "b" "c"';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
1, // PUSH_CURR_POS
15, 1, 2, 2, 19, 1, 20, 2, // <elements[0]>
12, 35, 4, // IF_NOT_ERROR
15, 3, 2, 2, 19, 3, 20, 4, // * <elements[1]>
12, 19, 5, // IF_NOT_ERROR
15, 5, 2, 2, 19, 5, 20, 6, // * <elements[2]>
12, 3, 5, // IF_NOT_ERROR
8, 3, // * WRAP
5, // NIP
4, 3, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
4, 2, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
2, // * POP
3, // POP_CURR_POS
0, 0 // PUSH
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'peg$FAILED',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }',
'"b"',
'{ type: "literal", value: "b", description: "\\"b\\"" }',
'"c"',
'{ type: "literal", value: "c", description: "\\"c\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
});
describe("for labeled", function() {
it("generates correct bytecode", function() {
expect(pass).toChangeAST('start = a:"a"', bytecodeDetails([
15, 0, 2, 2, 19, 0, 20, 1 // <expression>
]));
});
});
describe("for text", function() {
it("generates correct bytecode", function() {
expect(pass).toChangeAST('start = $"a"', bytecodeDetails([
1, // PUSH_CURR_POS
15, 0, 2, 2, 19, 0, 20, 1, // <expression>
12, 1, 0, // IF_NOT_ERROR
9, // * TEXT
5 // NIP
]));
});
});
describe("for simple and", function() {
var grammar = 'start = &"a"';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
1, // PUSH_CURR_POS
25, // SILENT_FAILS_ON
15, 2, 2, 2, 19, 2, 20, 3, // <expression>
26, // SILENT_FAILS_OFF
12, 4, 4, // IF_NOT_ERROR
2, // * POP
3, // POP_CURR_POS
0, 0, // PUSH
2, // * POP
2, // POP
0, 1 // PUSH
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
'""',
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'peg$FAILED',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("for simple not", function() {
var grammar = 'start = !"a"';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
1, // PUSH_CURR_POS
25, // SILENT_FAILS_ON
15, 2, 2, 2, 19, 2, 20, 3, // <expression>
26, // SILENT_FAILS_OFF
11, 4, 4, // IF_ERROR
2, // * POP
2, // POP
0, 0, // PUSH
2, // * POP
3, // POP_CURR_POS
0, 1 // PUSH
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
'""',
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'peg$FAILED',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("for semantic and", function() {
describe("without labels", function() {
var grammar = 'start = &{ code }';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
22, // REPORT_CURR_POS
23, 0, 0, 0, // CALL
10, 3, 3, // IF
2, // * POP
0, 1, // PUSH
2, // * POP
0, 2 // PUSH
]));
});
it("defines correct constants", function() {
expect(pass).toChangeAST(
grammar,
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
constsDetails(['function() { code }', '""', 'peg$FAILED'])
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
);
});
});
describe("with labels", function() {
var grammar = 'start = a:"a" b:"b" c:"c" &{ code }';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
1, // PUSH_CURR_POS
15, 1, 2, 2, 19, 1, 20, 2, // <elements[0]>
12, 60, 4, // IF_NOT_ERROR
15, 3, 2, 2, 19, 3, 20, 4, // * <elements[1]>
12, 44, 5, // IF_NOT_ERROR
15, 5, 2, 2, 19, 5, 20, 6, // * <elements[2]>
12, 28, 5, // IF_NOT_ERROR
22, // * REPORT_CURR_POS
23, 7, 0, 3, 2, 1, 0, // CALL
10, 3, 3, // IF
2, // * POP
0, 8, // PUSH
2, // * POP
0, 0, // PUSH
12, 3, 5, // IF_NOT_ERROR
8, 4, // * WRAP
5, // NIP
4, 4, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
4, 3, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
4, 2, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
2, // * POP
3, // POP_CURR_POS
0, 0 // PUSH
]));
});
it("defines correct constants", function() {
expect(pass).toChangeAST(grammar, constsDetails([
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'peg$FAILED',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"a"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "a", description: "\\"a\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"b"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "b", description: "\\"b\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"c"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "c", description: "\\"c\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'function(a, b, c) { code }',
'""'
]));
});
});
});
describe("for semantic not", function() {
describe("without labels", function() {
var grammar = 'start = !{ code }';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
22, // REPORT_CURR_POS
23, 0, 0, 0, // CALL_PREDICATE
10, 3, 3, // IF
2, // * POP
0, 2, // PUSH
2, // * POP
0, 1 // PUSH
]));
});
it("defines correct constants", function() {
expect(pass).toChangeAST(
grammar,
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
constsDetails(['function() { code }', '""', 'peg$FAILED'])
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
);
});
});
describe("with labels", function() {
var grammar = 'start = a:"a" b:"b" c:"c" !{ code }';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
1, // PUSH_CURR_POS
15, 1, 2, 2, 19, 1, 20, 2, // <elements[0]>
12, 60, 4, // IF_NOT_ERROR
15, 3, 2, 2, 19, 3, 20, 4, // * <elements[1]>
12, 44, 5, // IF_NOT_ERROR
15, 5, 2, 2, 19, 5, 20, 6, // * <elements[2]>
12, 28, 5, // IF_NOT_ERROR
22, // * REPORT_CURR_POS
23, 7, 0, 3, 2, 1, 0, // CALL
10, 3, 3, // IF
2, // * POP
0, 0, // PUSH
2, // * POP
0, 8, // PUSH
12, 3, 5, // IF_NOT_ERROR
8, 4, // * WRAP
5, // NIP
4, 4, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
4, 3, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
4, 2, // * POP_N
3, // POP_CURR_POS
0, 0, // PUSH
2, // * POP
3, // POP_CURR_POS
0, 0 // PUSH
]));
});
it("defines correct constants", function() {
expect(pass).toChangeAST(grammar, constsDetails([
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'peg$FAILED',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"a"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "a", description: "\\"a\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"b"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "b", description: "\\"b\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'"c"',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'{ type: "literal", value: "c", description: "\\"c\\"" }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'function(a, b, c) { code }',
'""'
]));
});
});
});
describe("for optional", function() {
var grammar = 'start = "a"?';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
15, 1, 2, 2, 19, 1, 20, 2, // <expression>
11, 3, 0, // IF_ERROR
2, // * POP
0, 0 // PUSH
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
Error handling: Make |?| return |null| on unsuccessful match Before this commit, the |?| operator returned an empty string upon unsuccessful match. This commit changes the returned value to |null|. It also updates the PEG.js grammar and the example grammars, which used the value returned by |?| quite often. Returning |null| is possible because it no longer indicates a match failure. I expect that this change will simplify many real-world grammars, as an empty string is almost never desirable as a return value (except some lexer-level rules) and it is often translated into |null| or some other value in action code. Implements part of #198.
11 years ago
'null',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("for zero or more", function() {
var grammar = 'start = "a"*';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
0, 0, // PUSH
15, 1, 2, 2, 19, 1, 20, 2, // <expression>
13, 9, // WHILE_NOT_ERROR
7, // * APPEND
15, 1, 2, 2, 19, 1, 20, 2, // <expression>
2 // POP
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
'[]',
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("for one or more", function() {
var grammar = 'start = "a"+';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
0, 0, // PUSH
15, 2, 2, 2, 19, 2, 20, 3, // <expression>
12, 12, 4, // IF_NOT_ERROR
13, 9, // * WHILE_NOT_ERROR
7, // * APPEND
15, 2, 2, 2, 19, 2, 20, 3, // <expression>
2, // POP
2, // * POP
2, // POP
0, 1 // PUSH
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
'[]',
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'peg$FAILED',
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("for rule reference", function() {
it("generates correct bytecode", function() {
expect(pass).toChangeAST([
'start = other',
'other = "other"'
].join("\n"), {
rules: [
{
bytecode: [24, 1] // RULE
},
{ }
]
});
});
});
describe("for literal", function() {
describe("empty", function() {
var grammar = 'start = ""';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
0, 0 // PUSH
]));
});
it("defines correct constants", function() {
expect(pass).toChangeAST(grammar, constsDetails(['""']));
});
});
describe("non-empty case-sensitive", function() {
var grammar = 'start = "a"';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
15, 0, 2, 2, // MATCH_STRING
19, 0, // * ACCEPT_STRING
20, 1 // * FAIL
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
'"a"',
'{ type: "literal", value: "a", description: "\\"a\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("non-empty case-insensitive", function() {
var grammar = 'start = "A"i';
it("generates correct bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
16, 0, 2, 2, // MATCH_STRING_IC
18, 1, // * ACCEPT_N
20, 1 // * FAIL
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(grammar, constsDetails([
'"a"',
'{ type: "literal", value: "A", description: "\\"A\\"" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
});
describe("for class", function() {
it("generates correct bytecode", function() {
expect(pass).toChangeAST('start = [a]', bytecodeDetails([
17, 0, 2, 2, // MATCH_REGEXP
18, 1, // * ACCEPT_N
20, 1 // * FAIL
]));
});
describe("non-empty non-inverted case-sensitive", function() {
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST('start = [a]', constsDetails([
'/^[a]/',
'{ type: "class", value: "[a]", description: "[a]" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("non-empty inverted case-sensitive", function() {
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST('start = [^a]', constsDetails([
'/^[^a]/',
'{ type: "class", value: "[^a]", description: "[^a]" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("non-empty non-inverted case-insensitive", function() {
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST('start = [a]i', constsDetails([
'/^[a]/i',
'{ type: "class", value: "[a]i", description: "[a]i" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("non-empty complex", function() {
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST('start = [ab-def-hij-l]', constsDetails([
'/^[ab-def-hij-l]/',
'{ type: "class", value: "[ab-def-hij-l]", description: "[ab-def-hij-l]" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("empty non-inverted", function() {
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST('start = []', constsDetails([
'/^(?!)/',
'{ type: "class", value: "[]", description: "[]" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
describe("empty inverted", function() {
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST('start = [^]', constsDetails([
'/^[\\S\\s]/',
'{ type: "class", value: "[^]", description: "[^]" }'
]));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
});
describe("for any", function() {
var grammar = 'start = .';
it("generates bytecode", function() {
expect(pass).toChangeAST(grammar, bytecodeDetails([
14, 2, 2, // MATCH_ANY
18, 1, // * ACCEPT_N
20, 0 // * FAIL
]));
});
it("defines correct constants", function() {
Error handling: Structured expectations Before this commit, the |expected| property of an exception object thrown when a generated parser encountered an error contained expectations as strings. These strings were in a human-readable format suitable for displaying in the UI but not suitable for machine processing. For example, expected string literals included quotes and a string "any character" was used when any character was expected. This commit makes expectations structured objects. This makes the machine processing easier, while still allowing to generate a human-readable representation if needed. Implements part of #198. Speed impact ------------ Before: 1180.41 kB/s After: 1165.31 kB/s Difference: -1.28% Size impact ----------- Before: 863523 b After: 950817 b Difference: 10.10% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
expect(pass).toChangeAST(
grammar,
constsDetails(['{ type: "any", description: "any character" }'])
);
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
});
});
});