joepie91
/
pegjs
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
You cannot select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
redux
master
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from '2f2152204a'
${ noResults }
pegjs/lib/compiler/passes/generate-javascript.js

992 lines
31 KiB
JavaScript
Raw Normal View History Unescape Escape

Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
var utils = require("../../utils"),
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
op = require("../opcodes");
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
/* Generates parser JavaScript code. */
module.exports = function(ast, options) {
/* These only indent non-empty lines to avoid trailing whitespace. */
function indent2(code) { return code.replace(/^(.+)$/gm, ' $1'); }
function indent4(code) { return code.replace(/^(.+)$/gm, ' $1'); }
function indent8(code) { return code.replace(/^(.+)$/gm, ' $1'); }
function indent10(code) { return code.replace(/^(.+)$/gm, ' $1'); }
function generateTables() {
if (options.optimize === "size") {
return [
'peg$consts = [',
indent2(ast.consts.join(',\n')),
'],',
'',
'peg$bytecode = [',
indent2(utils.map(
ast.rules,
function(rule) {
return 'peg$decode('
+ utils.quote(utils.map(
rule.bytecode,
function(b) { return String.fromCharCode(b + 32); }
).join(''))
+ ')';
}
).join(',\n')),
'],'
].join('\n');
} else {
return utils.map(
ast.consts,
function(c, i) { return 'peg$c' + i + ' = ' + c + ','; }
).join('\n');
}
}
function generateCacheHeader(ruleIndexCode) {
return [
'var key = peg$currPos * ' + ast.rules.length + ' + ' + ruleIndexCode + ',',
' cached = peg$cache[key];',
'',
'if (cached) {',
' peg$currPos = cached.nextPos;',
' return cached.result;',
'}',
''
].join('\n');
}
function generateCacheFooter(resultCode) {
return [
'',
'peg$cache[key] = { nextPos: peg$currPos, result: ' + resultCode + ' };'
].join('\n');
}
function generateInterpreter() {
var parts = [];
function generateCondition(cond, argsLength) {
var baseLength = argsLength + 3,
thenLengthCode = 'bc[ip + ' + (baseLength - 2) + ']',
elseLengthCode = 'bc[ip + ' + (baseLength - 1) + ']';
return [
'ends.push(end);',
'ips.push(ip + ' + baseLength + ' + ' + thenLengthCode + ' + ' + elseLengthCode + ');',
'',
'if (' + cond + ') {',
' end = ip + ' + baseLength + ' + ' + thenLengthCode + ';',
' ip += ' + baseLength + ';',
'} else {',
' end = ip + ' + baseLength + ' + ' + thenLengthCode + ' + ' + elseLengthCode + ';',
' ip += ' + baseLength + ' + ' + thenLengthCode + ';',
'}',
'',
'break;'
].join('\n');
}
function generateLoop(cond) {
var baseLength = 2,
bodyLengthCode = 'bc[ip + ' + (baseLength - 1) + ']';
return [
'if (' + cond + ') {',
' ends.push(end);',
' ips.push(ip);',
'',
' end = ip + ' + baseLength + ' + ' + bodyLengthCode + ';',
' ip += ' + baseLength + ';',
'} else {',
' ip += ' + baseLength + ' + ' + bodyLengthCode + ';',
'}',
'',
'break;'
].join('\n');
}
function generateCall() {
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
var baseLength = 4,
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
paramsLengthCode = 'bc[ip + ' + (baseLength - 1) + ']';
return [
'params = bc.slice(ip + ' + baseLength + ', ip + ' + baseLength + ' + ' + paramsLengthCode + ');',
'for (i = 0; i < ' + paramsLengthCode + '; i++) {',
' params[i] = stack[stack.length - 1 - params[i]];',
'}',
'',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'stack.splice(',
' stack.length - bc[ip + 2],',
' bc[ip + 2],',
' peg$consts[bc[ip + 1]].apply(null, params)',
');',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'',
'ip += ' + baseLength + ' + ' + paramsLengthCode + ';',
'break;'
].join('\n');
}
parts.push([
'function peg$decode(s) {',
' var bc = new Array(s.length), i;',
'',
' for (i = 0; i < s.length; i++) {',
' bc[i] = s.charCodeAt(i) - 32;',
' }',
'',
' return bc;',
'}',
'',
'function peg$parseRule(index) {',
' var bc = peg$bytecode[index],',
' ip = 0,',
' ips = [],',
' end = bc.length,',
' ends = [],',
' stack = [],',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' params, i;',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
''
].join('\n'));
if (options.cache) {
parts.push(indent2(generateCacheHeader('index')));
}
parts.push([
' function protect(object) {',
' return Object.prototype.toString.apply(object) === "[object Array]" ? [] : object;',
' }',
'',
/*
* The point of the outer loop and the |ips| & |ends| stacks is to avoid
* recursive calls for interpreting parts of bytecode. In other words, we
* implement the |interpret| operation of the abstract machine without
* function calls. Such calls would likely slow the parser down and more
* importantly cause stack overflows for complex grammars.
*/
' while (true) {',
' while (ip < end) {',
' switch (bc[ip]) {',
' case ' + op.PUSH + ':', // PUSH c
/*
* Hack: One of the constants can be an empty array. It needs to be cloned
* because it can be modified later on the stack by |APPEND|.
*/
' stack.push(protect(peg$consts[bc[ip + 1]]));',
' ip += 2;',
' break;',
'',
' case ' + op.PUSH_CURR_POS + ':', // PUSH_CURR_POS
' stack.push(peg$currPos);',
' ip++;',
' break;',
'',
' case ' + op.POP + ':', // POP
' stack.pop();',
' ip++;',
' break;',
'',
' case ' + op.POP_CURR_POS + ':', // POP_CURR_POS
' peg$currPos = stack.pop();',
' ip++;',
' break;',
'',
' case ' + op.POP_N + ':', // POP_N n
' stack.length -= bc[ip + 1];',
' ip += 2;',
' break;',
'',
' case ' + op.NIP + ':', // NIP
' stack.splice(-2, 1);',
' ip++;',
' break;',
'',
' case ' + op.NIP_CURR_POS + ':', // NIP_CURR_POS
' peg$currPos = stack.splice(-2, 1)[0];',
' ip++;',
' break;',
'',
' case ' + op.APPEND + ':', // APPEND
' stack[stack.length - 2].push(stack.pop());',
' ip++;',
' break;',
'',
' case ' + op.WRAP + ':', // WRAP n
' stack.push(stack.splice(stack.length - bc[ip + 1]));',
' ip += 2;',
' break;',
'',
' case ' + op.TEXT + ':', // TEXT
' stack.pop();',
' stack.push(input.substring(stack[stack.length - 1], peg$currPos));',
' ip++;',
' break;',
'',
' case ' + op.IF + ':', // IF t, f
indent10(generateCondition('stack[stack.length - 1]', 0)),
'',
' case ' + op.IF_ERROR + ':', // IF_ERROR t, f
indent10(generateCondition(
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'stack[stack.length - 1] === peg$FAILED',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
0
)),
'',
' case ' + op.IF_NOT_ERROR + ':', // IF_NOT_ERROR t, f
indent10(
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
generateCondition('stack[stack.length - 1] !== peg$FAILED',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
0
)),
'',
' case ' + op.WHILE_NOT_ERROR + ':', // WHILE_NOT_ERROR b
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
indent10(generateLoop('stack[stack.length - 1] !== peg$FAILED')),
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'',
' case ' + op.MATCH_ANY + ':', // MATCH_ANY a, f, ...
indent10(generateCondition('input.length > peg$currPos', 0)),
'',
' case ' + op.MATCH_STRING + ':', // MATCH_STRING s, a, f, ...
indent10(generateCondition(
'input.substr(peg$currPos, peg$consts[bc[ip + 1]].length) === peg$consts[bc[ip + 1]]',
1
)),
'',
' case ' + op.MATCH_STRING_IC + ':', // MATCH_STRING_IC s, a, f, ...
indent10(generateCondition(
'input.substr(peg$currPos, peg$consts[bc[ip + 1]].length).toLowerCase() === peg$consts[bc[ip + 1]]',
1
)),
'',
' case ' + op.MATCH_REGEXP + ':', // MATCH_REGEXP r, a, f, ...
indent10(generateCondition(
'peg$consts[bc[ip + 1]].test(input.charAt(peg$currPos))',
1
)),
'',
' case ' + op.ACCEPT_N + ':', // ACCEPT_N n
' stack.push(input.substr(peg$currPos, bc[ip + 1]));',
' peg$currPos += bc[ip + 1];',
' ip += 2;',
' break;',
'',
' case ' + op.ACCEPT_STRING + ':', // ACCEPT_STRING s
' stack.push(peg$consts[bc[ip + 1]]);',
' peg$currPos += peg$consts[bc[ip + 1]].length;',
' ip += 2;',
' break;',
'',
' case ' + op.FAIL + ':', // FAIL e
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' stack.push(peg$FAILED);',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' if (peg$silentFails === 0) {',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' peg$fail(peg$consts[bc[ip + 1]]);',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' }',
' ip += 2;',
' break;',
'',
' case ' + op.REPORT_SAVED_POS + ':', // REPORT_SAVED_POS p
' peg$reportedPos = stack[stack.length - 1 - bc[ip + 1]];',
' ip += 2;',
' break;',
'',
' case ' + op.REPORT_CURR_POS + ':', // REPORT_CURR_POS
' peg$reportedPos = peg$currPos;',
' ip++;',
' break;',
'',
' case ' + op.CALL + ':', // CALL f, n, pc, p1, p2, ..., pN
indent10(generateCall()),
'',
' case ' + op.RULE + ':', // RULE r
' stack.push(peg$parseRule(bc[ip + 1]));',
' ip += 2;',
' break;',
'',
' case ' + op.SILENT_FAILS_ON + ':', // SILENT_FAILS_ON
' peg$silentFails++;',
' ip++;',
' break;',
'',
' case ' + op.SILENT_FAILS_OFF + ':', // SILENT_FAILS_OFF
' peg$silentFails--;',
' ip++;',
' break;',
'',
' default:',
' throw new Error("Invalid opcode: " + bc[ip] + ".");',
' }',
' }',
'',
' if (ends.length > 0) {',
' end = ends.pop();',
' ip = ips.pop();',
' } else {',
' break;',
' }',
' }'
].join('\n'));
if (options.cache) {
parts.push(indent2(generateCacheFooter('stack[0]')));
}
parts.push([
'',
' return stack[0];',
'}'
].join('\n'));
return parts.join('\n');
}
function generateRuleFunction(rule) {
var parts = [], code;
function c(i) { return "peg$c" + i; } // |consts[i]| of the abstract machine
function s(i) { return "s" + i; } // |stack[i]| of the abstract machine
var stack = {
sp: -1,
maxSp: -1,
push: function(exprCode) {
var code = s(++this.sp) + ' = ' + exprCode + ';';
if (this.sp > this.maxSp) { this.maxSp = this.sp; }
return code;
},
pop: function() {
var n, values;
if (arguments.length === 0) {
return s(this.sp--);
} else {
n = arguments[0];
values = utils.map(utils.range(this.sp - n + 1, this.sp + 1), s);
this.sp -= n;
return values;
}
},
top: function() {
return s(this.sp);
},
index: function(i) {
return s(this.sp - i);
}
};
function compile(bc) {
var ip = 0,
end = bc.length,
parts = [],
value;
function compileCondition(cond, argCount) {
var baseLength = argCount + 3,
thenLength = bc[ip + baseLength - 2],
elseLength = bc[ip + baseLength - 1],
baseSp = stack.sp,
thenCode, elseCode;
ip += baseLength;
thenCode = compile(bc.slice(ip, ip + thenLength));
ip += thenLength;
if (elseLength > 0) {
stack.sp = baseSp;
elseCode = compile(bc.slice(ip, ip + elseLength));
ip += elseLength;
}
parts.push('if (' + cond + ') {');
parts.push(indent2(thenCode));
if (elseLength > 0) {
parts.push('} else {');
parts.push(indent2(elseCode));
}
parts.push('}');
}
function compileLoop(cond) {
var baseLength = 2,
bodyLength = bc[ip + baseLength - 1],
bodyCode;
ip += baseLength;
bodyCode = compile(bc.slice(ip, ip + bodyLength));
ip += bodyLength;
parts.push('while (' + cond + ') {');
parts.push(indent2(bodyCode));
parts.push('}');
}
function compileCall(cond) {
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
var baseLength = 4,
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
paramsLength = bc[ip + baseLength - 1];
var value = c(bc[ip + 1]) + '('
+ utils.map(
bc.slice(ip + baseLength, ip + baseLength + paramsLength),
stackIndex
Add whitespace to generated action calls Avoids implicit array to string conversion.
11 years ago
).join(', ')
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
+ ')';
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
stack.pop(bc[ip + 2]);
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
parts.push(stack.push(value));
ip += baseLength + paramsLength;
}
/*
* Extracted into a function just to silence JSHint complaining about
* creating functions in a loop.
*/
function stackIndex(p) {
return stack.index(p);
}
while (ip < end) {
switch (bc[ip]) {
case op.PUSH: // PUSH c
/*
* Hack: One of the constants can be an empty array. It needs to be
* handled specially because it can be modified later on the stack
* by |APPEND|.
*/
parts.push(
stack.push(ast.consts[bc[ip + 1]] === "[]" ? "[]" : c(bc[ip + 1]))
);
ip += 2;
break;
case op.PUSH_CURR_POS: // PUSH_CURR_POS
parts.push(stack.push('peg$currPos'));
ip++;
break;
case op.POP: // POP
stack.pop();
ip++;
break;
case op.POP_CURR_POS: // POP_CURR_POS
parts.push('peg$currPos = ' + stack.pop() + ';');
ip++;
break;
case op.POP_N: // POP_N n
stack.pop(bc[ip + 1]);
ip += 2;
break;
case op.NIP: // NIP
value = stack.pop();
stack.pop();
parts.push(stack.push(value));
ip++;
break;
case op.NIP_CURR_POS: // NIP_CURR_POS
value = stack.pop();
parts.push('peg$currPos = ' + stack.pop() + ';');
parts.push(stack.push(value));
ip++;
break;
case op.APPEND: // APPEND
value = stack.pop();
parts.push(stack.top() + '.push(' + value + ');');
ip++;
break;
case op.WRAP: // WRAP n
parts.push(
stack.push('[' + stack.pop(bc[ip + 1]).join(', ') + ']')
);
ip += 2;
break;
case op.TEXT: // TEXT
stack.pop();
parts.push(
stack.push('input.substring(' + stack.top() + ', peg$currPos)')
);
ip++;
break;
case op.IF: // IF t, f
compileCondition(stack.top(), 0);
break;
case op.IF_ERROR: // IF_ERROR t, f
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
compileCondition(stack.top() + ' === peg$FAILED', 0);
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
break;
case op.IF_NOT_ERROR: // IF_NOT_ERROR t, f
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
compileCondition(stack.top() + ' !== peg$FAILED', 0);
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
break;
case op.WHILE_NOT_ERROR: // WHILE_NOT_ERROR b
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
compileLoop(stack.top() + ' !== peg$FAILED', 0);
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
break;
case op.MATCH_ANY: // MATCH_ANY a, f, ...
compileCondition('input.length > peg$currPos', 0);
break;
case op.MATCH_STRING: // MATCH_STRING s, a, f, ...
compileCondition(
eval(ast.consts[bc[ip + 1]]).length > 1
? 'input.substr(peg$currPos, '
+ eval(ast.consts[bc[ip + 1]]).length
+ ') === '
+ c(bc[ip + 1])
: 'input.charCodeAt(peg$currPos) === '
+ eval(ast.consts[bc[ip + 1]]).charCodeAt(0),
1
);
break;
case op.MATCH_STRING_IC: // MATCH_STRING_IC s, a, f, ...
compileCondition(
'input.substr(peg$currPos, '
Fix matching of case-instensitive literals Code that calculated which part of the input to match against a literal was wrong in case of case-insensitive literals when generating speed-optimized parsers. As a result, matching of case-insensitive literals worked only at the end of the input (where too big length passed to the |substr| method didn't matter). Fixes GH-153.
11 years ago
+ eval(ast.consts[bc[ip + 1]]).length
Fix indentation
11 years ago
+ ').toLowerCase() === '
+ c(bc[ip + 1]),
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
1
);
break;
case op.MATCH_REGEXP: // MATCH_REGEXP r, a, f, ...
compileCondition(
c(bc[ip + 1]) + '.test(input.charAt(peg$currPos))',
1
);
break;
case op.ACCEPT_N: // ACCEPT_N n
parts.push(stack.push(
bc[ip + 1] > 1
? 'input.substr(peg$currPos, ' + bc[ip + 1] + ')'
: 'input.charAt(peg$currPos)'
));
parts.push(
bc[ip + 1] > 1
? 'peg$currPos += ' + bc[ip + 1] + ';'
: 'peg$currPos++;'
);
ip += 2;
break;
case op.ACCEPT_STRING: // ACCEPT_STRING s
parts.push(stack.push(c(bc[ip + 1])));
parts.push(
eval(ast.consts[bc[ip + 1]]).length > 1
? 'peg$currPos += ' + eval(ast.consts[bc[ip + 1]]).length + ';'
: 'peg$currPos++;'
);
ip += 2;
break;
case op.FAIL: // FAIL e
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
parts.push(stack.push('peg$FAILED'));
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
parts.push('if (peg$silentFails === 0) { peg$fail(' + c(bc[ip + 1]) + '); }');
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
ip += 2;
break;
case op.REPORT_SAVED_POS: // REPORT_SAVED_POS p
parts.push('peg$reportedPos = ' + stack.index(bc[ip + 1]) + ';');
ip += 2;
break;
case op.REPORT_CURR_POS: // REPORT_CURR_POS
parts.push('peg$reportedPos = peg$currPos;');
ip++;
break;
case op.CALL: // CALL f, n, pc, p1, p2, ..., pN
compileCall();
break;
case op.RULE: // RULE r
parts.push(stack.push("peg$parse" + ast.rules[bc[ip + 1]].name + "()"));
ip += 2;
break;
case op.SILENT_FAILS_ON: // SILENT_FAILS_ON
parts.push('peg$silentFails++;');
ip++;
break;
case op.SILENT_FAILS_OFF: // SILENT_FAILS_OFF
parts.push('peg$silentFails--;');
ip++;
break;
default:
throw new Error("Invalid opcode: " + bc[ip] + ".");
}
}
return parts.join('\n');
}
code = compile(rule.bytecode);
parts.push([
'function peg$parse' + rule.name + '() {',
' var ' + utils.map(utils.range(0, stack.maxSp + 1), s).join(', ') + ';',
''
].join('\n'));
if (options.cache) {
parts.push(indent2(
generateCacheHeader(utils.indexOfRuleByName(ast, rule.name))
));
}
parts.push(indent2(code));
if (options.cache) {
Use the |s| function instead of hardcoded |s0| value Based on a patch by @fresheneesz: https://github.com/dmajda/pegjs/pull/148
11 years ago
parts.push(indent2(generateCacheFooter(s(0))));
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
}
parts.push([
'',
Use the |s| function instead of hardcoded |s0| value Based on a patch by @fresheneesz: https://github.com/dmajda/pegjs/pull/148
11 years ago
' return ' + s(0) + ';',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'}'
].join('\n'));
return parts.join('\n');
}
var parts = [],
startRuleIndices, startRuleIndex,
startRuleFunctions, startRuleFunction;
parts.push([
'(function() {',
' /*',
' * Generated by PEG.js 0.7.0.',
' *',
' * http://pegjs.majda.cz/',
' */',
'',
s/subclass/peg$subclass/ The |subclass| function is not intended to be used by user code.
11 years ago
' function peg$subclass(child, parent) {',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' function ctor() { this.constructor = child; }',
' ctor.prototype = parent.prototype;',
' child.prototype = new ctor();',
' }',
'',
Error handling: Build error message out of |SyntaxError|'s constructor It will be possible to create errors with user-supplied messages soon. The |SyntaxError| class needs to be ready for that. Implements part of #198.
11 years ago
' function SyntaxError(message, expected, found, offset, line, column) {',
' this.message = message;',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' this.expected = expected;',
' this.found = found;',
' this.offset = offset;',
' this.line = line;',
' this.column = column;',
'',
' this.name = "SyntaxError";',
' }',
'',
s/subclass/peg$subclass/ The |subclass| function is not intended to be used by user code.
11 years ago
' peg$subclass(SyntaxError, Error);',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'',
' function parse(input) {',
' var options = arguments.length > 1 ? arguments[1] : {},',
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'',
' peg$FAILED = {},',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
''
].join('\n'));
if (options.optimize === "size") {
startRuleIndices = '{ '
+ utils.map(
options.allowedStartRules,
function(r) { return r + ': ' + utils.indexOfRuleByName(ast, r); }
).join(', ')
+ ' }';
startRuleIndex = utils.indexOfRuleByName(ast, options.allowedStartRules[0]);
parts.push([
' peg$startRuleIndices = ' + startRuleIndices + ',',
' peg$startRuleIndex = ' + startRuleIndex + ','
].join('\n'));
} else {
startRuleFunctions = '{ '
+ utils.map(
options.allowedStartRules,
function(r) { return r + ': peg$parse' + r; }
).join(', ')
+ ' }';
startRuleFunction = 'peg$parse' + options.allowedStartRules[0];
parts.push([
' peg$startRuleFunctions = ' + startRuleFunctions + ',',
' peg$startRuleFunction = ' + startRuleFunction + ','
].join('\n'));
}
parts.push('');
parts.push(indent8(generateTables()));
parts.push([
'',
' peg$currPos = 0,',
' peg$reportedPos = 0,',
' peg$cachedPos = 0,',
' peg$cachedPosDetails = { line: 1, column: 1, seenCR: false },',
' peg$maxFailPos = 0,',
' peg$maxFailExpected = [],',
' peg$silentFails = 0,', // 0 = report failures, > 0 = silence failures
''
].join('\n'));
if (options.cache) {
parts.push(' peg$cache = {},');
}
parts.push([
' peg$result;',
''
].join('\n'));
if (options.optimize === "size") {
parts.push([
' if ("startRule" in options) {',
' if (!(options.startRule in peg$startRuleIndices)) {',
' throw new Error("Can\'t start parsing from rule \\"" + options.startRule + "\\".");',
' }',
'',
' peg$startRuleIndex = peg$startRuleIndices[options.startRule];',
' }'
].join('\n'));
} else {
parts.push([
' if ("startRule" in options) {',
' if (!(options.startRule in peg$startRuleFunctions)) {',
' throw new Error("Can\'t start parsing from rule \\"" + options.startRule + "\\".");',
' }',
'',
' peg$startRuleFunction = peg$startRuleFunctions[options.startRule];',
' }'
].join('\n'));
}
parts.push([
'',
' function text() {',
' return input.substring(peg$reportedPos, peg$currPos);',
' }',
'',
' function offset() {',
' return peg$reportedPos;',
' }',
'',
' function line() {',
' return peg$computePosDetails(peg$reportedPos).line;',
' }',
'',
' function column() {',
' return peg$computePosDetails(peg$reportedPos).column;',
' }',
'',
Error handling: Implement the |expected| function The |expected| function allows users to report regular match failures inside actions. If the |expected| function is called, and the reported match failure turns out to be the cause of a parse error, the error message reported by the parser will be in the usual "Expected ... but found ..." format with the description specified in the |expected| call used as part of the message. Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' function expected(description) {',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' throw peg$buildException(',
' null,',
' [{ type: "other", description: description }],',
' peg$reportedPos',
' );',
Error handling: Implement the |expected| function The |expected| function allows users to report regular match failures inside actions. If the |expected| function is called, and the reported match failure turns out to be the cause of a parse error, the error message reported by the parser will be in the usual "Expected ... but found ..." format with the description specified in the |expected| call used as part of the message. Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' }',
'',
Error handling: Implement the |error| function The |error| function allows users to report custom match failures inside actions. If the |error| function is called, and the reported match failure turns out to be the cause of a parse error, the error message reported by the parser will be exactly the one specified in the |error| call. Implements part of #198. Speed impact ------------ Before: 999.83 kB/s After: 1000.84 kB/s Difference: 0.10% Size impact ----------- Before: 1017212 b After: 1019968 b Difference: 0.27% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' function error(message) {',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' throw peg$buildException(message, null, peg$reportedPos);',
Error handling: Implement the |error| function The |error| function allows users to report custom match failures inside actions. If the |error| function is called, and the reported match failure turns out to be the cause of a parse error, the error message reported by the parser will be exactly the one specified in the |error| call. Implements part of #198. Speed impact ------------ Before: 999.83 kB/s After: 1000.84 kB/s Difference: 0.10% Size impact ----------- Before: 1017212 b After: 1019968 b Difference: 0.27% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' }',
'',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' function peg$computePosDetails(pos) {',
Fix buggy position computation Fixes GH-152.
11 years ago
' function advance(details, startPos, endPos) {',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' var p, ch;',
'',
Fix buggy position computation Fixes GH-152.
11 years ago
' for (p = startPos; p < endPos; p++) {',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' ch = input.charAt(p);',
' if (ch === "\\n") {',
' if (!details.seenCR) { details.line++; }',
' details.column = 1;',
' details.seenCR = false;',
' } else if (ch === "\\r" || ch === "\\u2028" || ch === "\\u2029") {',
' details.line++;',
' details.column = 1;',
' details.seenCR = true;',
' } else {',
' details.column++;',
' details.seenCR = false;',
' }',
' }',
' }',
'',
' if (peg$cachedPos !== pos) {',
' if (peg$cachedPos > pos) {',
' peg$cachedPos = 0;',
' peg$cachedPosDetails = { line: 1, column: 1, seenCR: false };',
' }',
Fix buggy position computation Fixes GH-152.
11 years ago
' advance(peg$cachedPosDetails, peg$cachedPos, pos);',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' peg$cachedPos = pos;',
' }',
'',
' return peg$cachedPosDetails;',
' }',
'',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' function peg$fail(expected) {',
' if (peg$currPos < peg$maxFailPos) { return; }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' if (peg$currPos > peg$maxFailPos) {',
' peg$maxFailPos = peg$currPos;',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' peg$maxFailExpected = [];',
' }',
'',
' peg$maxFailExpected.push(expected);',
' }',
'',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' function peg$buildException(message, expected, pos) {',
Error handling: Extract exception building into its own function The exception-creating code will get somewhat hairy soon, so let's make sure them mess will be contained. Implements part of #198.
11 years ago
' function cleanupExpected(expected) {',
' var i = 1;',
'',
' expected.sort(function(a, b) {',
' if (a.description < b.description) {',
' return -1;',
' } else if (a.description > b.description) {',
' return 1;',
' } else {',
' return 0;',
' }',
' });',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
'',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
/*
* This works because the bytecode generator guarantees that every
* expectation object exists only once, so it's enough to use |===| instead
* of deeper structural comparison.
*/
Error handling: Extract exception building into its own function The exception-creating code will get somewhat hairy soon, so let's make sure them mess will be contained. Implements part of #198.
11 years ago
' while (i < expected.length) {',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' if (expected[i - 1] === expected[i]) {',
Error handling: Extract exception building into its own function The exception-creating code will get somewhat hairy soon, so let's make sure them mess will be contained. Implements part of #198.
11 years ago
' expected.splice(i, 1);',
' } else {',
' i++;',
' }',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' }',
' }',
Error handling: Extract exception building into its own function The exception-creating code will get somewhat hairy soon, so let's make sure them mess will be contained. Implements part of #198.
11 years ago
'',
Error handling: Build error message out of |SyntaxError|'s constructor It will be possible to create errors with user-supplied messages soon. The |SyntaxError| class needs to be ready for that. Implements part of #198.
11 years ago
' function buildMessage(expected, found) {',
' function stringEscape(s) {',
' function hex(ch) { return ch.charCodeAt(0).toString(16).toUpperCase(); }',
'',
/*
* ECMA-262, 5th ed., 7.8.4: All characters may appear literally in a string
* literal except for the closing quote character, backslash, carriage
* return, line separator, paragraph separator, and line feed. Any character
* may appear in the form of an escape sequence.
*
* For portability, we also escape all control and non-ASCII characters.
* Note that "\0" and "\v" escape sequences are not used because JSHint does
* not like the first and IE the second.
*/
' return s',
' .replace(/\\\\/g, \'\\\\\\\\\')', // backslash
' .replace(/"/g, \'\\\\"\')', // closing double quote
' .replace(/\\x08/g, \'\\\\b\')', // backspace
' .replace(/\\t/g, \'\\\\t\')', // horizontal tab
' .replace(/\\n/g, \'\\\\n\')', // line feed
' .replace(/\\f/g, \'\\\\f\')', // form feed
' .replace(/\\r/g, \'\\\\r\')', // carriage return
' .replace(/[\\x00-\\x07\\x0B\\x0E\\x0F]/g, function(ch) { return \'\\\\x0\' + hex(ch); })',
' .replace(/[\\x10-\\x1F\\x80-\\xFF]/g, function(ch) { return \'\\\\x\' + hex(ch); })',
' .replace(/[\\u0180-\\u0FFF]/g, function(ch) { return \'\\\\u0\' + hex(ch); })',
' .replace(/[\\u1080-\\uFFFF]/g, function(ch) { return \'\\\\u\' + hex(ch); });',
' }',
'',
' var expectedDescs, expectedDesc, foundDesc, i;',
'',
' switch (expected.length) {',
' case 0:',
' expectedDesc = "end of input";',
' break;',
'',
' case 1:',
' expectedDesc = expected[0].description;',
' break;',
'',
' default:',
' expectedDescs = new Array(expected.length);',
'',
' for (i = 0; i < expected.length; i++) {',
' expectedDescs[i] = expected[i].description;',
' }',
'',
' expectedDesc = expectedDescs.slice(0, -1).join(", ")',
' + " or "',
' + expectedDescs[expected.length - 1];',
' }',
'',
' foundDesc = found ? "\\"" + stringEscape(found) + "\\"" : "end of input";',
'',
' return "Expected " + expectedDesc + " but " + foundDesc + " found.";',
' }',
'',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' var posDetails = peg$computePosDetails(pos),',
' found = pos < input.length ? input.charAt(pos) : null;',
Error handling: Implement the |error| function The |error| function allows users to report custom match failures inside actions. If the |error| function is called, and the reported match failure turns out to be the cause of a parse error, the error message reported by the parser will be exactly the one specified in the |error| call. Implements part of #198. Speed impact ------------ Before: 999.83 kB/s After: 1000.84 kB/s Difference: 0.10% Size impact ----------- Before: 1017212 b After: 1019968 b Difference: 0.27% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
'',
' if (expected !== null) {',
' cleanupExpected(expected);',
' }',
Error handling: Extract exception building into its own function The exception-creating code will get somewhat hairy soon, so let's make sure them mess will be contained. Implements part of #198.
11 years ago
'',
' return new SyntaxError(',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' message !== null ? message : buildMessage(expected, found),',
Error handling: Implement the |error| function The |error| function allows users to report custom match failures inside actions. If the |error| function is called, and the reported match failure turns out to be the cause of a parse error, the error message reported by the parser will be exactly the one specified in the |error| call. Implements part of #198. Speed impact ------------ Before: 999.83 kB/s After: 1000.84 kB/s Difference: 0.10% Size impact ----------- Before: 1017212 b After: 1019968 b Difference: 0.27% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' expected,',
Error handling: Extract exception building into its own function The exception-creating code will get somewhat hairy soon, so let's make sure them mess will be contained. Implements part of #198.
11 years ago
' found,',
' pos,',
' posDetails.line,',
' posDetails.column',
' );',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' }',
''
].join('\n'));
if (options.optimize === "size") {
parts.push(indent4(generateInterpreter()));
parts.push('');
} else {
utils.each(ast.rules, function(rule) {
parts.push(indent4(generateRuleFunction(rule)));
parts.push('');
});
}
if (ast.initializer) {
parts.push(indent4(ast.initializer.code));
parts.push('');
}
if (options.optimize === "size") {
parts.push(' peg$result = peg$parseRule(peg$startRuleIndex);');
} else {
parts.push(' peg$result = peg$startRuleFunction();');
}
parts.push([
'',
Error handling: Use a special value (not |null|) to indicate failure Using a special value to indicate match failure instead of |null| allows actions to return |null| as a regular value. This simplifies e.g. the JSON parser. Note the special value is internal and intentionally undocumented. This means that there is currently no official way how to trigger a match failure from an action. This is a temporary state which will be fixed soon. The negative performance impact (see below) is probably caused by changing lot of comparisons against |null| (which likely check the value against a fixed constant representing |null| in the interpreter) to comparisons against the special value (which likely check the value against another value in the interpreter). Implements part of #198. Speed impact ------------ Before: 1146.82 kB/s After: 1031.25 kB/s Difference: -10.08% Size impact ----------- Before: 950817 b After: 973269 b Difference: 2.36% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' if (peg$result !== peg$FAILED && peg$currPos === input.length) {',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' return peg$result;',
' } else {',
Refine error handling further Before this commit, the |expected| and |error| functions didn't halt the parsing immediately, but triggered a regular match failure. After they were called, the parser could backtrack, try another branches, and only if no other branch succeeded, it triggered an exception with information possibly based on parameters passed to the |expected| or |error| function (this depended on positions where failures in other branches have occurred). While nice in theory, this solution didn't work well in practice. There were at least two problems: 1. Action expression could have easily triggered a match failure later in the input than the action itself. This resulted in the action-triggered failure to be shadowed by the expression-triggered one. Consider the following example: integer = digits:[0-9]+ { var result = parseInt(digits.join(""), 10); if (result % 2 === 0) { error("The number must be an odd integer."); return; } return result; } Given input "2", the |[0-9]+| expression would record a match failure at position 1 (an unsuccessful attempt to parse yet another digit after "2"). However, a failure triggered by the |error| call would occur at position 0. This problem could have been solved by silencing match failures in action expressions, but that would lead to severe performance problems (yes, I tried and measured). Other possible solutions are hacks which I didn't want to introduce into PEG.js. 2. Triggering a match failure in action code could have lead to unexpected backtracking. Consider the following example: class = "[" (charRange / char)* "]" charRange = begin:char "-" end:char { if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) { error("Invalid character range: " + begin + "-" + end + "."); } // ... } char = [a-zA-Z0-9_\-] Given input "[b-a]", the |charRange| rule would fail, but the parser would try the |char| rule and succeed repeatedly, resulting in "b-a" being parsed as a sequence of three |char|'s, which it is not. This problem could have been solved by using negative predicates, but that would complicate the grammar and still wouldn't get rid of unintuitive behavior. Given these problems I decided to change the semantics of the |expected| and |error| functions. They don't interact with regular match failure mechanism anymore, but they cause and immediate parse failure by throwing an exception. I think this is more intuitive behavior with less harmful side effects. The disadvantage of the new approach is that one can't backtrack from an action-triggered error. I don't see this as a big deal as I think this will be rarely needed and one can always use a semantic predicate as a workaround. Speed impact ------------ Before: 993.84 kB/s After: 998.05 kB/s Difference: 0.42% Size impact ----------- Before: 1019968 b After: 975434 b Difference: -4.37% (Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
11 years ago
' throw peg$buildException(',
' null,',
' peg$maxFailExpected,',
' Math.max(peg$currPos, peg$maxFailPos)',
' );',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' }',
' }',
'',
' return {',
' SyntaxError: SyntaxError,',
Fix object literal formatting in generated code
11 years ago
' parse: parse',
Code generator rewrite This is a complete rewrite of the PEG.js code generator. Its goals are: 1. Allow optimizing the generated parser code for code size as well as for parsing speed. 2. Prepare ground for future optimizations and big features (like incremental parsing). 2. Replace the old template-based code-generation system with something more lightweight and flexible. 4. General code cleanup (structure, style, variable names, ...). New Architecture ---------------- The new code generator consists of two steps: * Bytecode generator -- produces bytecode for an abstract virtual machine * JavaScript generator -- produces JavaScript code based on the bytecode The abstract virtual machine is stack-based. Originally I wanted to make it register-based, but it turned out that all the code related to it would be more complex and the bytecode itself would be longer (because of explicit register specifications in instructions). The only downsides of the stack-based approach seem to be few small inefficiencies (see e.g. the |NIP| instruction), which seem to be insignificant. The new generator allows optimizing for parsing speed or code size (you can choose using the |optimize| option of the |PEG.buildParser| method or the --optimize/-o option on the command-line). When optimizing for size, the JavaScript generator emits the bytecode together with its constant table and a generic bytecode interpreter. Because the interpreter is small and the bytecode and constant table grow only slowly with size of the grammar, the resulting parser is also small. When optimizing for speed, the JavaScript generator just compiles the bytecode into JavaScript. The generated code is relatively efficient, so the resulting parser is fast. Internal Identifiers -------------------- As a small bonus, all internal identifiers visible to user code in the initializer, actions and predicates are prefixed by |peg$|. This lowers the chance that identifiers in user code will conflict with the ones from PEG.js. It also makes using any internals in user code ugly, which is a good thing. This solves GH-92. Performance ----------- The new code generator improved parsing speed and parser code size significantly. The generated parsers are now: * 39% faster when optimizing for speed * 69% smaller when optimizing for size (without minification) * 31% smaller when optimizing for size (with minification) (Parsing speed was measured using the |benchmark/run| script. Code size was measured by generating parsers for examples in the |examples| directory and adding up the file sizes. Minification was done by |uglify --ascii| in version 1.3.4.) Final Note ---------- This is just a beginning! The new code generator lays a foundation upon which many optimizations and improvements can (and will) be made. Stay tuned :-)
12 years ago
' };',
'})()'
].join('\n'));
ast.code = parts.join('\n');
};