Before this commit, whitespace was handled at the lexical level by
making tokens consume any whitespace coming after them. This was
accomplished by appending |__| to every token rule.
This commit changes whitespace handling to be more explicit. Tokens no
longer consume whitespace coming after them and syntactic rules have to
cope with it. While this slightly complicates the syntactic grammar, I
think it's a cleaner way. Moreover, it is what JavaScript example
grammar does.
One small side-effect of thich change is that the grammar is now
stand-alone (it doesn't require utils.js anymore).
When rule names are capitalized, it's easier to visually distinguish
them from non-capitalized label names. Moreover, I use capitalized rule
names in all my grammars these days.
Before this commit, a line continuation (backslash followed by a line
terminator character) contributed a character to a string or a character
class it was used in. In JavaScript and many other languages, line
continuation doesn't contribute anything.
This commit aligns PEG.js line continuation behavior with JavaScript.
Before this commit, the value of the |rawText| property of "class" AST
nodes was created in a hackish way from processed input and it didn't
always exactly represent the actual input text.
This commit changes the code so that the value of the |rawText| property
is created using the |text| function. This is a clean way which also
resolves the exact representation problem.
This is a complete rewrite of the CSS example grammar. It is now based
on CSS 2.1 *including the errata* and the generated parser builds a
nicer syntax tree. There is also a number of cleanups, formatting
changes, naming changes, and bug fixes.
Beside this, the rewrite reflects how I write grammars today (as opposed
to few years ago) and what style I would recommend to others.
This is a complete rewrite of the JavaScript example grammar. It is now
based on ECMA-262, 5.1 Edition and the generated parser builds a syntax
tree compatible with Mozilla SpiderMonkey Parser API. There is also a
number of cleanups, formatting changes, naming changes, and bug fixes.
Beside this, the rewrite reflects how I write grammars today (as opposed
to few years ago) and what style I would recommend to others.
This is a complete rewrite of the JSON example grammar. It is now based
on RFC 7159 instead of an informal description at the JSON website.
Beside this, the rewrite reflects how I write grammars today (as opposed
to few years ago) and what style I would recommend to others.
This is a complete rewrite of the arithmetics example grammar. It now
allows whitespace between tokens, supports "-" and "/" operators, and
gets the operator associativity right. Also, rule names now match the usual
conventions (term, factor,...).
Beside this, the rewrite reflects how I write grammars today (as opposed
to few years ago) and what style I would recommend to others.
Also added few missing |hasOwnProperty| calls that JSHint didn't detect
because it only looks whether there is an |if| statement wrapping the
loop body.
Fixes the following JSHint errors:
bin/pegjs: line 66, col 14, 'extraOptions' used out of scope.
bin/pegjs: line 70, col 19, 'extraOptions' used out of scope.
bin/pegjs: line 71, col 20, 'extraOptions' used out of scope.
bin/pegjs: line 80, col 10, Wrap the /regexp/ literal in parens to disambiguate the slash operator.
bin/pegjs: line 128, col 43, Missing semicolon.
bin/pegjs: line 128, col 45, Don't make functions within a loop.
bin/pegjs: line 150, col 13, Redefinition of 'module'.
bin/pegjs: line 217, col 34, Expected '===' and instead saw '=='.
bin/pegjs: line 243, col 44, 'source' used out of scope.
bin/pegjs: line 243, col 61, 'source' used out of scope.
Fixes the following JSHint error:
lib/compiler/passes/generate-bytecode.js: line 334, col 71, Expected an assignment or function call and instead saw an expression.
The one-parameter |Array.prototype.splice| call is a SpiderMonkey
extension. Apparently, IE doesn't implement it (unlike other supported
browsers), so we need to replace it with two-parameter version.
In case the generated parser parsed successfully part of input and left
some input unparsed (trailing input), the error message produced was
sometimes wrong. The code worked correctly only if there were no match
failures in the successfully parsed part (highly unlikely).
This commit fixes things by explicitly triggering a match failure with the
following expectation at the end of the successfully parsed part of the
input:
peg$fail({ type: "end", description: "end of input" });
This change also made it possible to simplify the |buildMessage|
function, which can now ignore the case of no expectations.
Fixes#119.
There are two invariants in generated bytecode related to the stack:
1. Branches of a condition must move the stack pointer in the same way.
2. Body of a loop can't move the stack pointer.
These invariants were always true, but they were not checked. Now we
check them at least when compiling with optimization for speed, because
there we analyze the stack pointer movements statically.
It's not necessary to parse |parts| in the |integer| and |float| rule
into integer/float value. Everywhere these rules are used the result is
converted back into string anyway.
The error check was useful when actions could have returned |null| to
trigger a match failure. This is no longer supported so the check isn't
needed anymore.
Speed impact
------------
Before: 1022.70 kB/s
After: 1035.45 kB/s
Difference: 1.24%
Size impact
-----------
Before: 975434 b
After: 931540 b
Difference: -4.50%
(Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
Before this commit, the |expected| and |error| functions didn't halt the
parsing immediately, but triggered a regular match failure. After they
were called, the parser could backtrack, try another branches, and only
if no other branch succeeded, it triggered an exception with information
possibly based on parameters passed to the |expected| or |error|
function (this depended on positions where failures in other branches
have occurred).
While nice in theory, this solution didn't work well in practice. There
were at least two problems:
1. Action expression could have easily triggered a match failure later
in the input than the action itself. This resulted in the
action-triggered failure to be shadowed by the expression-triggered
one.
Consider the following example:
integer = digits:[0-9]+ {
var result = parseInt(digits.join(""), 10);
if (result % 2 === 0) {
error("The number must be an odd integer.");
return;
}
return result;
}
Given input "2", the |[0-9]+| expression would record a match
failure at position 1 (an unsuccessful attempt to parse yet another
digit after "2"). However, a failure triggered by the |error| call
would occur at position 0.
This problem could have been solved by silencing match failures in
action expressions, but that would lead to severe performance
problems (yes, I tried and measured). Other possible solutions are
hacks which I didn't want to introduce into PEG.js.
2. Triggering a match failure in action code could have lead to
unexpected backtracking.
Consider the following example:
class = "[" (charRange / char)* "]"
charRange = begin:char "-" end:char {
if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) {
error("Invalid character range: " + begin + "-" + end + ".");
}
// ...
}
char = [a-zA-Z0-9_\-]
Given input "[b-a]", the |charRange| rule would fail, but the
parser would try the |char| rule and succeed repeatedly, resulting
in "b-a" being parsed as a sequence of three |char|'s, which it is
not.
This problem could have been solved by using negative predicates,
but that would complicate the grammar and still wouldn't get rid of
unintuitive behavior.
Given these problems I decided to change the semantics of the |expected|
and |error| functions. They don't interact with regular match failure
mechanism anymore, but they cause and immediate parse failure by
throwing an exception. I think this is more intuitive behavior with less
harmful side effects.
The disadvantage of the new approach is that one can't backtrack from an
action-triggered error. I don't see this as a big deal as I think this
will be rarely needed and one can always use a semantic predicate as a
workaround.
Speed impact
------------
Before: 993.84 kB/s
After: 998.05 kB/s
Difference: 0.42%
Size impact
-----------
Before: 1019968 b
After: 975434 b
Difference: -4.37%
(Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
David Majda
David Majda