I no longer think that using raw literal texts in error messages is the
right thing to do. The main reason is that it couples error messages
with details of the grammar such as use of single or double quotes in
literals. A better solution is coming in the next commit.
This reverts commit 69a0f769fc.
Labels in expressions like "(a:"a")" or "(a:"a" b:"b" c:"c")" were
visible to the outside despite being wrapped in parens. This commit
makes them invisible, as they should be.
Note this required introduction of a new "group" AST node, whose purpose
is purely to provide label scope isolation. This was necessary because
"label" and "sequence" nodes don't (and can't!) provide this isolation
themselves.
Part of a fix of #396.
In the past year I worked on various grammars where first/rest or
head/tail were used as labels for parts of lists. I found I associate
head/tail with a list immediately, while in case of first/rest I have to
"parse" grammar rules for a while before understanding their structure.
Moreover, I tend to assume that rest is a list of the same thigs as
first, but I don't have such assumption in case of head/tail. This
assumption was in conflict with the grammar structure.
I'm not sure how much these observations are applicable to others, but I
decided to act on them and switch from first/rest to head/tail.
This is mostly done for consistency with the JavaScript example grammar,
from which the |Identifier| rule is taken from. See the previous commit
for details.
Instead of matching segments between blocks character by character,
match them as a whole. Also align the style with other similar rules
(e.g. the comment ones).
Before this commit, line continuations in character classes contributed
an empty string to the list of characters and character ranges matched
by a class. While this didn't lead to a buggy behavior with the current
code generator, the AST was wrong and the problem could have caused bugs
later.
This commit fixes the problem.
Semantic predicates are kind of |PrimaryExpression|, not kind of
|PrefixedExpression|. Therefore I extracted a rule for them and
referenced it from the |PrimaryExpression|.
Initializer and rules are now separated in a similar way as JavaScript
statements -- either by a semicolon or a line terminator, possibly with
whitespace and comments mixed in.
One consequence is that the grammars like this are now illegal:
foo = "a" bar = "b"
A semicolon needs to be inserted between the rules:
foo = "a";bar = "b"
I consider this a good change as the now-illegal syntax was somewhat
confusing.
This makes the |Primary| rule a bit more tidy. Also, matching the |.|
character really belongs to the lexical part of the grammar, next to
literals and character classes.
* Rename the |Action| rule to |CodeBlock| (it better describes what
the rule matches).
* Implement the rule in a simpler way and move it after more basic
lexical elements.
This change has two side effects:
* Label names can no longer be JavaScript reserved words.
* |$| is allowed again in label names. However, because of the
preference rules, names starting with it will be usually parsed as a
text operator followed by another identifier (denoting a rule
reference or label name).
Before this commit, whitespace was handled at the lexical level by
making tokens consume any whitespace coming after them. This was
accomplished by appending |__| to every token rule.
This commit changes whitespace handling to be more explicit. Tokens no
longer consume whitespace coming after them and syntactic rules have to
cope with it. While this slightly complicates the syntactic grammar, I
think it's a cleaner way. Moreover, it is what JavaScript example
grammar does.
One small side-effect of thich change is that the grammar is now
stand-alone (it doesn't require utils.js anymore).
When rule names are capitalized, it's easier to visually distinguish
them from non-capitalized label names. Moreover, I use capitalized rule
names in all my grammars these days.
Before this commit, a line continuation (backslash followed by a line
terminator character) contributed a character to a string or a character
class it was used in. In JavaScript and many other languages, line
continuation doesn't contribute anything.
This commit aligns PEG.js line continuation behavior with JavaScript.
Before this commit, the value of the |rawText| property of "class" AST
nodes was created in a hackish way from processed input and it didn't
always exactly represent the actual input text.
This commit changes the code so that the value of the |rawText| property
is created using the |text| function. This is a clean way which also
resolves the exact representation problem.
Before this commit, the |expected| and |error| functions didn't halt the
parsing immediately, but triggered a regular match failure. After they
were called, the parser could backtrack, try another branches, and only
if no other branch succeeded, it triggered an exception with information
possibly based on parameters passed to the |expected| or |error|
function (this depended on positions where failures in other branches
have occurred).
While nice in theory, this solution didn't work well in practice. There
were at least two problems:
1. Action expression could have easily triggered a match failure later
in the input than the action itself. This resulted in the
action-triggered failure to be shadowed by the expression-triggered
one.
Consider the following example:
integer = digits:[0-9]+ {
var result = parseInt(digits.join(""), 10);
if (result % 2 === 0) {
error("The number must be an odd integer.");
return;
}
return result;
}
Given input "2", the |[0-9]+| expression would record a match
failure at position 1 (an unsuccessful attempt to parse yet another
digit after "2"). However, a failure triggered by the |error| call
would occur at position 0.
This problem could have been solved by silencing match failures in
action expressions, but that would lead to severe performance
problems (yes, I tried and measured). Other possible solutions are
hacks which I didn't want to introduce into PEG.js.
2. Triggering a match failure in action code could have lead to
unexpected backtracking.
Consider the following example:
class = "[" (charRange / char)* "]"
charRange = begin:char "-" end:char {
if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) {
error("Invalid character range: " + begin + "-" + end + ".");
}
// ...
}
char = [a-zA-Z0-9_\-]
Given input "[b-a]", the |charRange| rule would fail, but the
parser would try the |char| rule and succeed repeatedly, resulting
in "b-a" being parsed as a sequence of three |char|'s, which it is
not.
This problem could have been solved by using negative predicates,
but that would complicate the grammar and still wouldn't get rid of
unintuitive behavior.
Given these problems I decided to change the semantics of the |expected|
and |error| functions. They don't interact with regular match failure
mechanism anymore, but they cause and immediate parse failure by
throwing an exception. I think this is more intuitive behavior with less
harmful side effects.
The disadvantage of the new approach is that one can't backtrack from an
action-triggered error. I don't see this as a big deal as I think this
will be rarely needed and one can always use a semantic predicate as a
workaround.
Speed impact
------------
Before: 993.84 kB/s
After: 998.05 kB/s
Difference: 0.42%
Size impact
-----------
Before: 1019968 b
After: 975434 b
Difference: -4.37%
(Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
Before this commit, the |?| operator returned an empty string upon
unsuccessful match. This commit changes the returned value to |null|. It
also updates the PEG.js grammar and the example grammars, which used the
value returned by |?| quite often.
Returning |null| is possible because it no longer indicates a match
failure.
I expect that this change will simplify many real-world grammars, as an
empty string is almost never desirable as a return value (except some
lexer-level rules) and it is often translated into |null| or some other
value in action code.
Implements part of #198.
Implement a new syntax to extract matched strings from expressions. For
example, instead of:
identifier = first:[a-zA-Z_] rest:[a-zA-Z0-9_]* { return first + rest.join(""); }
you can now just write:
identifier = $([a-zA-Z_] [a-zA-Z0-9_]*)
This is useful mostly for "lexical" rules at the bottom of many
grammars.
Note that structured match results are still built for the expressions
prefixed by "$", they are just ignored. I plan to optimize this later
(sometime after the code generator rewrite).
Includes:
* Moving the source code from /src to /lib.
* Adding an explicit file list to package.json
* Updating the Makefile.
* Updating the spec and benchmark suites and their READMEs.
Part of a fix for GH-32.
PEG.js source code becomes a set of Node.js modules that include each
other as needed. The distribution version is built by bundling these
modules together, wrapping them inside a bit of boilerplate code that
makes |module.exports| and |require| work.
Part of a fix for GH-32.
When the Git repository will be a npm package, there will be no
preprocessing step and thus no @VERSION substitution. Let's get rid of
it.
Part of a fix for GH-32.
Change the value of the |name| property of |PEG.GrammarError| instances
from "PEG.GrammarError" to just "GrammarError". This better reflects the
fact that PEG.js can get required under different name than "PEG".
Before this commit, generated parser were able to start parsing from any
rule. This was nice, but it made rule code inlining impossible.
Since this commit, the list of allowed start rules has to be specified
explicitly using the |allowedStartRules| option of the |PEG.buildParser|
method (or the --allowed-start-rule option on the command-line). These
rules will be excluded from inlining when it's implemented.
David Majda
David Majda
Almad