Reserved word detection as it was implemented in the JavaScript example
grammar had two big downsides:
1. It required changes in ordering of choices in some rules in order
not to trigger the detection prematurely. One of the changes was
already implemented (in the |Statement| rule, see the diff), but
apparently more were needed (the grammar didn't parse inputs like
|true| or |function f() {}|). And I'm not 100% sure that would be
the end of it (maybe deeper structural changes would be needed).
2. It made error messages confusing. Consider the following example:
var a = @;
Instead of reporting:
Expected ... but "@" found.
the generated parser reported:
Reserved word "var" can't be used as an identifier.
This was because the parser parsed the statement first as
|VariableStatement| and when this failed, it tried to parse it as
|ExpressionStatement|, triggering the reserved word detection.
Because of these, I decided to remove reserved word detection from the
JavaScript example grammar.
Fixes a problem where statements starting with a reserved word produced
errors like this:
Reserved word "return" can't be used as an identifier.
The problem was in a wrong ordering of choices in the |Statement| rule
together with aggressive reserved word detection in the |Identifier|
rule.
Instead of matching segments between blocks character by character,
match them as a whole. Also align the style with other similar rules
(e.g. the comment ones).
Before this commit, line continuations in character classes contributed
an empty string to the list of characters and character ranges matched
by a class. While this didn't lead to a buggy behavior with the current
code generator, the AST was wrong and the problem could have caused bugs
later.
This commit fixes the problem.
Semantic predicates are kind of |PrimaryExpression|, not kind of
|PrefixedExpression|. Therefore I extracted a rule for them and
referenced it from the |PrimaryExpression|.
Initializer and rules are now separated in a similar way as JavaScript
statements -- either by a semicolon or a line terminator, possibly with
whitespace and comments mixed in.
One consequence is that the grammars like this are now illegal:
foo = "a" bar = "b"
A semicolon needs to be inserted between the rules:
foo = "a";bar = "b"
I consider this a good change as the now-illegal syntax was somewhat
confusing.
This makes the |Primary| rule a bit more tidy. Also, matching the |.|
character really belongs to the lexical part of the grammar, next to
literals and character classes.
* Rename the |Action| rule to |CodeBlock| (it better describes what
the rule matches).
* Implement the rule in a simpler way and move it after more basic
lexical elements.
This change has two side effects:
* Label names can no longer be JavaScript reserved words.
* |$| is allowed again in label names. However, because of the
preference rules, names starting with it will be usually parsed as a
text operator followed by another identifier (denoting a rule
reference or label name).
Before this commit, whitespace was handled at the lexical level by
making tokens consume any whitespace coming after them. This was
accomplished by appending |__| to every token rule.
This commit changes whitespace handling to be more explicit. Tokens no
longer consume whitespace coming after them and syntactic rules have to
cope with it. While this slightly complicates the syntactic grammar, I
think it's a cleaner way. Moreover, it is what JavaScript example
grammar does.
One small side-effect of thich change is that the grammar is now
stand-alone (it doesn't require utils.js anymore).
When rule names are capitalized, it's easier to visually distinguish
them from non-capitalized label names. Moreover, I use capitalized rule
names in all my grammars these days.
Before this commit, a line continuation (backslash followed by a line
terminator character) contributed a character to a string or a character
class it was used in. In JavaScript and many other languages, line
continuation doesn't contribute anything.
This commit aligns PEG.js line continuation behavior with JavaScript.
Before this commit, the value of the |rawText| property of "class" AST
nodes was created in a hackish way from processed input and it didn't
always exactly represent the actual input text.
This commit changes the code so that the value of the |rawText| property
is created using the |text| function. This is a clean way which also
resolves the exact representation problem.
This is a complete rewrite of the CSS example grammar. It is now based
on CSS 2.1 *including the errata* and the generated parser builds a
nicer syntax tree. There is also a number of cleanups, formatting
changes, naming changes, and bug fixes.
Beside this, the rewrite reflects how I write grammars today (as opposed
to few years ago) and what style I would recommend to others.
This is a complete rewrite of the JavaScript example grammar. It is now
based on ECMA-262, 5.1 Edition and the generated parser builds a syntax
tree compatible with Mozilla SpiderMonkey Parser API. There is also a
number of cleanups, formatting changes, naming changes, and bug fixes.
Beside this, the rewrite reflects how I write grammars today (as opposed
to few years ago) and what style I would recommend to others.
This is a complete rewrite of the JSON example grammar. It is now based
on RFC 7159 instead of an informal description at the JSON website.
Beside this, the rewrite reflects how I write grammars today (as opposed
to few years ago) and what style I would recommend to others.
This is a complete rewrite of the arithmetics example grammar. It now
allows whitespace between tokens, supports "-" and "/" operators, and
gets the operator associativity right. Also, rule names now match the usual
conventions (term, factor,...).
Beside this, the rewrite reflects how I write grammars today (as opposed
to few years ago) and what style I would recommend to others.
Also added few missing |hasOwnProperty| calls that JSHint didn't detect
because it only looks whether there is an |if| statement wrapping the
loop body.
Fixes the following JSHint errors:
bin/pegjs: line 66, col 14, 'extraOptions' used out of scope.
bin/pegjs: line 70, col 19, 'extraOptions' used out of scope.
bin/pegjs: line 71, col 20, 'extraOptions' used out of scope.
bin/pegjs: line 80, col 10, Wrap the /regexp/ literal in parens to disambiguate the slash operator.
bin/pegjs: line 128, col 43, Missing semicolon.
bin/pegjs: line 128, col 45, Don't make functions within a loop.
bin/pegjs: line 150, col 13, Redefinition of 'module'.
bin/pegjs: line 217, col 34, Expected '===' and instead saw '=='.
bin/pegjs: line 243, col 44, 'source' used out of scope.
bin/pegjs: line 243, col 61, 'source' used out of scope.
Fixes the following JSHint error:
lib/compiler/passes/generate-bytecode.js: line 334, col 71, Expected an assignment or function call and instead saw an expression.
David Majda
David Majda