Change how found strings are escaped when building syntax error
messages:
* Do not escape non-ASCII characters (U+0100-U+FFFF). They are
typically more readable in their raw form.
* Escape DEL (U+007F). It is a control character.
* Escape NUL (U+0000) as "\0", not "\x00".
* Do not use less known shortcut escape sequences ("\b", "\f"), only the
well-known ones ("\0", "\t", "\n", "\r").
These changes mirror expectation escaping changes done in
4fe682794d.
Part of work on #428.
Before this commit, descriptions of literals used in error messages were
built by applying JavaScript string escaping to their values, making the
descriptions look like JavaScript strings. Descriptions of character
classes were built using their raw text. These approaches were mutually
inconsistent and lead to descriptions which were over-escaped and not
necessarily human-friendly (in case of literals) or coupled with details
of the grammar (in case of character classes).
This commit changes description building code in both cases and unifies
it. The intent is to generate human-friendly descriptions of matched
expressions which are clean, unambiguous, and which don't escape too
many characters, while handling special characters such as newlines
well.
Fixes#127.
I no longer think that using raw literal texts in error messages is the
right thing to do. The main reason is that it couples error messages
with details of the grammar such as use of single or double quotes in
literals. A better solution is coming in the next commit.
This reverts commit 69a0f769fc.
In most places, we talk about "generating a parser", not "building a
parser", which the function name should reflect. Also, mentioning a
parser in the name is not necessary as in case of a parser generator
it's pretty clear what is generated.
So far, PEG.js was exported in a "PEG" global variable when no module
loader was detected. The same variable name was also conventionally used
when requiring it in Node.js or otherwise referring to it. This was
reflected in various places in the code, documentation, examples, etc.
This commit changes the variable name to "peg" and fixes all relevant
occurrences. The main reason for the change is that in Node.js, modules
are generally referred to by lower-case variable names, so "PEG" was
sticking out when used in Node.js projects.
The wrapping functions are also generated by PEG.js, so the comment
should be above them to mark them as such. This shouldn't cause any
problems technically.
Introduce two ways of specifying parser dependencies: the "dependencies"
option of PEG.buildParser and the -d/--dependency CLI option. Specified
dependencies are translated into AMD dependencies and Node.js's
"require" calls when generating an UMD parser.
Part of work on #362.
Extract "generateWrapper" code which generates code of the intro and the
returned parser object into helper functions. This is pure refactoring,
generated parser code is exactly the same as before.
This change will make it easier to modifiy "generateWrapper" to produce
UMD modules.
Part of work on #362.
Code which was at the toplevel of the "generateJS" function in the code
generator is now split into "generateToplevel" (which genreates parser
toplevel code) and "generateWrapper" (which generates a wrapper around
it). This is pure refactoring, generated parser code is exactly the same
as before.
This change will make it easier to modifiy the code genreator to produce
UMD modules.
Part of work on #362.
Instead of testing arguments.length to see whether an optional parameter
was passed to a function, compare its value to "undefined". This
approach has two advantages:
* It is in line with handling of default parameters in ES6.
* Optional parameters are actually spelled out in the parameter
list.
There is also one important disadvantage, namely that it's impossible to
pass "undefined" as an optional parameter value. This required a small
change in two tests.
Additional notes:
* Default parameter values are set in assignments immediately
after the function header. This reflects the fact that these
assignments really belong to the parameter list (which is where they
are in ES6).
* Parameter values are checked against "void 0" in places where
"undefined" can potentially be redefiend.
The "toParse" matcher in generated-parser-behavior.spec.js effectively
had these signatures:
toParse(input)
toParse(input, expected)
toParse(input, options, expected)
This commit regularizes them to:
toParse(input)
toParse(input, expected)
toParse(input, expected, options)
Similarly, the "toFailToParse" matcher in
generated-parser-behavior.spec.js effectively had these signatures:
toFailToParse(input)
toFailToParse(input, details)
toFailToParse(input, options, details)
This commit regularizes them to:
toFailToParse(input)
toFailToParse(input, details)
toFailToParse(input, details, options)
Finally, the "toChangeAST" matcher in helpers.js effectively had these
signatures:
toChangeAST(grammar, details)
toChangeAST(grammar, options, details)
This commit regularizes them to:
toChangeAST(grammar, details)
toChangeAST(grammar, details, options)
The overall purpose of these changes is to avoid different parameters
appearing at the same position, which is hard to manage without using
"arguments".
Labels in expressions like "(a:"a")" or "(a:"a" b:"b" c:"c")" were
visible to the outside despite being wrapped in parens. This commit
makes them invisible, as they should be.
Note this required introduction of a new "group" AST node, whose purpose
is purely to provide label scope isolation. This was necessary because
"label" and "sequence" nodes don't (and can't!) provide this isolation
themselves.
Part of a fix of #396.
Semantic predicate and action specs which verified label scope didn't
exercise labels in subexpressions. This commit adds cases exercising
them, including few commented-out cases which reveal #396 (these will be
uncommented when the bug gets fixed).
Note that added specs exercise all relevant expression types. This is
needed because code that makes subexpression labels invisible to the
outside is separate for each expression type so one generic test
wouldn't generate enough coverage.
Part of a fix of #396.
So far, semantic predicate and action specs which verified scope of
labels from containing or outer sequences exercised only cases where
label variables were defined. This commit adds also some negative cases.
The idea comes from @Mingun.
Semantic predicate and action specs which verified scope of labels from
outer sequences exercised all relevant expression types. This is not
needed as the behavior is common for all expression types and no extra
code needs to be written to make it work for each of them.
This commit changes the specs to verify scope of labels from outer
sequences using only one expression type.
Semantic predicate specs which verified scope of labels from containing
sequences used 3 elements where 1 is enough.
This commit removes the redundant elements.
Semantic predicate and action specs which verified label scope used
repetitive "it" blocks. Rewrite them to use just one "it" block and a
list of testcases. This makes them more concise.
Semantic predicate specs which verified label scope also checked parser
results. This is not necessary because for the purpose of these specs it
is enough to verify that label variables have correct values, which is
done in predicate code already.
This commit removes parser result checks from these specs.
The "unescaped" rule was created by mechanically translating original
RFC 7159 rule:
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
into:
unescaped = [\x20-\x21\x23-\x5B\x5D-\u10FFFF]
However, this mechanical translation was incorrect as PEG.js grammars
don't have 6-digit Unicode escape sequences. Sequence "\u10FFFF" was
interpreted as "\u10FF" followed by two "F" characters.
This commit rewrites the "unescaped" rule into a form which, while not
being a mechanical translation of the original rule, matches the same
characters in the whole Unicode range. It also macthes textual
description of string representation in RFC 7159:
All Unicode characters may be placed within the quotation marks,
except for the characters that must be escaped: quotation mark,
reverse solidus, and the control characters (U+0000 through U+001F).
Fixes#417.
Running bin/pegjs with one argument which was an extension-less file
name caused the file to be overwritten. This was because internal
extension rewriting logic didn't handle this case corectly.
This commit changes the logic from regexp-based to path.extname-based,
fixing the problem. The new code generates file names like this:
Input file name Output file name
------------------------------------
grammar.ext grammar.js
grammar.ext1.ext2 grammar.ext1.js
grammar. grammar.js
grammar grammar.js
Fixes#405.
Before this commit, generated parsers considered the following character
sequences as newlines:
Sequence Description
------------------------------
"\n" Unix
"\r" Old Mac
"\r\n" Windows
"\u2028" line separator
"\u2029" paragraph separator
This commit limits the sequences only to "\n" and "\r\n". The reason is
that nobody uses Unicode newlines or "\r" in practice.
A positive side effect of the change is that newline-handling code
became simpler (and likely faster).
Instead of setting ESLint environment to "node" globally, set it on
per-directory basis using separate .eslintrc.json files:
Directory Environment
-----------------------
bin node
lib commonjs
spec jasmine
It was impossible to use this approach for the "benchmark" directory
which contains a mix of files used in various environments. For
benchmark/run, the environment is set inline. For the other files, as
well as spec/helpers.js, the globals are declared manually (it is
impossible to express how these files are used just by a list of
environments).
Fixes#408.
Fix the following errors:
31:9 error "parser" is defined but never used no-unused-vars
406:14 error "expected" is defined but never used no-unused-vars
1304:15 error "s1" is defined but never used no-unused-vars
1386:15 error "s1" is defined but never used no-unused-vars
1442:15 error "s1" is defined but never used no-unused-vars
David Majda
Balázs Kutil