45 Commits (cf294ef23634be532b02fdc969f7e11a93e2c0fe)

Author SHA1 Message Date
David Majda cf294ef236 PEG.js grammar: Add limitations
The limitations are inherited from the JavaScript example grammar.
10 years ago
David Majda 0459ab6b37 PEG.js grammar: Formatting & comments 10 years ago
David Majda 6f2510e49e PEG.js grammar: Make rules with operators more generic 10 years ago
David Majda 45c29a886f PEG.js grammar: Extract the |SemanticPredicateExpression| rule
Semantic predicates are kind of |PrimaryExpression|, not kind of
|PrefixedExpression|. Therefore I extracted a rule for them and
referenced it from the |PrimaryExpression|.
10 years ago
David Majda da18f6a729 PEG.js grammar: Extract the |RuleReferenceExpression| rule
This makes the |Primary| rule a bit more tidy.
10 years ago
David Majda 8e6f98e45c PEG.js grammar: Extract the |ActionExpression| rule
Having it separated from the |SequenceExpression| rule is cleaner and
more logical.
10 years ago
David Majda 5c6f4dd38b PEG.js grammar: Append |Expression| to expression rule names
Makes the rule names a bit longer but also clearer.
10 years ago
David Majda 27c2d26203 PEG.js grammar: More JavaScript-like initializer/rule separation
Initializer and rules are now separated in a similar way as JavaScript
statements -- either by a semicolon or a line terminator, possibly with
whitespace and comments mixed in.

One consequence is that the grammars like this are now illegal:

  foo = "a" bar = "b"

A semicolon needs to be inserted between the rules:

  foo = "a";bar = "b"

I consider this a good change as the now-illegal syntax was somewhat
confusing.
10 years ago
David Majda 4ce7593f5f PEG.js grammar: Extract the |AnyMatcher| rule
This makes the |Primary| rule a bit more tidy. Also, matching the |.|
character really belongs to the lexical part of the grammar, next to
literals and character classes.
10 years ago
David Majda c0df01b092 PEG.js grammar: Improve code block handling
* Rename the |Action| rule to |CodeBlock| (it better describes what
    the rule matches).

  * Implement the rule in a simpler way and move it after more basic
    lexical elements.
10 years ago
David Majda 13f72bb19d PEG.js grammar: More JavaScript-like rules for identifiers
This change has two side effects:

  * Label names can no longer be JavaScript reserved words.

  * |$| is allowed again in label names. However, because of the
    preference rules, names starting with it will be usually parsed as a
    text operator followed by another identifier (denoting a rule
    reference or label name).
10 years ago
David Majda 0d6b91cb20 PEG.js grammar: More JavaScript-like rules for strings/literals/classes 10 years ago
David Majda bcb5271649 PEG.js grammar: More JavaScript-like rules for skipped elements 10 years ago
David Majda b463808b3f PEG.js grammar: Replace several smaller comments by a big initial one 10 years ago
David Majda a5a0609505 PEG.js grammar: Inline trivial character rules 10 years ago
David Majda ae89f5e469 PEG.js grammar: Change whitespace handling
Before this commit, whitespace was handled at the lexical level by
making tokens consume any whitespace coming after them. This was
accomplished by appending |__| to every token rule.

This commit changes whitespace handling to be more explicit. Tokens no
longer consume whitespace coming after them and syntactic rules have to
cope with it. While this slightly complicates the syntactic grammar, I
think it's a cleaner way. Moreover, it is what JavaScript example
grammar does.

One small side-effect of thich change is that the grammar is now
stand-alone (it doesn't require utils.js anymore).
10 years ago
David Majda 4725632641 PEG.js grammar: Capitalize rule names
When rule names are capitalized, it's easier to visually distinguish
them from non-capitalized label names. Moreover, I use capitalized rule
names in all my grammars these days.
10 years ago
David Majda fb72c430e6 PEG.js grammar: Fix line continuation handling
Before this commit, a line continuation (backslash followed by a line
terminator character) contributed a character to a string or a character
class it was used in. In JavaScript and many other languages, line
continuation doesn't contribute anything.

This commit aligns PEG.js line continuation behavior with JavaScript.
10 years ago
David Majda 3dbec0b30d PEG.js grammar: Fix how |rawText| is created
Before this commit, the value of the |rawText| property of "class" AST
nodes was created in a hackish way from processed input and it didn't
always exactly represent the actual input text.

This commit changes the code so that the value of the |rawText| property
is created using the |text| function. This is a clean way which also
resolves the exact representation problem.
10 years ago
David Majda df154daafb PEG.js grammar: Disallow empty sequences
Empty sequences are useless and they only confused users. Let's disallow
them.
10 years ago
David Majda 2f2152204a Refine error handling further
Before this commit, the |expected| and |error| functions didn't halt the
parsing immediately, but triggered a regular match failure. After they
were called, the parser could backtrack, try another branches, and only
if no other branch succeeded, it triggered an exception with information
possibly based on parameters passed to the |expected| or |error|
function (this depended on positions where failures in other branches
have occurred).

While nice in theory, this solution didn't work well in practice. There
were at least two problems:

  1. Action expression could have easily triggered a match failure later
     in the input than the action itself. This resulted in the
     action-triggered failure to be shadowed by the expression-triggered
     one.

     Consider the following example:

       integer = digits:[0-9]+ {
         var result = parseInt(digits.join(""), 10);

         if (result % 2 === 0) {
           error("The number must be an odd integer.");
           return;
         }

         return result;
       }

     Given input "2", the |[0-9]+| expression would record a match
     failure at position 1 (an unsuccessful attempt to parse yet another
     digit after "2"). However, a failure triggered by the |error| call
     would occur at position 0.

     This problem could have been solved by silencing match failures in
     action expressions, but that would lead to severe performance
     problems (yes, I tried and measured). Other possible solutions are
     hacks which I didn't want to introduce into PEG.js.

  2. Triggering a match failure in action code could have lead to
     unexpected backtracking.

     Consider the following example:

       class = "[" (charRange / char)* "]"

       charRange = begin:char "-" end:char {
         if (begin.data.charCodeAt(0) > end.data.charCodeAt(0)) {
           error("Invalid character range: " + begin + "-" + end + ".");
         }

         // ...
       }

       char = [a-zA-Z0-9_\-]

     Given input "[b-a]", the |charRange| rule would fail, but the
     parser would try the |char| rule and succeed repeatedly, resulting
     in "b-a" being parsed as a sequence of three |char|'s, which it is
     not.

     This problem could have been solved by using negative predicates,
     but that would complicate the grammar and still wouldn't get rid of
     unintuitive behavior.

Given these problems I decided to change the semantics of the |expected|
and |error| functions. They don't interact with regular match failure
mechanism anymore, but they cause and immediate parse failure by
throwing an exception. I think this is more intuitive behavior with less
harmful side effects.

The disadvantage of the new approach is that one can't backtrack from an
action-triggered error. I don't see this as a big deal as I think this
will be rarely needed and one can always use a semantic predicate as a
workaround.

Speed impact
------------
Before:     993.84 kB/s
After:      998.05 kB/s
Difference: 0.42%

Size impact
-----------
Before:     1019968 b
After:      975434 b
Difference: -4.37%

(Measured by /tools/impact with Node.js v0.6.18 on x86_64 GNU/Linux.)
10 years ago
David Majda f8b5e04bba Error handling: Use the new |error| function in PEG.js's grammar itself
Implements part of #198.
10 years ago
David Majda 86769a6c5c Error handling: Make |?| return |null| on unsuccessful match
Before this commit, the |?| operator returned an empty string upon
unsuccessful match. This commit changes the returned value to |null|. It
also updates the PEG.js grammar and the example grammars, which used the
value returned by |?| quite often.

Returning |null| is possible because it no longer indicates a match
failure.

I expect that this change will simplify many real-world grammars, as an
empty string is almost never desirable as a return value (except some
lexer-level rules) and it is often translated into |null| or some other
value in action code.

Implements part of #198.
10 years ago
David Majda 5942988f66 Remove the |startRule| property from the AST
It's redundant.
11 years ago
David Majda f0a6bc92cc Text nodes: Use text nodes in PEG.js grammar 11 years ago
David Majda 5e146fce38 Text nodes: Implement text nodes
Implement a new syntax to extract matched strings from expressions. For
example, instead of:

  identifier = first:[a-zA-Z_] rest:[a-zA-Z0-9_]* { return first + rest.join(""); }

you can now just write:

  identifier = $([a-zA-Z_] [a-zA-Z0-9_]*)

This is useful mostly for "lexical" rules at the bottom of many
grammars.

Note that structured match results are still built for the expressions
prefixed by "$", they are just ignored. I plan to optimize this later
(sometime after the code generator rewrite).
11 years ago
David Majda af20f024c7 Text nodes: Disallow the "$" character in identifiers
The "$" character will mark text nodes in the future.
11 years ago
David Majda 4cda79951a Git repo npmization: Compose PEG.js from Node.js modules
PEG.js source code becomes a set of Node.js modules that include each
other as needed. The distribution version is built by bundling these
modules together, wrapping them inside a bit of boilerplate code that
makes |module.exports| and |require| work.

Part of a fix for GH-32.
12 years ago
David Majda a4df483159 s/Modelled/Modeled/
"modelled" is a British variant, "modeled" an US one. PEG.js officially
uses American English.

Based on pull request by John Gietzen:

  https://github.com/dmajda/pegjs/pull/102
12 years ago
David Majda cdf23e0a49 Change ordering of "literal", "class" and "any" code
Changes all code that does something with "literal", "class" or "any"
AST nodes so that the code deals with these in the follwing order:

  1. literal
  2. class
  3. any

Previously the code used this ordering:

  1. literal
  2. any
  3. class

The new ordering is more logical because the nodes are handled from the
most specific to the most generic.
12 years ago
David Majda eb4badab24 Refactor named rules AST representation
PEG.js grammar rules are represented by |rule| nodes in the AST. Until
now, all such nodes had a |displayName| property which was either |null|
or stored rule's human-readable name. This commit gets rid of the
|displayName| property and starts representing rules with a
human-readable name using a new |named| node (a child of the |rule|
node).

This change simplifies code generation code a bit as tests for
|displayName| can be removed (see changes in generate-code.js). It also
separates different concerns from each other nicely.
12 years ago
David Majda a59516f89b Small reordering of properties when creating |class| nodes
General rule: Least important things/flags go last.
12 years ago
David Majda 7900b66c70 Fix |braced| rule in the PEG.js grammar
This fix does not change the behavior, it just makes the
|nonBraceCharacters| rule un-dead (as originally intended).
12 years ago
David Majda 4d5b1d58aa AST: Store rules in an array instead of an object
This simplifies the code a bit and makes the AST more regular (each node
type has a fixed set of properties). The latter may get useful later
when generalizing visitors.
12 years ago
David Majda c639c1fc83 PEG.js grammar: Replace two instances of |string / ""| by |string?| 13 years ago
David Majda c04af99df8 Implament case-insensitive class matching 13 years ago
David Majda b540b2d460 Implement case-insensitive literal matching 13 years ago
David Majda 1c11e4aaa3 Split |literal| rule in the PEG.js grammar to |literal| and |string|
This is just a formality now but it will make sense later when literals
(but not strings) will allow "i" flag for case-insensitive matching.
13 years ago
David Majda 67afc788ad src/parser.pegjs: Use radix in |parseInt| calls instead of "0x" prefix
Fixes the following JSHint errors:

  ./src/parser.js: line 2878, col 44, Missing radix parameter.
  ./src/parser.js: line 2949, col 44, Missing radix parameter.
13 years ago
David Majda 13c47d6c4f src/parser.pegjs: Replace "\0" with "\x00"
Fixes the following JSHint error:

  ./src/parser.js: line 2820, col 44, Bad escapement.
13 years ago
David Majda b80cd9cb02 src/parser.pegjs: Use strict comparison
Fixes the following JSHint errors:

  ./src/parser.js: line 460, col 50, Expected '!==' and instead saw '!='.
  ./src/parser.js: line 486, col 42, Expected '!==' and instead saw '!='.
13 years ago
David Majda cc416199be src/parser.pegjs: Add missing semicolons
Fixes the following JSHint errors:

  ./src/parser.js: line 193, col 18, Missing semicolon.
  ./src/parser.js: line 407, col 20, Missing semicolon.
  ./src/parser.js: line 2493, col 18, Missing semicolon.
  ./src/parser.js: line 2759, col 40, Missing semicolon.
13 years ago
David Majda 98cbd57ead Add two missing |var|s (fix global namespace pollution) 13 years ago
David Majda 1279e87766 Simplify utility functions structure + do not export them as part of the PEG object 14 years ago
David Majda e59f3ba338 Split the source code into several files, introduce build system
The source code is now in the src directory. The library needs to be
built using "rake", which creates the lib/peg.js file by combining the
source files.
14 years ago