The term "Pegex" can be used to mean both the Pegex Parser Framework and also the Pegex Grammar Language Syntax that is used to write Pegex grammar files. This document details the Pegex Syntax.
Pegex is a self-hosting language. That means that the grammar for defining the Pegex Language is written in the Pegex Language itself. You can see it for yourself here: https://github.com/ingydotnet/pegex-pgx/blob/master/pegex.pgx.
I encourage you to take a quick look at that link even now. A Pegex grammar (like this one) is made up of 2 parts: a meta section and a rule section.
The meta section just contains keyword/value meta attributes about the grammar. Things like the grammar's name and version.
The real meat of a Pegex grammar is in its rules. The very first rule of the grammar above is (basically):
grammar: meta_section rule_section
Which says, a grammar IS a meta_section followed by a rule_section. But hey, we already knew that!
The meta statements ate the top of a grammar file look like this:
Let's look at the top the the pegex.pgx grammar:
# This is the Pegex grammar for Pegex grammars! %grammar pegex %version 0.1.0
This defines two meta values:
version, which specify the name and the version of the grammar, respectively.
You'll also notice that the first line is a comment. Comments start with a
# and go until the end of the line. Comments are allowed almost anywhere in the grammar, both on their own lines, after statements, and even within regex definitions as we will see later.
The Pegex Meta Section ends when the Pegex Rule Section begins (with the first rule definition).
The remainder of a Pegex grammar is a set of named rules. Each rule is a rule name, followed by a ':', followed by the definition of the rule, followed by a ';' or a newline.
Here are a couple rules from the pegex.pgx grammar. (These are the rules that start to define a rule!).
rule_definition: rule_start rule_group ending rule_start: / ( <rule_name> ) # Capture the rule_name <BLANK>* <COLON> ~ /
Rule definitions are infix expressions. They consist of tokens separated by operators, with parentheses to disambiguate binding precedence. There are 3 distinct tokens and 3 operators.
The 3 token types are: rule-reference, regex and error-message. The 3 operators are AND (' '), OR ('|') and ALT ('%', '%%').
Here's an example from a Pegex grammar for parsing JSON:
json: hash | array array: /<LSQUARE>/ ( node* % /<COMMA>/ ) ( /<RSQUARE>/ | `missing ']'` )
This is saying: "json is either a hash or array. array is '[', zero or more nodes separated by commas, and a ']'. error if no ']'".
node are rule references, meaning that they refer to named rules within the grammar that must match at that point. Text surrounded by a pair of '/' chars forms a regex. Text surrounding by backticks is an error message.
COMMA are also rule references. Rules may be referred to inside of regexes, as long as they refer to regexes themselves. In this way big regexes can be assembled from smaller ones, thus leading to reuse and readability. Finally, the '*' after
node is called a "quntifier". More about those later.
A rule reference is the name of a rule inside angle brackets. The brackets are usually optional outside of a regex, and always required inside a regex.
When used outside a regex, a reference can have a number of prefix modifiers. Note the the angle brackets are not required here, but add to readability.
=<rule> # Zero-width positive assertion (look-ahead) !<rule> # Zero-width negative assertion (look-ahead) .<rule> # Skip (ie: parse but don't capture a subpattern) -<rule> # Pass (never wrap) +<rule> # Always wrap
A reference can also have a number of suffixed quantifiers. Similar to regular expression syntax, a quantifier indicates how many times a rule (reference) should match.
rule? # optional rule* # 0 or more times rule+ # 1 or more times <rule>8 # exactly 8 times <rule>2+ # 2 or more times <rule>2-3 # 2 or 3 times <rule>0-6 # 0 to 6 times
Note that you must use angle brackets if you are using a numbered modifier:
rule8 # WRONG! This would match rule "rule8". rule2+ # WRONG! This would match rule "rule2", 1 or more times. rule2-3 # WRONG! Pegex syntax error
There is a special set of predefined "Atoms" that refer to regular expression fragments. Atoms exist for every punctuation character and for characters commonly found in regular expressions. Atoms enchance readability in grammar texts, and allow special characters (like slash or hash) to be used as Pegex syntax.
For example, a regex to match a comment might be '#' followed by anything, followed by a newline. In Pegex, you would write:
comment: / <HASH> <ANY>* <EOL> /
Pegex would compile the former into the latter.
Here are some atoms:
<TILDE> # ~ <SLASH> # / <HASH> # # (literal) <QMARK> # ? (literal) <STAR> # * (literal) <LPAREN> # ( (literal) <RPAREN> # ) (literal) <WORD> # \w <WS> # \s
The full list can be found in the Atoms source code.
In Pegex we call the syntax for a regular expression a "regex". ie When the term "regex" is used, it is referring to Pegex syntax, and when the term "regular expression" is used it refers to the actual regular expression that the regex is compiled into.
A regex is a string inside forward slashes.
The regex syntax mostly follows Perl, with the following exceptions:
# Any rules in angle brackets are referenced in the regex / ( <rule1> | non_rule ) / # "non_rule" is interpreted literally # The syntax implies a /x modifier, so whitespace and comments are # ignored. / ( <rule1>+ # Match rule1 one or more times | <rule2> ) / # Whitespace is declared with tildes. / ~ <rule3> ~~ / # ~ = \s*, ~~ = \s+, etc. # Any (?XX ) syntax can have the question mark removed / (: <a> | <b> ) / # same as / (?: <a> | <b> ) /
An error message is a string inside backticks. If the parser gets to an error message in the grammar, it throws a parse error with that message.
The Pegex operators in descending precedence order are: ALT, AND, and OR.
AND and OR are the most common operators. AND is represented by the absence of an operator. Like in these rules:
r1: <a><b> r2: a b
Those are both the same. They mean rule
a AND (followed immediately by) rule
OR means match one or the other.
r: a | b | c
means match rule
a OR rule
b OR rule
c. The rules are checked in order and if one matches, the others are skipped.
ALT means alternation. It's a way to specify a separator in a list.
r: a+ % b
would match these:
a aba ababa
%% means that a trailing separator is optional.
r: a+ %% b
would match these:
a ab aba abab
These rules have the same binding precedence:
r1: a b | c % d r2: (a b) | (c % d)
Parens are not only used for indicating binding precedence; they also can create quantifiable groups:
r1: (a b)+ c
The same quantifiers that can be applied to references, can be applied to groups too.