4 Enforestation
For expressions and other contexts that involve infix operators, the parse process can further divided into two interleaved processes:
Enforestation handles operator precedence within a ‹group› so that, for example, 1 + 2 * 3 is recognized as an addition expression overall with a multiplication subexpression.
Expansion consumes an enforestation result, such as + having two subexpression inputs, and it determines how to combine the pieces, such as generating a math.sum call to perform addition.
These processes are interleaved because the meaning of an operator is determined by the scope in which it is used. While *’s dominant use is clearly multiplication, it may also be used in a regular expression context to mean repetition or, in a language for writing typing rules, it might be used for the type of tuples. Accordingly, Rhombus operators acquire a specific precedence in a specific scope via the definition form for the operator. As we will see, macro definition forms like operator and expr.macro include precedence specifications, so the operator or macro that’s being defined has a specification of its behavior at both the enforestation and expansion layers. And, while precedence and associativity are the most common use cases to motivate enforestation, it is actually a general process of that converts a linear sequence of terms into a tree that can be programmed in a general way.
The enforestation and expansion layers also both work on shrubbery representations. Although enforestation may deliver fully parsed subexpressions to the expander for a simple infix or prefix operator, in the most general case, enforestation delivers only a parsed left argument to an infix operator. An expander can determine how to treat any shrubbery forms to the right of an infix or prefix operator. Expansion might choose to recur to enforestation to parse some terms, it may consume some parts directly, and it may leave any number of tokens in the enclosing ‹group›’s stream to be parsed further. For example, parsing [1, 2, 3].remove(2) |> println enforests the left-hand side of . to the expression [1, 2, 3], but the right-hand side of . starts with a method name remove, not an expression. The . expander determines that remove(2) should be consumed in order to expand into a method call, but it leaves |> println for further parsing. Accordingly, once the . operator finishes, the expansion and enforestation process picks up from there with the |> operator.
Going forward, we can mostly forget about the internal composition of enforestation and expansion and think in terms of parsing shrubbery representations, but sometimes it is useful to keep the interplay of the two processes in mind.
4.1 Defining Operators
The simplest way to define a new operator is using the operator form, which is analogous to fun for defining a function:
fun revlist(a, b):
[b, a]
> revlist("apple", "banana")
["banana", "apple"]
operator a <~> b:
[b, a]
> "apple" <~> "banana"
["banana", "apple"]
The term “operator” is somewhat overloaded. At the shrubbery level, operator refers to a syntactic category that is distinct from identifiers. A Rhombus operator in the operator sense can be an identifier, like mod.
Although they are normally defined at the top of a module, operator definitions can be in a local-definition context, just like function definitions.
> [
operator a <~> b:
["just", a]
"apple" <~> "banana",
operator a <~> b:
[b, "only"]
"apple" <~> "banana"
]
[["just", "apple"], ["banana", "only"]]
As defined, <~> can be used with single-term expressions to the left or right:
["banana cream cake", "apple pie"]
If we leave out parentheses, however, we get an error:
<~>: explicit parenthesization needed;
found operators without declared precedence or associativity
operator kind: expression operator
earlier operator: ++
We could declare <~> to be weaker than ++ specifically, but let’s declare weaker precedence than anything that uses the concatenation order, which is the order that ++ adopts.
Precedence relationships are pairwise, and they are not required to form an order in the mathematical sense, much less fit on a numerical scale. Operator orders like concatenation provide an indirection to specify pairwise relationships among groups of operators.
operator a <~> b:
~weaker_than: concatenation
[b, a]
["banana cream cake", "apple pie"]
["banana2", "apple1"]
The operator form supports prefix and postfix operators, too, and that works the way you’d expect.
4.2 Expression Macros
Like a function, an operator defined with operator receives value arguments, and it has no control over the order of evaluation or the parsing of those arguments at a place where the operator is used. A macro has that kind of control, because it can rearrange the syntax of each use of the macro.
An infix macro with expr.macro is similar to an infix operator defined with operator, except that a syntax pattern is used for the definition, and the body must produce a syntax object. The prefix operator’s name is extracted from the middle of the pattern.
The macro form is a simplified variant of expr.macro that is exported by rhombus, so it can be used without switching to rhombus/and_meta or importing rhombus/meta. Unlike expr.macro, macro does not allow arbitrary parsing-time code, and it restricts its body to have an immediate template that escapes using $ only to refer to pattern bindings. For this first example, macro would work just as well as expr.macro.
expr.macro '$a <~~~> $b':
~weaker_than: concatenation
'block:
[b_val, a_val]'
> "apple" <~~~> "banana"
["banana", "apple"]
In this example, the parser takes a syntax-object representation of the input stream, '"apple" <~~~> "banana"', and matches it to the pattern '$a <~~~> $b'. The resulting expansion with "apple" in place of $a and "banana" in place of $b is incorporated back into the input stream for further parsing.
The block and let forms are not really necessary to achieve the intended ordering here, since '[$b, $a]' would also cause the argument for b to be evaluated before the argument for a. Either way, the reverse order illustrates how the macro has control, in contrast to an operator defined with operator, and we can expose the change in order by using arguments that have side effects.
> trace("apple") <~> trace("banana")
"apple"
"banana"
["banana", "apple"]
> trace("apple") <~~~> trace("banana")
"banana"
"apple"
["banana", "apple"]
The last example above illustrates a subtlety of expr.macro patterns. The a and b arguments are matched to multi-term expressions trace("apple") and trace("banana"), which would not happen with the same pattern in match:
> match 'trace("apple") <~~~> trace("banana")'
"ok"
match: expected the operator ‘<~~~>‘
The <~~~> macro works because expr.macro recognizes a specific shape as a shorthand for parsing left- and right-hand expressions first, taking into account the ~weaker_than precedence declaration. It completes the pattern match only after expressions to the left and right of <~~~> are parsed. Those parsed terms as then passed along to the macro as single-term representations of already-parsed expressions. We will return later to the precise rules for patterns and the specific shapes that expr.macro recognizes and automatically adjusts. For now, it suffices to realize that expr.macro has some smarts to make the common case easy.
4.3 Exercise
Implement a log_as infix macro that prints the result of its left-hand argument expression, but first prints its right-hand argument expression as a debugging label (without evaluating the expression), and print => in between. In other words, make this test pass:
~prints "2 * 3 => 6"
You may find it helpful to first experiment with this expression:
Be sure to generate a string for the right-hand argument of log_as at compile time, instead of delaying the string conversion to run time.
4.4 Exercise
It may have occurred to you that operator can be implemented by generating (1) a function to hold the operator body and (2) an expr.macro form to expand a use of the operator to a function call. Your task in this exercise is to define my_operator that way.
The starting code my_operator.rhm solves a few problems for you:
A suitable pattern for my_operator is set up already. This pattern limits my_operator to defining an infix operator with plain identifiers to represent the arguments. The Name syntax class matches both identifiers and operators. Since $body is not only alone within its ‹group› but also alone within it ‹block›, it can match a sequence of ‹group›s.
You will need to generate an expr.macro form, which uses '…' itself. So, the my_operator implementation uses '«…»' for the result template to avoid an opening ' in generated code from being treated as a closing ' for the overall template.
Generating an expr.macro form will be tricky for a second reason: you need $ sometimes as an escape for the '«…»' template, and sometimes as a literal $ that should appear in the expansion. Use $('$') to generate a literal $ in the macro expansion; that works because a $ by itself in '$' is not treated as an escape.