Skip to content

XText Reference Implementation — Insights for sysmlpy

Status: Research notes (not a release) Author: Comparative analysis of the OMG SysML v2 Pilot Implementation Repository studied: https://github.com/Systems-Modeling/SysML-v2-Pilot-Implementation

1. The OMG Pilot Implementation is XText-based

The OMG SysML v2 Pilot Implementation is built on Eclipse XText (a Java/Eclipse framework for domain-specific languages). XText's grammar is a superset of ANTLR with semantic-action syntax ({Model::Class.feature = current}) for directly constructing the AST model during parsing.

The expression grammar lives in: - org.omg.kerml.expressions.xtext/src/org/omg/kerml/expressions/xtext/KerMLExpressions.xtext - org.omg.sysml.xtext/src/org/omg/sysml/xtext/SysML.xtext

SysML.xtext imports org.omg.kerml.expressions.xtext.KerMLExpressions (line: grammar org.omg.sysml.xtext.SysML with org.omg.kerml.expressions.xtext.KerMLExpressions).

2. The OMG grammar HAS proper precedence (per-operator rules)

The OMG XText grammar expresses each operator level as a separate grammar rule with a clear precedence cascade. This is the canonical, correct way to handle precedence in XText/ANTLR.

OwnedExpression → ConditionalExpression
ConditionalExpression → NullCoalescingExpression | 'if' cond '?' then 'else' else
NullCoalescingExpression → ImpliesExpression ( '??' ImpliesExpressionReference )*
ImpliesExpression → OrExpression ( 'implies' OrExpressionReference )*
OrExpression → XorExpression ( '|' | 'or' XorExpressionReference )*
XorExpression → AndExpression ( 'xor' AndExpression )*
AndExpression → EqualityExpression ( '&' | 'and' EqualityExpressionReference )*
EqualityExpression → ClassificationExpression ( '==' | '!=' | '===' | '!==' )*
ClassificationExpression → RelationalExpression ( 'hastype' | 'istype' | '@' | 'as' type )?
RelationalExpression → RangeExpression ( '<' | '>' | '<=' | '>=' )*
RangeExpression → AdditiveExpression ( '..' AdditiveExpression )?
AdditiveExpression → MultiplicativeExpression ( '+' | '-' )*                    [left-assoc]
MultiplicativeExpression → ExponentiationExpression ( '*' | '/' | '%' )*          [left-assoc]
ExponentiationExpression → UnaryExpression ( '**' | '^' ExponentiationExpression )?   [right-assoc via ?]
UnaryExpression → ( '+' | '-' | '~' | 'not' ) ExtentExpression | ExtentExpression
ExtentExpression → 'all' type | PrimaryExpression
PrimaryExpression → BaseExpression ( '.' FeatureChainMember ( '[' ']' | '#' '(' ')' | '->' ... )* )*
BaseExpression → NullExpression | LiteralExpression | FeatureReferenceExpression
                | MetadataAccessExpression | InvocationExpression | ConstructorExpression
                | BodyExpression | '(' SequenceExpression ')'

Key design choices in the OMG grammar

  1. Same-precedence operators are merged in a single ( op operand )* loop with left-associativity. E.g. + and - are both at the additive level: 

    AdditiveExpression → MultiplicativeExpression ( ('+' | '-') MultiplicativeExpression )*

  2. Right-associative operators use ( op level )? (single optional op): 

    ExponentiationExpression → UnaryExpression ( ('**' | '^') ExponentiationExpression )?

  3. "Reference" variants wrap lower-precedence expressions as cross-references. The trick: when an operator's operand needs to be at a LOWER binding level (e.g., and taking an or-level operand), the operand is wrapped in <X>Reference which holds a membership/relationship to the lower-level expression. This produces the correct semantic structure for the AST.

Example from the OMG grammar: 

AndExpression → EqualityExpression ( ('&' EqualityExpression | 'and' EqualityExpressionReference) )*
                                                                                 ^^^^^^^^^^^^^^^^^^^^
The keyword form and takes an EqualityExpressionReference (a membership relationship) instead of a direct EqualityExpression. The membership relationship is how the AST preserves the source-level grouping.

  1. XText semantic actions attach the operator and operand to the current expression being built. Example: 
    AdditiveExpression returns SysML::Expression :
    MultiplicativeExpression ( {SysML::OperatorExpression.operand += current}
                               operator = AdditiveOperator operand += MultiplicativeExpression )*
;
    The {...} block runs after the LHS is matched, creating an OperatorExpression model element with operand set to the LHS and then assigning operator and the next operand to it.

3. The daltskin auto-generated ANTLR4 grammar has a precedence bug

The vendored grammar in src/sysmlpy/grammar/antlr4/SysMLv2Parser.g4 was auto-generated by daltskin/sysml-v2-grammar. The daltskin README states:

"Transform — Converts to ANTLR4 with precedence-climbing expressions, keyword extraction, and spec-ambiguity patches"

But the actual generated grammar has ALL operators collapsed into a SINGLE ownedExpression rule (line 22 in the vendored file):

ownedExpression
    : IF ownedExpression QUESTION ownedExpression ELSE ownedExpression
    | ownedExpression QUESTION_QUESTION ownedExpression
    | ownedExpression IMPLIES ownedExpression
    | ownedExpression OR ownedExpression
    | ownedExpression AND ownedExpression
    | ownedExpression XOR ownedExpression
    | ownedExpression PIPE ownedExpression
    | ownedExpression AMP ownedExpression
    | ownedExpression ( EQ_EQ | BANG_EQ | EQ_EQ_EQ | BANG_EQ_EQ) ownedExpression
    | ownedExpression ( LT | GT | GT | LE | GE) ownedExpression
    | ownedExpression DOT_DOT ownedExpression
    | ownedExpression ( PLUS | MINUS) ownedExpression
    | ownedExpression ( STAR | SLASH | PERCENT) ownedExpression
    | <assoc = right> ownedExpression ( STAR_STAR | CARET) ownedExpression
    | ( PLUS | MINUS | TILDE | NOT) ownedExpression
    | ...
    ;

This is a flat list — all operators are at the same precedence level. ANTLR4 resolves this with left-association and greedy matching: when a parser is at a ownedExpression choice-point and sees a + b * c, it consumes a + b (because the LHS of the + alternative greedily takes as much as possible), then sees * next which matches the * alternative, so the parse is (a + b) * c. The mathematically correct parse is a + (b * c) because * binds tighter than +, but the grammar doesn't enforce that.

This is the root cause of the precedence issue documented in docs/v0.46.0_expression_capture_plan.md and worked around in the visitor with the precedence-climbing pass added in v0.52.0.

4. The fix: regenerate (or patch) the grammar with per-precedence rules

The cleanest fix is to regenerate the grammar with the per-precedence structure the OMG grammar uses. Two paths:

Path A: Regenerate from daltskin (preferred)

File an issue / PR on https://github.com/daltskin/sysml-v2-grammar asking them to apply the "precedence-climbing expressions" transform they document. Their scripts/generate_grammar.py has an apply_patches() method that should be updated to generate per-precedence rules.

Once the grammar is regenerated, the sysmlpy visitor can be simplified back to a straightforward recursive-descent emit (no precedence-climbing needed in the visitor because the grammar does it).

Path B: Patch the vendored grammar manually

Patch src/sysmlpy/grammar/antlr4/SysMLv2Parser.g4 to split the ownedExpression rule into the per-precedence rules shown in §2. This requires regenerating the Python ANTLR parser from the patched grammar:

java -jar antlr-4.13.2-complete.jar -Xexact-output-dir \
    -o src/sysmlpy/antlr \
    -visitor -listener -Dlanguage=Python3 \
    src/sysmlpy/grammar/antlr4/SysMLv2Lexer.g4 \
    src/sysmlpy/grammar/antlr4/SysMLv2Parser.g4

After the grammar fix, the visitor's precedence-climbing pass can be removed (replaced with straightforward recursive-descent emit) and the chain layout will match the grammar class layout exactly.

5. Why sysmlpy works today anyway (v0.52.0)

Even with the grammar precedence bug, sysmlpy v0.52.0's Phase 1 expression capture works because the visitor does its own precedence-climbing pass (_find_split_index / _build_binary_chain / _splice_operator / _add_op_to_layer / _embed_rhs_higher_ops in antlr_visitor.py). The pass: 1. Walks the ANTLR parse tree 2. Finds the lowest-precedence operator in the children 3. Re-arranges the structured chain so operators are at their correct binding-power layer 4. Embeds rhs's higher-layer operators when re-arranging

This duplicates what a correct grammar would do automatically. The grammar fix would let us remove ~300 lines of visitor code.

6. Concrete v0.53.0 work plan (if/when we want the grammar fix)

  1. Patch the vendored grammar (Path B above) to add per-precedence rules for ownedExpression.
  2. Regenerate the ANTLR Python parser with the patched grammar.
  3. Run conformance tests to verify the new grammar still passes 118/123.
  4. Simplify the visitor by removing the precedence-climbing helpers (_PRECEDENCE_RANK, _OPERATOR_TO_LAYER, _find_split_index, _splice_operator, _add_op_to_layer, _embed_rhs_higher_ops, _embed_layer_at_path, _unwrap_to_layer, _rhs_top_op_layer, _rhs_has_higher_op, _LAYER_PATHS, _LAYER_OP_FIELDS, _LAYER_TO_OPERATORS, etc.).
  5. Re-run the test suite to verify all 143 grammar tests + 11 phase1 tests still pass without the precedence-climbing pass.
  6. Document the change in CHANGELOG as "v0.53.0 — fix grammar precedence; remove visitor-side precedence-climbing".
  7. Submit the grammar patch upstream to daltskin/sysml-v2-grammar so other ANTLR4-based SysML v2 consumers can benefit.

7. References

  • OMG SysML v2 Release: https://github.com/Systems-Modeling/SysML-v2-Release
  • OMG SysML v2 Pilot Implementation: https://github.com/Systems-Modeling/SysML-v2-Pilot-Implementation
  • KerML Expressions XText grammar: org.omg.kerml.expressions.xtext/src/org/omg/kerml/expressions/xtext/KerMLExpressions.xtext
  • SysML XText grammar: org.omg.sysml.xtext/src/org/omg/sysml/xtext/SysML.xtext
  • daltskin ANTLR4 grammar: https://github.com/daltskin/sysml-v2-grammar
  • sysmlpy Phase 1 plan: docs/v0.46.0_expression_capture_plan.md
  • sysmlpy v0.52.0 CHANGELOG entry: implementation uses visitor-side precedence-climbing as a workaround for the grammar bug.