Accessing JavaCC SPECIAL_TOKEN Comments in JJTree Visitors

In a JavaCC developer community I participate in, a user recently asked: “How can I access comments defined as SPECIAL_TOKEN from within my JJTree Visitor? The comments are in the file, but my nodes seem to ignore them.”

This is a classic hurdle when building compilers or linters with JavaCC. By design, SPECIAL_TOKEN definitions (like comments or whitespace) are ignored by the parser’s production rules so they don’t clutter the grammar. However, JavaCC attaches these tokens to the “real” tokens that follow them via a linked list. If you are using JJTree to generate an AST (Abstract Syntax Tree), those comments aren’t automatically properties of your nodes—you have to manually traverse the token chain.

The Immediate Fix: Traversal via specialToken

To access a comment associated with a node in your Visitor, you must navigate from the node’s first token back through the specialToken pointer.

/* Illustrative example — JavaCC 7.0.12 / JJTree */

public Object visit(ASTMyNode node, Object data) {
    // 1. Get the first token associated with this AST node
    Token t = node.jjtGetFirstToken();

    // 2. Look for special tokens (comments) appearing immediately before this token
    if (t.specialToken != null) {
        Token tmp = t.specialToken;

        // The special tokens are in a reverse linked list.
        // We navigate back to find the first one in the block.
        while (tmp.specialToken != null) {
            tmp = tmp.specialToken;
        }

        // 3. Print or process the comments
        while (tmp != null) {
            System.out.println("Found Comment: " + tmp.image);
            tmp = tmp.next;
        }
    }
    return node.childrenAccept(this, data);
}

---

Detailed Explanation

When you define a token as SPECIAL_TOKEN in your .jj or .jjt file, JavaCC doesn’t discard it. Instead, it populates a field called specialToken in the next “regular” token that the parser encounters.

If your grammar looks like this:

SPECIAL_TOKEN : { <SINGLE_LINE_COMMENT: "//" (~["\n","\r"])* ("\n"|"\r"|"\r\n")> }
TOKEN : { <IDENTIFIER: ["a"-"z"] (["a"-"z","0"-"9"])*> }

And your input is:

// This is a comment
myVariable

The Token object for myVariable will have its specialToken field pointing to the // This is a comment token.

Why the Visitor doesn’t see them by default

JJTree creates nodes based on your productions. While JJTree can be configured to store the “First” and “Last” tokens for every node, it does not automatically concatenate the specialToken strings into the node’s data because that would be memory-intensive and often unnecessary for simple expression evaluation.

---

Alternative Solution: Capturing Comments in the Node

If you need to access comments frequently and don’t want to traverse the token chain in every visitor method, you can use the NODE_USES_PARSER option or a custom Node class to capture the comment during the parsing phase.

Step 1: Update JJTree Options Set these in your .jjt file:

options {
  TRACK_TOKENS = true; // Ensures jjtGetFirstToken() is available
  VISITOR = true;
}

Step 2: Add a helper method to your Base Node (e.g., SimpleNode.java) Modify the generated SimpleNode.java (or your custom base class) to include a helper that extracts the comment:

/* Illustrative example — JavaCC 6.x / 7.x */

public String getLeadingComments() {
    Token t = jjtGetFirstToken();
    if (t == null || t.specialToken == null) return "";

    StringBuilder sb = new StringBuilder();
    Token special = t.specialToken;
    // Walk back to the start of the special token chain
    while (special.specialToken != null) special = special.specialToken;
    // Walk forward to collect them in order
    while (special != null) {
        sb.append(special.image);
        special = special.next;
    }
    return sb.toString();
}

---

Edge Cases and Pitfalls

1. Multiple Comments: If you have three separate comment lines before a production, they are linked together. Always walk the specialToken list until you find a null to ensure you catch the entire block.
2. Trailing Comments: A comment at the very end of a file, with no regular token following it, will be attached to the EOF token. If your AST node doesn’t include the EOF, you won’t find that comment via node.jjtGetLastToken().
3. Whitespace: If you defined whitespace as a SPECIAL_TOKEN instead of using SKIP, your specialToken chain will contain both the comments and the space/tab characters. You will need to filter the kind of token while iterating.

---

Related Developer Concerns

How do I handle comments inside a list of items? If you have a list like [ item1, /* comment */ item2 ], the comment is attached to item2. If you are visiting the parent “List” node, you must iterate through the children and check each child’s jjtGetFirstToken().

Can I modify comments in the Visitor and write them back? JavaCC is primarily a parser, not a transformation engine. While you can modify the image field of a Token, re-serializing the tree into source code requires a custom “unparser” or “pretty printer” Visitor that manually prints specialToken content, then the regular token content, recursively.

Is there a performance impact? Using TRACK_TOKENS = true increases the memory footprint of your AST because every node now holds references to two Token objects. For massive source files (e.g., 100k+ lines), this can be significant. If performance is critical, only capture tokens on specific nodes using #MyNode(true) syntax.