All Downloads are FREE. Search and download functionalities are using the official Maven repository.

JavaScript.src.antlr4.atn.ParserATNSimulator.js Maven / Gradle / Ivy

There is a newer version: 4.13.2
Show newest version
//
// [The "BSD license"]
//  Copyright (c) 2012 Terence Parr
//  Copyright (c) 2012 Sam Harwell
//  Copyright (c) 2014 Eric Vergnaud
//  All rights reserved.
//
//  Redistribution and use in source and binary forms, with or without
//  modification, are permitted provided that the following conditions
//  are met:
//
//  1. Redistributions of source code must retain the above copyright
//     notice, this list of conditions and the following disclaimer.
//  2. Redistributions in binary form must reproduce the above copyright
//     notice, this list of conditions and the following disclaimer in the
//     documentation and/or other materials provided with the distribution.
//  3. The name of the author may not be used to endorse or promote products
//     derived from this software without specific prior written permission.
//
//  THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS OR
//  IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES
//  OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
//  IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT,
//  INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
//  NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
//  DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
//  THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
//  (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF
//  THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//

//
// The embodiment of the adaptive LL(*), ALL(*), parsing strategy.
//
// 

// The basic complexity of the adaptive strategy makes it harder to understand. // We begin with ATN simulation to build paths in a DFA. Subsequent prediction // requests go through the DFA first. If they reach a state without an edge for // the current symbol, the algorithm fails over to the ATN simulation to // complete the DFA path for the current input (until it finds a conflict state // or uniquely predicting state).

// //

// All of that is done without using the outer context because we want to create // a DFA that is not dependent upon the rule invocation stack when we do a // prediction. One DFA works in all contexts. We avoid using context not // necessarily because it's slower, although it can be, but because of the DFA // caching problem. The closure routine only considers the rule invocation stack // created during prediction beginning in the decision rule. For example, if // prediction occurs without invoking another rule's ATN, there are no context // stacks in the configurations. When lack of context leads to a conflict, we // don't know if it's an ambiguity or a weakness in the strong LL(*) parsing // strategy (versus full LL(*)).

// //

// When SLL yields a configuration set with conflict, we rewind the input and // retry the ATN simulation, this time using full outer context without adding // to the DFA. Configuration context stacks will be the full invocation stacks // from the start rule. If we get a conflict using full context, then we can // definitively say we have a true ambiguity for that input sequence. If we // don't get a conflict, it implies that the decision is sensitive to the outer // context. (It is not context-sensitive in the sense of context-sensitive // grammars.)

// //

// The next time we reach this DFA state with an SLL conflict, through DFA // simulation, we will again retry the ATN simulation using full context mode. // This is slow because we can't save the results and have to "interpret" the // ATN each time we get that input.

// //

// CACHING FULL CONTEXT PREDICTIONS

// //

// We could cache results from full context to predicted alternative easily and // that saves a lot of time but doesn't work in presence of predicates. The set // of visible predicates from the ATN start state changes depending on the // context, because closure can fall off the end of a rule. I tried to cache // tuples (stack context, semantic context, predicted alt) but it was slower // than interpreting and much more complicated. Also required a huge amount of // memory. The goal is not to create the world's fastest parser anyway. I'd like // to keep this algorithm simple. By launching multiple threads, we can improve // the speed of parsing across a large number of files.

// //

// There is no strict ordering between the amount of input used by SLL vs LL, // which makes it really hard to build a cache for full context. Let's say that // we have input A B C that leads to an SLL conflict with full context X. That // implies that using X we might only use A B but we could also use A B C D to // resolve conflict. Input A B C D could predict alternative 1 in one position // in the input and A B C E could predict alternative 2 in another position in // input. The conflicting SLL configurations could still be non-unique in the // full context prediction, which would lead us to requiring more input than the // original A B C. To make a prediction cache work, we have to track the exact // input used during the previous prediction. That amounts to a cache that maps // X to a specific DFA for that context.

// //

// Something should be done for left-recursive expression predictions. They are // likely LL(1) + pred eval. Easier to do the whole SLL unless error and retry // with full LL thing Sam does.

// //

// AVOIDING FULL CONTEXT PREDICTION

// //

// We avoid doing full context retry when the outer context is empty, we did not // dip into the outer context by falling off the end of the decision state rule, // or when we force SLL mode.

// //

// As an example of the not dip into outer context case, consider as super // constructor calls versus function calls. One grammar might look like // this:

// //
// ctorBody
//   : '{' superCall? stat* '}'
//   ;
// 
// //

// Or, you might see something like

// //
// stat
//   : superCall ';'
//   | expression ';'
//   | ...
//   ;
// 
// //

// In both cases I believe that no closure operations will dip into the outer // context. In the first case ctorBody in the worst case will stop at the '}'. // In the 2nd case it should stop at the ';'. Both cases should stay within the // entry rule and not dip into the outer context.

// //

// PREDICATES

// //

// Predicates are always evaluated if present in either SLL or LL both. SLL and // LL simulation deals with predicates differently. SLL collects predicates as // it performs closure operations like ANTLR v3 did. It delays predicate // evaluation until it reaches and accept state. This allows us to cache the SLL // ATN simulation whereas, if we had evaluated predicates on-the-fly during // closure, the DFA state configuration sets would be different and we couldn't // build up a suitable DFA.

// //

// When building a DFA accept state during ATN simulation, we evaluate any // predicates and return the sole semantically valid alternative. If there is // more than 1 alternative, we report an ambiguity. If there are 0 alternatives, // we throw an exception. Alternatives without predicates act like they have // true predicates. The simple way to think about it is to strip away all // alternatives with false predicates and choose the minimum alternative that // remains.

// //

// When we start in the DFA and reach an accept state that's predicated, we test // those and return the minimum semantically viable alternative. If no // alternatives are viable, we throw an exception.

// //

// During full LL ATN simulation, closure always evaluates predicates and // on-the-fly. This is crucial to reducing the configuration set size during // closure. It hits a landmine when parsing with the Java grammar, for example, // without this on-the-fly evaluation.

// //

// SHARING DFA

// //

// All instances of the same parser share the same decision DFAs through a // static field. Each instance gets its own ATN simulator but they share the // same {@link //decisionToDFA} field. They also share a // {@link PredictionContextCache} object that makes sure that all // {@link PredictionContext} objects are shared among the DFA states. This makes // a big size difference.

// //

// THREAD SAFETY

// //

// The {@link ParserATNSimulator} locks on the {@link //decisionToDFA} field when // it adds a new DFA object to that array. {@link //addDFAEdge} // locks on the DFA for the current decision when setting the // {@link DFAState//edges} field. {@link //addDFAState} locks on // the DFA for the current decision when looking up a DFA state to see if it // already exists. We must make sure that all requests to add DFA states that // are equivalent result in the same shared DFA object. This is because lots of // threads will be trying to update the DFA at once. The // {@link //addDFAState} method also locks inside the DFA lock // but this time on the shared context cache when it rebuilds the // configurations' {@link PredictionContext} objects using cached // subgraphs/nodes. No other locking occurs, even during DFA simulation. This is // safe as long as we can guarantee that all threads referencing // {@code s.edge[t]} get the same physical target {@link DFAState}, or // {@code null}. Once into the DFA, the DFA simulation does not reference the // {@link DFA//states} map. It follows the {@link DFAState//edges} field to new // targets. The DFA simulator will either find {@link DFAState//edges} to be // {@code null}, to be non-{@code null} and {@code dfa.edges[t]} null, or // {@code dfa.edges[t]} to be non-null. The // {@link //addDFAEdge} method could be racing to set the field // but in either case the DFA simulator works; if {@code null}, and requests ATN // simulation. It could also race trying to get {@code dfa.edges[t]}, but either // way it will work because it's not doing a test and set operation.

// //

// Starting with SLL then failing to combined SLL/LL (Two-Stage // Parsing)

// //

// Sam pointed out that if SLL does not give a syntax error, then there is no // point in doing full LL, which is slower. We only have to try LL if we get a // syntax error. For maximum speed, Sam starts the parser set to pure SLL // mode with the {@link BailErrorStrategy}:

// //
// parser.{@link Parser//getInterpreter() getInterpreter()}.{@link //setPredictionMode setPredictionMode}{@code (}{@link PredictionMode//SLL}{@code )};
// parser.{@link Parser//setErrorHandler setErrorHandler}(new {@link BailErrorStrategy}());
// 
// //

// If it does not get a syntax error, then we're done. If it does get a syntax // error, we need to retry with the combined SLL/LL strategy.

// //

// The reason this works is as follows. If there are no SLL conflicts, then the // grammar is SLL (at least for that input set). If there is an SLL conflict, // the full LL analysis must yield a set of viable alternatives which is a // subset of the alternatives reported by SLL. If the LL set is a singleton, // then the grammar is LL but not SLL. If the LL set is the same size as the SLL // set, the decision is SLL. If the LL set has size > 1, then that decision // is truly ambiguous on the current input. If the LL set is smaller, then the // SLL conflict resolution might choose an alternative that the full LL would // rule out as a possibility based upon better context information. If that's // the case, then the SLL parse will definitely get an error because the full LL // analysis says it's not viable. If SLL conflict resolution chooses an // alternative within the LL set, them both SLL and LL would choose the same // alternative because they both choose the minimum of multiple conflicting // alternatives.

// //

// Let's say we have a set of SLL conflicting alternatives {@code {1, 2, 3}} and // a smaller LL set called s. If s is {@code {2, 3}}, then SLL // parsing will get an error because SLL will pursue alternative 1. If // s is {@code {1, 2}} or {@code {1, 3}} then both SLL and LL will // choose the same alternative because alternative one is the minimum of either // set. If s is {@code {2}} or {@code {3}} then SLL will get a syntax // error. If s is {@code {1}} then SLL will succeed.

// //

// Of course, if the input is invalid, then we will get an error for sure in // both SLL and LL parsing. Erroneous input will therefore require 2 passes over // the input.

// var Utils = require('./../Utils'); var Set = Utils.Set; var BitSet = Utils.BitSet; var DoubleDict = Utils.DoubleDict; var ATN = require('./ATN').ATN; var ATNConfig = require('./ATNConfig').ATNConfig; var ATNConfigSet = require('./ATNConfigSet').ATNConfigSet; var Token = require('./../Token').Token; var DFAState = require('./../dfa/DFAState').DFAState; var PredPrediction = require('./../dfa/DFAState').PredPrediction; var ATNSimulator = require('./ATNSimulator').ATNSimulator; var PredictionMode = require('./PredictionMode').PredictionMode; var RuleContext = require('./../RuleContext').RuleContext; var ParserRuleContext = require('./../ParserRuleContext').ParserRuleContext; var SemanticContext = require('./SemanticContext').SemanticContext; var StarLoopEntryState = require('./ATNState').StarLoopEntryState; var RuleStopState = require('./ATNState').RuleStopState; var PredictionContext = require('./../PredictionContext').PredictionContext; var Interval = require('./../IntervalSet').Interval; var Transitions = require('./Transition'); var Transition = Transitions.Transition; var SetTransition = Transitions.SetTransition; var NotSetTransition = Transitions.NotSetTransition; var RuleTransition = Transitions.RuleTransition; var ActionTransition = Transitions.ActionTransition; var NoViableAltException = require('./../error/Errors').NoViableAltException; var SingletonPredictionContext = require('./../PredictionContext').SingletonPredictionContext; var predictionContextFromRuleContext = require('./../PredictionContext').predictionContextFromRuleContext; function ParserATNSimulator(parser, atn, decisionToDFA, sharedContextCache) { ATNSimulator.call(this, atn, sharedContextCache); this.parser = parser; this.decisionToDFA = decisionToDFA; // SLL, LL, or LL + exact ambig detection?// this.predictionMode = PredictionMode.LL; // LAME globals to avoid parameters!!!!! I need these down deep in predTransition this._input = null; this._startIndex = 0; this._outerContext = null; this._dfa = null; // Each prediction operation uses a cache for merge of prediction contexts. // Don't keep around as it wastes huge amounts of memory. DoubleKeyMap // isn't synchronized but we're ok since two threads shouldn't reuse same // parser/atnsim object because it can only handle one input at a time. // This maps graphs a and b to merged result c. (a,b)→c. We can avoid // the merge if we ever see a and b again. Note that (b,a)→c should // also be examined during cache lookup. // this.mergeCache = null; return this; } ParserATNSimulator.prototype = Object.create(ATNSimulator.prototype); ParserATNSimulator.prototype.constructor = ParserATNSimulator; ParserATNSimulator.prototype.debug = false; ParserATNSimulator.prototype.debug_list_atn_decisions = false; ParserATNSimulator.prototype.dfa_debug = false; ParserATNSimulator.prototype.retry_debug = false; ParserATNSimulator.prototype.reset = function() { }; ParserATNSimulator.prototype.adaptivePredict = function(input, decision, outerContext) { if (this.debug || this.debug_list_atn_decisions) { console.log("adaptivePredict decision " + decision + " exec LA(1)==" + this.getLookaheadName(input) + " line " + input.LT(1).line + ":" + input.LT(1).column); } this._input = input; this._startIndex = input.index; this._outerContext = outerContext; var dfa = this.decisionToDFA[decision]; this._dfa = dfa; var m = input.mark(); var index = input.index; // Now we are certain to have a specific decision's DFA // But, do we still need an initial state? try { var s0; if (dfa.precedenceDfa) { // the start state for a precedence DFA depends on the current // parser precedence, and is provided by a DFA method. s0 = dfa.getPrecedenceStartState(this.parser.getPrecedence()); } else { // the start state for a "regular" DFA is just s0 s0 = dfa.s0; } if (s0===null) { if (outerContext===null) { outerContext = RuleContext.EMPTY; } if (this.debug || this.debug_list_atn_decisions) { console.log("predictATN decision " + dfa.decision + " exec LA(1)==" + this.getLookaheadName(input) + ", outerContext=" + outerContext.toString(this.parser.ruleNames)); } // If this is not a precedence DFA, we check the ATN start state // to determine if this ATN start state is the decision for the // closure block that determines whether a precedence rule // should continue or complete. // if (!dfa.precedenceDfa && (dfa.atnStartState instanceof StarLoopEntryState)) { if (dfa.atnStartState.precedenceRuleDecision) { dfa.setPrecedenceDfa(true); } } var fullCtx = false; var s0_closure = this.computeStartState(dfa.atnStartState, RuleContext.EMPTY, fullCtx); if( dfa.precedenceDfa) { // If this is a precedence DFA, we use applyPrecedenceFilter // to convert the computed start state to a precedence start // state. We then use DFA.setPrecedenceStartState to set the // appropriate start state for the precedence level rather // than simply setting DFA.s0. // s0_closure = this.applyPrecedenceFilter(s0_closure); s0 = this.addDFAState(dfa, new DFAState(null, s0_closure)); dfa.setPrecedenceStartState(this.parser.getPrecedence(), s0); } else { s0 = this.addDFAState(dfa, new DFAState(null, s0_closure)); dfa.s0 = s0; } } var alt = this.execATN(dfa, s0, input, index, outerContext); if (this.debug) { console.log("DFA after predictATN: " + dfa.toString(this.parser.literalNames)); } return alt; } finally { this._dfa = null; this.mergeCache = null; // wack cache after each prediction input.seek(index); input.release(m); } }; // Performs ATN simulation to compute a predicted alternative based // upon the remaining input, but also updates the DFA cache to avoid // having to traverse the ATN again for the same input sequence. // There are some key conditions we're looking for after computing a new // set of ATN configs (proposed DFA state): // if the set is empty, there is no viable alternative for current symbol // does the state uniquely predict an alternative? // does the state have a conflict that would prevent us from // putting it on the work list? // We also have some key operations to do: // add an edge from previous DFA state to potentially new DFA state, D, // upon current symbol but only if adding to work list, which means in all // cases except no viable alternative (and possibly non-greedy decisions?) // collecting predicates and adding semantic context to DFA accept states // adding rule context to context-sensitive DFA accept states // consuming an input symbol // reporting a conflict // reporting an ambiguity // reporting a context sensitivity // reporting insufficient predicates // cover these cases: // dead end // single alt // single alt + preds // conflict // conflict + preds // ParserATNSimulator.prototype.execATN = function(dfa, s0, input, startIndex, outerContext ) { if (this.debug || this.debug_list_atn_decisions) { console.log("execATN decision " + dfa.decision + " exec LA(1)==" + this.getLookaheadName(input) + " line " + input.LT(1).line + ":" + input.LT(1).column); } var alt; var previousD = s0; if (this.debug) { console.log("s0 = " + s0); } var t = input.LA(1); while(true) { // while more work var D = this.getExistingTargetState(previousD, t); if(D===null) { D = this.computeTargetState(dfa, previousD, t); } if(D===ATNSimulator.ERROR) { // if any configs in previous dipped into outer context, that // means that input up to t actually finished entry rule // at least for SLL decision. Full LL doesn't dip into outer // so don't need special case. // We will get an error no matter what so delay until after // decision; better error message. Also, no reachable target // ATN states in SLL implies LL will also get nowhere. // If conflict in states that dip out, choose min since we // will get error no matter what. var e = this.noViableAlt(input, outerContext, previousD.configs, startIndex); input.seek(startIndex); alt = this.getSynValidOrSemInvalidAltThatFinishedDecisionEntryRule(previousD.configs, outerContext); if(alt!==ATN.INVALID_ALT_NUMBER) { return alt; } else { throw e; } } if(D.requiresFullContext && this.predictionMode !== PredictionMode.SLL) { // IF PREDS, MIGHT RESOLVE TO SINGLE ALT => SLL (or syntax error) var conflictingAlts = null; if (D.predicates!==null) { if (this.debug) { console.log("DFA state has preds in DFA sim LL failover"); } var conflictIndex = input.index; if(conflictIndex !== startIndex) { input.seek(startIndex); } conflictingAlts = this.evalSemanticContext(D.predicates, outerContext, true); if (conflictingAlts.length===1) { if(this.debug) { console.log("Full LL avoided"); } return conflictingAlts.minValue(); } if (conflictIndex !== startIndex) { // restore the index so reporting the fallback to full // context occurs with the index at the correct spot input.seek(conflictIndex); } } if (this.dfa_debug) { console.log("ctx sensitive state " + outerContext +" in " + D); } var fullCtx = true; var s0_closure = this.computeStartState(dfa.atnStartState, outerContext, fullCtx); this.reportAttemptingFullContext(dfa, conflictingAlts, D.configs, startIndex, input.index); alt = this.execATNWithFullContext(dfa, D, s0_closure, input, startIndex, outerContext); return alt; } if (D.isAcceptState) { if (D.predicates===null) { return D.prediction; } var stopIndex = input.index; input.seek(startIndex); var alts = this.evalSemanticContext(D.predicates, outerContext, true); if (alts.length===0) { throw this.noViableAlt(input, outerContext, D.configs, startIndex); } else if (alts.length===1) { return alts.minValue(); } else { // report ambiguity after predicate evaluation to make sure the correct set of ambig alts is reported. this.reportAmbiguity(dfa, D, startIndex, stopIndex, false, alts, D.configs); return alts.minValue(); } } previousD = D; if (t !== Token.EOF) { input.consume(); t = input.LA(1); } } }; // // Get an existing target state for an edge in the DFA. If the target state // for the edge has not yet been computed or is otherwise not available, // this method returns {@code null}. // // @param previousD The current DFA state // @param t The next input symbol // @return The existing target DFA state for the given input symbol // {@code t}, or {@code null} if the target state for this edge is not // already cached // ParserATNSimulator.prototype.getExistingTargetState = function(previousD, t) { var edges = previousD.edges; if (edges===null) { return null; } else { return edges[t + 1] || null; } }; // // Compute a target state for an edge in the DFA, and attempt to add the // computed state and corresponding edge to the DFA. // // @param dfa The DFA // @param previousD The current DFA state // @param t The next input symbol // // @return The computed target DFA state for the given input symbol // {@code t}. If {@code t} does not lead to a valid DFA state, this method // returns {@link //ERROR}. // ParserATNSimulator.prototype.computeTargetState = function(dfa, previousD, t) { var reach = this.computeReachSet(previousD.configs, t, false); if(reach===null) { this.addDFAEdge(dfa, previousD, t, ATNSimulator.ERROR); return ATNSimulator.ERROR; } // create new target state; we'll add to DFA after it's complete var D = new DFAState(null, reach); var predictedAlt = this.getUniqueAlt(reach); if (this.debug) { var altSubSets = PredictionMode.getConflictingAltSubsets(reach); console.log("SLL altSubSets=" + Utils.arrayToString(altSubSets) + ", previous=" + previousD.configs + ", configs=" + reach + ", predict=" + predictedAlt + ", allSubsetsConflict=" + PredictionMode.allSubsetsConflict(altSubSets) + ", conflictingAlts=" + this.getConflictingAlts(reach)); } if (predictedAlt!==ATN.INVALID_ALT_NUMBER) { // NO CONFLICT, UNIQUELY PREDICTED ALT D.isAcceptState = true; D.configs.uniqueAlt = predictedAlt; D.prediction = predictedAlt; } else if (PredictionMode.hasSLLConflictTerminatingPrediction(this.predictionMode, reach)) { // MORE THAN ONE VIABLE ALTERNATIVE D.configs.conflictingAlts = this.getConflictingAlts(reach); D.requiresFullContext = true; // in SLL-only mode, we will stop at this state and return the minimum alt D.isAcceptState = true; D.prediction = D.configs.conflictingAlts.minValue(); } if (D.isAcceptState && D.configs.hasSemanticContext) { this.predicateDFAState(D, this.atn.getDecisionState(dfa.decision)); if( D.predicates!==null) { D.prediction = ATN.INVALID_ALT_NUMBER; } } // all adds to dfa are done after we've created full D state D = this.addDFAEdge(dfa, previousD, t, D); return D; }; ParserATNSimulator.prototype.predicateDFAState = function(dfaState, decisionState) { // We need to test all predicates, even in DFA states that // uniquely predict alternative. var nalts = decisionState.transitions.length; // Update DFA so reach becomes accept state with (predicate,alt) // pairs if preds found for conflicting alts var altsToCollectPredsFrom = this.getConflictingAltsOrUniqueAlt(dfaState.configs); var altToPred = this.getPredsForAmbigAlts(altsToCollectPredsFrom, dfaState.configs, nalts); if (altToPred!==null) { dfaState.predicates = this.getPredicatePredictions(altsToCollectPredsFrom, altToPred); dfaState.prediction = ATN.INVALID_ALT_NUMBER; // make sure we use preds } else { // There are preds in configs but they might go away // when OR'd together like {p}? || NONE == NONE. If neither // alt has preds, resolve to min alt dfaState.prediction = altsToCollectPredsFrom.minValue(); } }; // comes back with reach.uniqueAlt set to a valid alt ParserATNSimulator.prototype.execATNWithFullContext = function(dfa, D, // how far we got before failing over s0, input, startIndex, outerContext) { if (this.debug || this.debug_list_atn_decisions) { console.log("execATNWithFullContext "+s0); } var fullCtx = true; var foundExactAmbig = false; var reach = null; var previous = s0; input.seek(startIndex); var t = input.LA(1); var predictedAlt = -1; while (true) { // while more work reach = this.computeReachSet(previous, t, fullCtx); if (reach===null) { // if any configs in previous dipped into outer context, that // means that input up to t actually finished entry rule // at least for LL decision. Full LL doesn't dip into outer // so don't need special case. // We will get an error no matter what so delay until after // decision; better error message. Also, no reachable target // ATN states in SLL implies LL will also get nowhere. // If conflict in states that dip out, choose min since we // will get error no matter what. var e = this.noViableAlt(input, outerContext, previous, startIndex); input.seek(startIndex); var alt = this.getSynValidOrSemInvalidAltThatFinishedDecisionEntryRule(previous, outerContext); if(alt!==ATN.INVALID_ALT_NUMBER) { return alt; } else { throw e; } } var altSubSets = PredictionMode.getConflictingAltSubsets(reach); if(this.debug) { console.log("LL altSubSets=" + altSubSets + ", predict=" + PredictionMode.getUniqueAlt(altSubSets) + ", resolvesToJustOneViableAlt=" + PredictionMode.resolvesToJustOneViableAlt(altSubSets)); } reach.uniqueAlt = this.getUniqueAlt(reach); // unique prediction? if(reach.uniqueAlt!==ATN.INVALID_ALT_NUMBER) { predictedAlt = reach.uniqueAlt; break; } else if (this.predictionMode !== PredictionMode.LL_EXACT_AMBIG_DETECTION) { predictedAlt = PredictionMode.resolvesToJustOneViableAlt(altSubSets); if(predictedAlt !== ATN.INVALID_ALT_NUMBER) { break; } } else { // In exact ambiguity mode, we never try to terminate early. // Just keeps scarfing until we know what the conflict is if (PredictionMode.allSubsetsConflict(altSubSets) && PredictionMode.allSubsetsEqual(altSubSets)) { foundExactAmbig = true; predictedAlt = PredictionMode.getSingleViableAlt(altSubSets); break; } // else there are multiple non-conflicting subsets or // we're not sure what the ambiguity is yet. // So, keep going. } previous = reach; if( t !== Token.EOF) { input.consume(); t = input.LA(1); } } // If the configuration set uniquely predicts an alternative, // without conflict, then we know that it's a full LL decision // not SLL. if (reach.uniqueAlt !== ATN.INVALID_ALT_NUMBER ) { this.reportContextSensitivity(dfa, predictedAlt, reach, startIndex, input.index); return predictedAlt; } // We do not check predicates here because we have checked them // on-the-fly when doing full context prediction. // // In non-exact ambiguity detection mode, we might actually be able to // detect an exact ambiguity, but I'm not going to spend the cycles // needed to check. We only emit ambiguity warnings in exact ambiguity // mode. // // For example, we might know that we have conflicting configurations. // But, that does not mean that there is no way forward without a // conflict. It's possible to have nonconflicting alt subsets as in: // altSubSets=[{1, 2}, {1, 2}, {1}, {1, 2}] // from // // [(17,1,[5 $]), (13,1,[5 10 $]), (21,1,[5 10 $]), (11,1,[$]), // (13,2,[5 10 $]), (21,2,[5 10 $]), (11,2,[$])] // // In this case, (17,1,[5 $]) indicates there is some next sequence that // would resolve this without conflict to alternative 1. Any other viable // next sequence, however, is associated with a conflict. We stop // looking for input because no amount of further lookahead will alter // the fact that we should predict alternative 1. We just can't say for // sure that there is an ambiguity without looking further. this.reportAmbiguity(dfa, D, startIndex, input.index, foundExactAmbig, null, reach); return predictedAlt; }; ParserATNSimulator.prototype.computeReachSet = function(closure, t, fullCtx) { if (this.debug) { console.log("in computeReachSet, starting closure: " + closure); } if( this.mergeCache===null) { this.mergeCache = new DoubleDict(); } var intermediate = new ATNConfigSet(fullCtx); // Configurations already in a rule stop state indicate reaching the end // of the decision rule (local context) or end of the start rule (full // context). Once reached, these configurations are never updated by a // closure operation, so they are handled separately for the performance // advantage of having a smaller intermediate set when calling closure. // // For full-context reach operations, separate handling is required to // ensure that the alternative matching the longest overall sequence is // chosen when multiple such configurations can match the input. var skippedStopStates = null; // First figure out where we can reach on input t for (var i=0; iWhen {@code lookToEndOfRule} is true, this method uses // {@link ATN//nextTokens} for each configuration in {@code configs} which is // not already in a rule stop state to see if a rule stop state is reachable // from the configuration via epsilon-only transitions.

// // @param configs the configuration set to update // @param lookToEndOfRule when true, this method checks for rule stop states // reachable by epsilon-only transitions from each configuration in // {@code configs}. // // @return {@code configs} if all configurations in {@code configs} are in a // rule stop state, otherwise return a new configuration set containing only // the configurations from {@code configs} which are in a rule stop state // ParserATNSimulator.prototype.removeAllConfigsNotInRuleStopState = function(configs, lookToEndOfRule) { if (PredictionMode.allConfigsInRuleStopStates(configs)) { return configs; } var result = new ATNConfigSet(configs.fullCtx); for(var i=0; i //
  • Evaluate the precedence predicates for each configuration using // {@link SemanticContext//evalPrecedence}.
  • //
  • Remove all configurations which predict an alternative greater than // 1, for which another configuration that predicts alternative 1 is in the // same ATN state with the same prediction context. This transformation is // valid for the following reasons: //
      //
    • The closure block cannot contain any epsilon transitions which bypass // the body of the closure, so all states reachable via alternative 1 are // part of the precedence alternatives of the transformed left-recursive // rule.
    • //
    • The "primary" portion of a left recursive rule cannot contain an // epsilon transition, so the only way an alternative other than 1 can exist // in a state that is also reachable via alternative 1 is by nesting calls // to the left-recursive rule, with the outer calls not being at the // preferred precedence level.
    • //
    //
  • // // //

    // The prediction context must be considered by this filter to address // situations like the following. //

    // //
    // grammar TA;
    // prog: statement* EOF;
    // statement: letterA | statement letterA 'b' ;
    // letterA: 'a';
    // 
    //
    //

    // If the above grammar, the ATN state immediately before the token // reference {@code 'a'} in {@code letterA} is reachable from the left edge // of both the primary and closure blocks of the left-recursive rule // {@code statement}. The prediction context associated with each of these // configurations distinguishes between them, and prevents the alternative // which stepped out to {@code prog} (and then back in to {@code statement} // from being eliminated by the filter. //

    // // @param configs The configuration set computed by // {@link //computeStartState} as the start state for the DFA. // @return The transformed configuration set representing the start state // for a precedence DFA at a particular precedence level (determined by // calling {@link Parser//getPrecedence}). // ParserATNSimulator.prototype.applyPrecedenceFilter = function(configs) { var config; var statesFromAlt1 = []; var configSet = new ATNConfigSet(configs.fullCtx); for(var i=0; i1 // (basically a graph subtraction algorithm). if (!config.precedenceFilterSuppressed) { var context = statesFromAlt1[config.state.stateNumber] || null; if (context!==null && context.equals(config.context)) { // eliminated continue; } } configSet.add(config, this.mergeCache); } return configSet; }; ParserATNSimulator.prototype.getReachableTarget = function(trans, ttype) { if (trans.matches(ttype, 0, this.atn.maxTokenType)) { return trans.target; } else { return null; } }; ParserATNSimulator.prototype.getPredsForAmbigAlts = function(ambigAlts, configs, nalts) { // REACH=[1|1|[]|0:0, 1|2|[]|0:1] // altToPred starts as an array of all null contexts. The entry at index i // corresponds to alternative i. altToPred[i] may have one of three values: // 1. null: no ATNConfig c is found such that c.alt==i // 2. SemanticContext.NONE: At least one ATNConfig c exists such that // c.alt==i and c.semanticContext==SemanticContext.NONE. In other words, // alt i has at least one unpredicated config. // 3. Non-NONE Semantic Context: There exists at least one, and for all // ATNConfig c such that c.alt==i, c.semanticContext!=SemanticContext.NONE. // // From this, it is clear that NONE||anything==NONE. // var altToPred = []; for(var i=0;i // The default implementation of this method uses the following // algorithm to identify an ATN configuration which successfully parsed the // decision entry rule. Choosing such an alternative ensures that the // {@link ParserRuleContext} returned by the calling rule will be complete // and valid, and the syntax error will be reported later at a more // localized location.

    // //
      //
    • If a syntactically valid path or paths reach the end of the decision rule and // they are semantically valid if predicated, return the min associated alt.
    • //
    • Else, if a semantically invalid but syntactically valid path exist // or paths exist, return the minimum associated alt. //
    • //
    • Otherwise, return {@link ATN//INVALID_ALT_NUMBER}.
    • //
    // //

    // In some scenarios, the algorithm described above could predict an // alternative which will result in a {@link FailedPredicateException} in // the parser. Specifically, this could occur if the only configuration // capable of successfully parsing to the end of the decision rule is // blocked by a semantic predicate. By choosing this alternative within // {@link //adaptivePredict} instead of throwing a // {@link NoViableAltException}, the resulting // {@link FailedPredicateException} in the parser will identify the specific // predicate which is preventing the parser from successfully parsing the // decision rule, which helps developers identify and correct logic errors // in semantic predicates. //

    // // @param configs The ATN configurations which were valid immediately before // the {@link //ERROR} state was reached // @param outerContext The is the \gamma_0 initial parser context from the paper // or the parser stack at the instant before prediction commences. // // @return The value to return from {@link //adaptivePredict}, or // {@link ATN//INVALID_ALT_NUMBER} if a suitable alternative was not // identified and {@link //adaptivePredict} should report an error instead. // ParserATNSimulator.prototype.getSynValidOrSemInvalidAltThatFinishedDecisionEntryRule = function(configs, outerContext) { var cfgs = this.splitAccordingToSemanticValidity(configs, outerContext); var semValidConfigs = cfgs[0]; var semInvalidConfigs = cfgs[1]; var alt = this.getAltThatFinishedDecisionEntryRule(semValidConfigs); if (alt!==ATN.INVALID_ALT_NUMBER) { // semantically/syntactically viable path exists return alt; } // Is there a syntactically valid path with a failed pred? if (semInvalidConfigs.items.length>0) { alt = this.getAltThatFinishedDecisionEntryRule(semInvalidConfigs); if (alt!==ATN.INVALID_ALT_NUMBER) { // syntactically viable path exists return alt; } } return ATN.INVALID_ALT_NUMBER; }; ParserATNSimulator.prototype.getAltThatFinishedDecisionEntryRule = function(configs) { var alts = []; for(var i=0;i0 || ((c.state instanceof RuleStopState) && c.context.hasEmptyPath())) { if(alts.indexOf(c.alt)<0) { alts.push(c.alt); } } } if (alts.length===0) { return ATN.INVALID_ALT_NUMBER; } else { return Math.min.apply(null, alts); } }; // Walk the list of configurations and split them according to // those that have preds evaluating to true/false. If no pred, assume // true pred and include in succeeded set. Returns Pair of sets. // // Create a new set so as not to alter the incoming parameter. // // Assumption: the input stream has been restored to the starting point // prediction, which is where predicates need to evaluate. // ParserATNSimulator.prototype.splitAccordingToSemanticValidity = function( configs, outerContext) { var succeeded = new ATNConfigSet(configs.fullCtx); var failed = new ATNConfigSet(configs.fullCtx); for(var i=0;i50) { throw "problem"; } } if (config.state instanceof RuleStopState) { // We hit rule end. If we have context info, use it // run thru all possible stack tops in ctx if (! config.context.isEmpty()) { for ( var i =0; i 0. if (closureBusy.add(c)!==c) { // avoid infinite recursion for right-recursive rules continue; } if (this._dfa !== null && this._dfa.precedenceDfa) { if (t.outermostPrecedenceReturn === this._dfa.atnStartState.ruleIndex) { c.precedenceFilterSuppressed = true; } } c.reachesIntoOuterContext += 1; configs.dipsIntoOuterContext = true; // TODO: can remove? only care when we add to set per middle of this method newDepth -= 1; if (this.debug) { console.log("dips into outer ctx: " + c); } } else if (t instanceof RuleTransition) { // latch when newDepth goes negative - once we step out of the entry context we can't return if (newDepth >= 0) { newDepth += 1; } } this.closureCheckingStopState(c, configs, closureBusy, continueCollecting, fullCtx, newDepth, treatEofAsEpsilon); } } }; ParserATNSimulator.prototype.getRuleName = function( index) { if (this.parser!==null && index>=0) { return this.parser.ruleNames[index]; } else { return ""; } }; ParserATNSimulator.prototype.getEpsilonTarget = function(config, t, collectPredicates, inContext, fullCtx, treatEofAsEpsilon) { switch(t.serializationType) { case Transition.RULE: return this.ruleTransition(config, t); case Transition.PRECEDENCE: return this.precedenceTransition(config, t, collectPredicates, inContext, fullCtx); case Transition.PREDICATE: return this.predTransition(config, t, collectPredicates, inContext, fullCtx); case Transition.ACTION: return this.actionTransition(config, t); case Transition.EPSILON: return new ATNConfig({state:t.target}, config); case Transition.ATOM: case Transition.RANGE: case Transition.SET: // EOF transitions act like epsilon transitions after the first EOF // transition is traversed if (treatEofAsEpsilon) { if (t.matches(Token.EOF, 0, 1)) { return new ATNConfig({state: t.target}, config); } } return null; default: return null; } }; ParserATNSimulator.prototype.actionTransition = function(config, t) { if (this.debug) { console.log("ACTION edge " + t.ruleIndex + ":" + t.actionIndex); } return new ATNConfig({state:t.target}, config); }; ParserATNSimulator.prototype.precedenceTransition = function(config, pt, collectPredicates, inContext, fullCtx) { if (this.debug) { console.log("PRED (collectPredicates=" + collectPredicates + ") " + pt.precedence + ">=_p, ctx dependent=true"); if (this.parser!==null) { console.log("context surrounding pred is " + Utils.arrayToString(this.parser.getRuleInvocationStack())); } } var c = null; if (collectPredicates && inContext) { if (fullCtx) { // In full context mode, we can evaluate predicates on-the-fly // during closure, which dramatically reduces the size of // the config sets. It also obviates the need to test predicates // later during conflict resolution. var currentPosition = this._input.index; this._input.seek(this._startIndex); var predSucceeds = pt.getPredicate().evaluate(this.parser, this._outerContext); this._input.seek(currentPosition); if (predSucceeds) { c = new ATNConfig({state:pt.target}, config); // no pred context } } else { newSemCtx = SemanticContext.andContext(config.semanticContext, pt.getPredicate()); c = new ATNConfig({state:pt.target, semanticContext:newSemCtx}, config); } } else { c = new ATNConfig({state:pt.target}, config); } if (this.debug) { console.log("config from pred transition=" + c); } return c; }; ParserATNSimulator.prototype.predTransition = function(config, pt, collectPredicates, inContext, fullCtx) { if (this.debug) { console.log("PRED (collectPredicates=" + collectPredicates + ") " + pt.ruleIndex + ":" + pt.predIndex + ", ctx dependent=" + pt.isCtxDependent); if (this.parser!==null) { console.log("context surrounding pred is " + Utils.arrayToString(this.parser.getRuleInvocationStack())); } } var c = null; if (collectPredicates && ((pt.isCtxDependent && inContext) || ! pt.isCtxDependent)) { if (fullCtx) { // In full context mode, we can evaluate predicates on-the-fly // during closure, which dramatically reduces the size of // the config sets. It also obviates the need to test predicates // later during conflict resolution. var currentPosition = this._input.index; this._input.seek(this._startIndex); var predSucceeds = pt.getPredicate().evaluate(this.parser, this._outerContext); this._input.seek(currentPosition); if (predSucceeds) { c = new ATNConfig({state:pt.target}, config); // no pred context } } else { var newSemCtx = SemanticContext.andContext(config.semanticContext, pt.getPredicate()); c = new ATNConfig({state:pt.target, semanticContext:newSemCtx}, config); } } else { c = new ATNConfig({state:pt.target}, config); } if (this.debug) { console.log("config from pred transition=" + c); } return c; }; ParserATNSimulator.prototype.ruleTransition = function(config, t) { if (this.debug) { console.log("CALL rule " + this.getRuleName(t.target.ruleIndex) + ", ctx=" + config.context); } var returnState = t.followState; var newContext = SingletonPredictionContext.create(config.context, returnState.stateNumber); return new ATNConfig({state:t.target, context:newContext}, config ); }; ParserATNSimulator.prototype.getConflictingAlts = function(configs) { var altsets = PredictionMode.getConflictingAltSubsets(configs); return PredictionMode.getAlts(altsets); }; // Sam pointed out a problem with the previous definition, v3, of // ambiguous states. If we have another state associated with conflicting // alternatives, we should keep going. For example, the following grammar // // s : (ID | ID ID?) ';' ; // // When the ATN simulation reaches the state before ';', it has a DFA // state that looks like: [12|1|[], 6|2|[], 12|2|[]]. Naturally // 12|1|[] and 12|2|[] conflict, but we cannot stop processing this node // because alternative to has another way to continue, via [6|2|[]]. // The key is that we have a single state that has config's only associated // with a single alternative, 2, and crucially the state transitions // among the configurations are all non-epsilon transitions. That means // we don't consider any conflicts that include alternative 2. So, we // ignore the conflict between alts 1 and 2. We ignore a set of // conflicting alts when there is an intersection with an alternative // associated with a single alt state in the state→config-list map. // // It's also the case that we might have two conflicting configurations but // also a 3rd nonconflicting configuration for a different alternative: // [1|1|[], 1|2|[], 8|3|[]]. This can come about from grammar: // // a : A | A | A B ; // // After matching input A, we reach the stop state for rule A, state 1. // State 8 is the state right before B. Clearly alternatives 1 and 2 // conflict and no amount of further lookahead will separate the two. // However, alternative 3 will be able to continue and so we do not // stop working on this state. In the previous example, we're concerned // with states associated with the conflicting alternatives. Here alt // 3 is not associated with the conflicting configs, but since we can continue // looking for input reasonably, I don't declare the state done. We // ignore a set of conflicting alts when we have an alternative // that we still need to pursue. // ParserATNSimulator.prototype.getConflictingAltsOrUniqueAlt = function(configs) { var conflictingAlts = null; if (configs.uniqueAlt!== ATN.INVALID_ALT_NUMBER) { conflictingAlts = new BitSet(); conflictingAlts.add(configs.uniqueAlt); } else { conflictingAlts = configs.conflictingAlts; } return conflictingAlts; }; ParserATNSimulator.prototype.getTokenName = function( t) { if (t===Token.EOF) { return "EOF"; } if( this.parser!==null && this.parser.literalNames!==null) { if (t >= this.parser.literalNames.length) { console.log("" + t + " ttype out of range: " + this.parser.literalNames); console.log("" + this.parser.getInputStream().getTokens()); } else { return this.parser.literalNames[t] + "<" + t + ">"; } } return "" + t; }; ParserATNSimulator.prototype.getLookaheadName = function(input) { return this.getTokenName(input.LA(1)); }; // Used for debugging in adaptivePredict around execATN but I cut // it out for clarity now that alg. works well. We can leave this // "dead" code for a bit. // ParserATNSimulator.prototype.dumpDeadEndConfigs = function(nvae) { console.log("dead end configs: "); var decs = nvae.getDeadEndConfigs(); for(var i=0; i0) { var t = c.state.transitions[0]; if (t instanceof AtomTransition) { trans = "Atom "+ this.getTokenName(t.label); } else if (t instanceof SetTransition) { var neg = (t instanceof NotSetTransition); trans = (neg ? "~" : "") + "Set " + t.set; } } console.error(c.toString(this.parser, true) + ":" + trans); } }; ParserATNSimulator.prototype.noViableAlt = function(input, outerContext, configs, startIndex) { return new NoViableAltException(this.parser, input, input.get(startIndex), input.LT(1), configs, outerContext); }; ParserATNSimulator.prototype.getUniqueAlt = function(configs) { var alt = ATN.INVALID_ALT_NUMBER; for(var i=0;iIf {@code to} is {@code null}, this method returns {@code null}. // Otherwise, this method returns the {@link DFAState} returned by calling // {@link //addDFAState} for the {@code to} state.

    // // @param dfa The DFA // @param from The source state for the edge // @param t The input symbol // @param to The target state for the edge // // @return If {@code to} is {@code null}, this method returns {@code null}; // otherwise this method returns the result of calling {@link //addDFAState} // on {@code to} // ParserATNSimulator.prototype.addDFAEdge = function(dfa, from_, t, to) { if( this.debug) { console.log("EDGE " + from_ + " -> " + to + " upon " + this.getTokenName(t)); } if (to===null) { return null; } to = this.addDFAState(dfa, to); // used existing if possible not incoming if (from_===null || t < -1 || t > this.atn.maxTokenType) { return to; } if (from_.edges===null) { from_.edges = []; } from_.edges[t+1] = to; // connect if (this.debug) { var names = this.parser===null ? null : this.parser.literalNames; console.log("DFA=\n" + dfa.toString(names)); } return to; }; // // Add state {@code D} to the DFA if it is not already present, and return // the actual instance stored in the DFA. If a state equivalent to {@code D} // is already in the DFA, the existing state is returned. Otherwise this // method returns {@code D} after adding it to the DFA. // //

    If {@code D} is {@link //ERROR}, this method returns {@link //ERROR} and // does not change the DFA.

    // // @param dfa The dfa // @param D The DFA state to add // @return The state stored in the DFA. This will be either the existing // state if {@code D} is already in the DFA, or {@code D} itself if the // state was not already present. // ParserATNSimulator.prototype.addDFAState = function(dfa, D) { if (D == ATNSimulator.ERROR) { return D; } var hash = D.hashString(); var existing = dfa.states[hash] || null; if(existing!==null) { return existing; } D.stateNumber = dfa.states.length; if (! D.configs.readOnly) { D.configs.optimizeConfigs(this); D.configs.setReadonly(true); } dfa.states[hash] = D; if (this.debug) { console.log("adding new DFA state: " + D); } return D; }; ParserATNSimulator.prototype.reportAttemptingFullContext = function(dfa, conflictingAlts, configs, startIndex, stopIndex) { if (this.debug || this.retry_debug) { var interval = new Interval(startIndex, stopIndex + 1); console.log("reportAttemptingFullContext decision=" + dfa.decision + ":" + configs + ", input=" + this.parser.getTokenStream().getText(interval)); } if (this.parser!==null) { this.parser.getErrorListenerDispatch().reportAttemptingFullContext(this.parser, dfa, startIndex, stopIndex, conflictingAlts, configs); } }; ParserATNSimulator.prototype.reportContextSensitivity = function(dfa, prediction, configs, startIndex, stopIndex) { if (this.debug || this.retry_debug) { var interval = new Interval(startIndex, stopIndex + 1); console.log("reportContextSensitivity decision=" + dfa.decision + ":" + configs + ", input=" + this.parser.getTokenStream().getText(interval)); } if (this.parser!==null) { this.parser.getErrorListenerDispatch().reportContextSensitivity(this.parser, dfa, startIndex, stopIndex, prediction, configs); } }; // If context sensitive parsing, we know it's ambiguity not conflict// ParserATNSimulator.prototype.reportAmbiguity = function(dfa, D, startIndex, stopIndex, exact, ambigAlts, configs ) { if (this.debug || this.retry_debug) { var interval = new Interval(startIndex, stopIndex + 1); console.log("reportAmbiguity " + ambigAlts + ":" + configs + ", input=" + this.parser.getTokenStream().getText(interval)); } if (this.parser!==null) { this.parser.getErrorListenerDispatch().reportAmbiguity(this.parser, dfa, startIndex, stopIndex, exact, ambigAlts, configs); } }; exports.ParserATNSimulator = ParserATNSimulator;




    © 2015 - 2025 Weber Informatics LLC | Privacy Policy