apply_mwe_pipeline

logicaffeine_language::mwe

Function apply_mwe_pipeline 

Source 
pub fn apply_mwe_pipeline(
    tokens: Vec<Token>,
    trie: &MweTrie,
    interner: &mut Interner,
) -> Vec<Token>
Expand description

Apply MWE collapsing to a token stream. Matches on lemmas (not raw strings) to handle morphological variants.