Skip to main content

tokenize

Function tokenize 

Source
pub fn tokenize(
    source: &str,
    module: &str,
) -> Result<Vec<Token>, Vec<Diagnostic>>
Expand description

Tokenizes source without discarding comments. Spans are UTF-8 byte offsets.