Tokens you get back include the delimiters. Generally, if its a comment, you know its delimited by /* */. Similarly, if its a quoted string, you know its delimited by ' or ". I think in the most common use cases, users will expect the tokens returned to be without the delimiters.
Right now the only way to do that is to do some additional parsing on the returned tokens or using substrings or some similar means. I think the best way to do this will be in the state objects (PKQuoteState, PKCommentState) etc. One way would be to have an includeDelimiter property, that when not set, does not include delimiters in the stringValue.
Tokens you get back include the delimiters. Generally, if its a comment, you know its delimited by /* */. Similarly, if its a quoted string, you know its delimited by ' or ". I think in the most common use cases, users will expect the tokens returned to be without the delimiters.
Right now the only way to do that is to do some additional parsing on the returned tokens or using substrings or some similar means. I think the best way to do this will be in the state objects (
PKQuoteState,PKCommentState) etc. One way would be to have anincludeDelimiterproperty, that when not set, does not include delimiters in the stringValue.