Skip to content

Add TextTape::from(Vec<TextToken>, bool) for Tape construction after token manipulation. - #181

Open
KitsuneDev wants to merge 1 commit into
rakaly:masterfrom
KitsuneDev:kitsune/raw_token
Open

Add TextTape::from(Vec<TextToken>, bool) for Tape construction after token manipulation.#181
KitsuneDev wants to merge 1 commit into
rakaly:masterfrom
KitsuneDev:kitsune/raw_token

Conversation

@KitsuneDev

Copy link
Copy Markdown

Currently, there is no way to reconstruct a Tape after performing raw token manipulation.
A use case (which is what I have in a project I'm working on) is parsing scripted_variables files and replacing their reference tokens on, say, technologies with their actual values.

@nickbabcock

nickbabcock commented Jun 12, 2025

Copy link
Copy Markdown
Contributor

Looks good, let me think about if it makes sense to supply the bom marker in the constructor, as that should only surface from initial parsing 🤔

@nickbabcock

Copy link
Copy Markdown
Contributor

A thought ocurred to me, is your use case covered by something like:

// pseudo-code
impl TextTape {
    pub fn tokens_mut(&mut self) -> &mut Vec<TextToken>
    ...
    }
}

A benefit is that you don't need to clone the tokens (that's how I'm assuming you are able to pass in a Vec<TextToken> to the constructor).

If that does work, it would be nice to include a rustdoc example and give the caveat that any modification needs to respect expected invariants or could otherwise risk object / array index corruption.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants