Skip to content

fix(rt): reserve stack slots in Table raw operations - #39

Merged
pjankiewicz merged 2 commits into
pjankiewicz:mainfrom
vi2q:fix/table-stack-checks
Aug 31, 2026
Merged

fix(rt): reserve stack slots in Table raw operations#39
pjankiewicz merged 2 commits into
pjankiewicz:mainfrom
vi2q:fix/table-stack-checks

Conversation

@vi2q

@vi2q vi2q commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Problem

Table::raw_set and Table::raw_get push up to three stack slots without calling lua_checkstack first:

self.reference.push();              // table
lua.push_value(&k)?;                // key
lua.push_value(&v)?;                // value (raw_set)
lua_rawset(state, -3);

value_from_stack also pushes one extra copy for reference types (Table / Function / etc.) inside raw_get. When the caller's current call-info frame margin is already thin, lua_pushvalue trips the (*L).top < (*(*L).ci).top LUAU_ASSERT and aborts the process (EXC_BREAKPOINT / SIGTRAP) instead of returning a Result::Err.

Real-world reproduction: loading a scene that compiles ~35 scripts in one VM, then probing the script environment with raw_get from the host, crashed with the assert. The host cannot defend itself — there is no public API to check the remaining frame margin, and the failing push is inside luaur-rt.

mlua is unaffected: its raw accessors run under ensure_stack / StackGuard.

Fix

Add an internal ensure_stack helper and reserve the needed slots up front:

pub(crate) fn ensure_stack(state: *mut lua_State, slots: c_int) -> Result<()> {
    if unsafe { lua_checkstack(state, slots) } == 0 {
        return Err(crate::error::Error::MemoryError(
            "stack overflow while preparing a table operation".to_string(),
        ));
    }
    Ok(())
}

raw_set reserves 3 slots; raw_get reserves 3 (table + key + the copy value_from_stack may push). A failed reservation surfaces as Error::MemoryError so embedders can recover.

The same pattern likely applies to other raw accessors (set_metatable, raw_insert, raw_pop, …); I started with the two that crashed in practice and kept the change reviewable. Happy to extend the helper across the remaining stack-touching APIs in this PR or a follow-up.

Testing

  • New test_raw_table_ops_survive_a_thin_stack_margin in mlua_core.rs: 2000 raw_set/raw_get pairs, a 150-deep VM recursion, then another 500 raw operations — all after interpreter work that drains the frame margin.
  • cargo test -p luaur-rt: 248 passed, 0 failed.

Found while embedding luaur-rt in a game engine (scene loading compiles every script into one VM; the probe step crashed before this fix).

raw_set/raw_get push up to three stack slots (plus the extra copy
value_from_stack adds for reference types) without calling lua_checkstack.
When the caller's call-info frame margin is already thin — for example
after a long sequence of VM operations, like compiling many scripts in
one VM — lua_pushvalue trips the top < ci.top LUAU_ASSERT and aborts the
process (EXC_BREAKPOINT) instead of returning an error.

Add an ensure_stack helper that reserves the needed slots up front and
returns a MemoryError when the reservation fails, and use it in raw_set
and raw_get.
@pjankiewicz

Copy link
Copy Markdown
Owner

@vi2q Thank you for all 3 PRs. I will review them and merge them.

…t claim

- Match the crate's existing convention (Function::call, Thread::resume,
  callback results): a stack reservation failure is a catchable
  RuntimeError("stack overflow: ..."), not a MemoryError.
- Rename/re-comment the new test: it exercises sustained raw traffic, but a
  C frame always gets LUA_MINSTACK headroom, so it does not by itself drive
  the margin low enough to trip the assertion.
@pjankiewicz

Copy link
Copy Markdown
Owner

Verified and pushed a small follow-up to this branch (2edef5f).

The guard itself is right and it's the convention the rest of the crate already follows — Function::call, Thread::resume/send, the callback result path and exec_raw all lua_checkstack before pushing; table.rs was simply the gap. lua_checkstackexpandstacklimit!(L, top + size) is exactly what raises ci->top, so the reasoning checks out against luaur's own VM sources.

Two changes:

  1. RuntimeError, not MemoryError. Every other reservation failure in the crate returns Error::RuntimeError("stack overflow: ..."); MemoryError would be a new shape for the same condition.
  2. Renamed/re-commented the test. I ran test_raw_table_ops_survive_a_thin_stack_margin against unpatched main and it passes — so it isn't gating the regression it names. That's expected: luaD_precall gives every C frame ci->top = L->top + LUA_MINSTACK (20 slots), and the raw ops need 3, so the margin is never thin by that route. I couldn't reproduce the abort from the host API at all, including from a callback taking 24 arguments. If you still have the scene-loading repro, a minimal version of it would be worth having — there may be a path here that doesn't get the LUA_MINSTACK headroom, and that path is the one actually worth fixing.

Merging it regardless: it's the correct C-API discipline, it's what mlua does via StackGuard, and it costs a comparison. Re: the other raw accessors — agreed, follow-up rather than growing this PR.

@pjankiewicz
pjankiewicz merged commit c782d8e into pjankiewicz:main Aug 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants