Skip to content

fix(rt): honor a custom __index meta-method on fieldless userdata - #38

Merged
pjankiewicz merged 2 commits into
pjankiewicz:mainfrom
vi2q:fix/fieldless-userdata-index
Aug 31, 2026
Merged

fix(rt): honor a custom __index meta-method on fieldless userdata#38
pjankiewicz merged 2 commits into
pjankiewicz:mainfrom
vi2q:fix/fieldless-userdata-index

Conversation

@vi2q

@vi2q vi2q commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Problem

create_userdata (and the scoped variant) assemble the metatable in two phases:

  1. User meta-methods registered via add_meta_method are set on the metatable — including a custom __index.
  2. The has_fields branch then runs. When the type registered no fields, it unconditionally overwrites __index with the method table:
} else {
    // No fields: the metatable's __index is just the method table.
    metatable.set("__index", method_table)?;
}

A custom __index registered through add_meta_method(MetaMethod::Index, ...) is silently discarded. The dynamic-proxy pattern — obj["key"] resolved in Rust against external data — returns nil for every key. mlua honors the custom handler in this situation. Because the failure is silent (nil, not an error), it is easy to miss.

Fix

When no fields are registered but a custom __index exists, install a dispatcher that calls the user's __index first and falls back to the method table when it returns nil. Applied to both create_userdata and create_scoped_userdata.

Field-ful userdata keeps the existing getter→method dispatcher unchanged.

Testing

  • New test_custom_index_metamethod_without_fields in mlua_userdata.rs: a fieldless userdata with a custom __index resolves obj["key"], returns nil for missing keys, and keeps regular methods reachable.
  • cargo test -p luaur-rt: 248 passed, 0 failed (the new test included; all existing userdata tests pass unchanged).

Found while embedding luaur-rt in a game engine: a read-only JSON data cursor exposed to gameplay scripts relied on a custom __index and silently returned nil on every key.

vi2q and others added 2 commits August 29, 2026 07:43
create_userdata assembled the metatable in two phases: user meta-methods
were set first, then the has_fields branch unconditionally overwrote
__index with the method table when the type registered no fields. A
custom __index registered via add_meta_method was therefore silently
discarded, and dynamic-proxy userdata (e.g. JSON cursors resolving
obj["key"] in Rust) returned nil.

Route the fieldless branch through a dispatcher that calls the user's
__index first and falls back to the method table when it returns nil.
Applied to both create_userdata and the scoped variant.
- Extract one `install_index_dispatcher` used by both `create_userdata` and
  `create_scoped_userdata` (the two copies had drifted apart).
- Cover the field-ful branch too: it overwrote a custom `__index` with the
  getter dispatcher, exactly like the fieldless branch did.
- Use mlua's precedence (field getters -> method table -> custom `__index`)
  instead of resolving the custom `__index` first, so a method or field is
  not shadowed by the user handler.
- Keep the fast path: with no fields and no custom `__index`, `__index` stays
  the method table itself.
- Tests: field-ful, precedence, and scoped-userdata cases alongside the
  original fieldless one. All four fail without the src change.

cargo fmt --all --check now passes.
@pjankiewicz

Copy link
Copy Markdown
Owner

Verified locally and pushed a follow-up commit to this branch (010cc47) — thanks, this is a real bug and the silent-nil failure mode is nasty.

What I changed on top of yours:

  1. cargo fmt --all --check was failing (import order + the closure wrapping). Fixed.
  2. The field-ful branch has the same bug. if has_fields { ... metatable.set("__index", index_fn) } overwrites a custom __index exactly like the fieldless branch did, so a userdata with even one field still silently dropped the handler. Both branches now go through the same code path.
  3. Lookup order flipped to mlua's. mlua's generated __index (src/userdata/util.rs) resolves field getters → methods → custom __index; this PR resolved the custom __index first, so a user handler would shadow a registered method of the same name. Your test still passes either way ("alpha" is not a method), but the precedence is observable, and parity is the point of this crate.
  4. Deduplicated: one install_index_dispatcher for create_userdata + create_scoped_userdata (the two copies had already drifted). The cheap path is preserved — no fields and no custom __index still sets __index to the method table directly, so plain method lookup stays a table access rather than a Rust call.
  5. Added three tests next to yours: field-ful, precedence (field and method shadow the handler), and scoped userdata. All four fail on main and pass with the fix.

One note on the test evidence in the description: cargo test -p luaur-rt builds no optional features, so the serde/async/typecheck suites aren't in that 248. Worth running --features serde,macros,async,typecheck locally — it's what CI's matrix does.

@pjankiewicz
pjankiewicz merged commit ce34011 into pjankiewicz:main Aug 31, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants