Bug#102586: multi-table DELETE with ON DELETE CASCADE breaks row-based replication

[matanbaruch]

What does this change do?

A multi-table DELETE that names both a foreign key parent table and its ON DELETE CASCADE child breaks row-based replication: the replica applier stops with ER_KEY_NOT_FOUND. This defers the delete until the join has finished when a cascade dependency exists between tables inside the query, on both the classic and the hypergraph optimizer path.

BUG#102586, BUG#80821

Why is it needed?

Both bugs are Verified and still unfixed. #80821 was reported in 2016, #102586 in 2021, and neither has moved since October 2023. #102586 is S1.

What happens:

1. The parent row is deleted while the join is still scanning.
2. The cascade removes the child rows and logs row events for them.
3. The statement also logs row events for the child rows it deletes itself.
4. On the replica the parent delete is applied first, its own cascade removes the child rows, and the logged child events cannot find them. Error 1032.

A single-table DELETE that relies on the cascade is fine. Only the multi-table form is affected.

We hit this in production on 8.4.8 LTS, which is listed on neither bug report. Three replicas broke inside the same five minute window and stayed broken for three weeks, because Amazon RDS kept restarting the applier and the lag looked normal. A local harness reproduces it identically on 8.0.35, so this is not a regression in a recent version, it has just never been fixed.

The design is Zsolt Parragi's, contributed on #80821 in 2019 and acknowledged by Oracle at the time. That patch no longer applies: get_cascade_foreign_key_table_list() was removed. This version resolves the cascade dependency from TABLE_SHARE::foreign_key_parent instead, and covers the hypergraph optimizer, which did not exist in 2019.

How was it tested?

• Added/updated MTR tests under mysql-test/
• scripts/ci/mtr.sh passes locally
• Ran the relevant full suite (name it): rpl, 1100 tests, --parallel=8 --force

New test: rpl.rpl_multi_table_delete_fk_cascade. Debug build, 26.7.0, Linux, before and after the same commit:

	classic optimizer	hypergraph optimizer
trunk	fail: 1032, HA_ERR_KEY_NOT_FOUND on test.t2	fail: same
this branch	pass	pass

Both optimizer paths were checked separately because they decide immediate deletion in different places, and trunk fails in both.

Also passing on this branch: main.foreign_key_cascade, innodb.innodb, innodb.innodb_misc1 (the existing tests that combine cascading foreign keys with multi-table DELETE), and the 25 tests matching --do-test=delete.

The test keeps an ON DELETE SET NULL case as negative coverage. That one replicates correctly on trunk, so it is deliberately left in immediate mode: deferring it would change which rows the statement removes.

Two tests failed in the full rpl run and both pass in isolation on the same binary, so I read them as flaky under --parallel=8 rather than caused by this change: rpl_crash_on_pfs_worker_table_against_replica_stop and rpl_parallel_alter_db_table. Neither involves DELETE or foreign keys. If they are known-stable in your CI, say so and I will dig further.

Contributor checklist

• I have signed the OCA with the email on these commits
• Code is formatted (scripts/ci/format.sh)
• Commits are focused with descriptive messages

AI assistance

• I did not use AI assistance for this contribution
• I used AI assistance for this contribution

Claude Code wrote the patch and the MTR test. The fix design is not new, it is Parragi's 2019 approach from #80821 rebased onto current trunk. Everything claimed above was verified by running it, not by inspection: the before and after matrix comes from real MTR runs on a local Debug build, in both optimizer modes. The production root cause behind it was diagnosed separately against a Docker reproduction of our schema.

Areas touched

optimizer, replication

[matanbaruch]

I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it

[matanbaruch]

The three red checks here are CI-side and all of them reproduce without this patch. Writing up what I found, in case it is useful:

1. Build, MTR and Format Check never see the code. All three workflows trigger on pull_request_target and check out refs/pull/N/merge, which GitHub now refuses for forks:

Refusing to check out fork pull request code from a 'pull_request_target' workflow.

Checkout fails in about 5 seconds and every later step then fails with working directory .../source: No such file or directory. #714 and #716 fail identically. #713 still built for 8m21s, so this arrived with the recent contribution automation change.

2. Format Check fails on trunk itself. It runs clang-format-18 --style=file --dry-run --Werror over whole changed files. Three of the files this PR touches are not clean under clang-format 18 at base commit 06a5c1c99, with nothing applied: sql/sql_base.cc:3185, sql/sql_delete.cc:634, sql/join_optimizer/join_optimizer.cc:4760 and :10034. Those are exactly the lines reported on this PR, shifted by my insertions. Running the formatter over those three files rewrites 2306 lines, including include ordering and signature reflows, which looks like the tree was formatted with a different clang-format version. I left them alone rather than bury a 54 line fix. The lines this PR adds are clean under git clang-format.

3. MTR cannot finish inside the budget. mtr.yml is runs-on: ubuntu-latest with timeout-minutes: 360, and GitHub's hard job limit is 6h. A Debug build plus the default MTR selection reached 25% in 6h on that runner class and was killed, with no test failures. There are no successful MTR runs in this repository's history.

Since CI here cannot run the code, I ran your workflows unmodified on a fork-internal PR, where checkout is allowed:

• Debug build (gcc): pass, 1h23m
• Debug build (clang): pass, 1h40m
• rpl.rpl_multi_table_delete_fk_cascade 'row': pass

Local Debug build of 26.7.0, same commit, before and after:

	classic	hypergraph
trunk	1032, HA_ERR_KEY_NOT_FOUND on test.t2	same
this branch	pass	pass

Both optimizer paths decide immediate deletion in different places and trunk fails in both, which is why the patch touches both.

If you want the format gate green on this PR I can add a separate commit that reformats those three files, but it is 2306 unrelated lines and I would rather not unless you ask for it.

[RidhaOracle]

@matanbaruch You are totally right regarding about the github actions failures. I have described the issue in more details in the issue #715. I have a patch for it that will be deployed early next week, it took a bit of time to deploy it as I also tried to stabilize the MTRs. Sorry about the noise !

[RidhaOracle]

@matanbaruch The fix for issue #715 have been merged in trunk. Can you please rebase your branch to trigger the pipelines ?

[matanbaruch]

@RidhaOracle MTR (replication) failed only on rpl_stm_gtid_disconnect_drop_temporary_table, a REPLICA_OPEN_TEMP_TABLES timing race unrelated to this patch (no DELETE, no FK, and the same commit passed that shard on my fork). Can you re-run the shard?