Skip to content

hw: the S2 PLL lock check waits for the wrong condition - #340

Open
vrilutza wants to merge 2 commits into
patjak:masterfrom
vrilutza:pll-lock-check
Open

hw: the S2 PLL lock check waits for the wrong condition#340
vrilutza wants to merge 2 commits into
patjak:masterfrom
vrilutza:pll-lock-check

Conversation

@vrilutza

@vrilutza vrilutza commented Sep 4, 2026

Copy link
Copy Markdown
Contributor

Two defects around the S2 PLL lock check in fthd_hw_s2_pll_init().

The polling loop is inverted. It spins while S2_PLL_CMU_STATUS_LOCKED is set, but fthd_reg.h documents that bit as 1 = PLL locked. So it waits for the PLL to lose lock, and both of its outcomes are reported backwards:

PLL loop message
locks 10000 iterations, ~256 ms Failed to lock S2 PLL
does not lock exits on the first read S2 PLL is locked after 10 us

Which branch is taken depends only on whether the PLL had settled by the time of the very first read, so the driver picks one of two paths at random.

The error is dropped. fthd_hw_s2_init_ddr_controller_soc() ignores the return value, and the error path leaves before putting the PLL into bypass — so the DDR40 setup that follows runs against a clock configuration nothing intended.

Measured on a MacBookPro14,1, 36 loads of each, alternating master and patched within the same boot:

                      "Failed to lock"   entered bypass
master                    18 of 36          18 of 36
patched                    0 of 36          36 of 36

With the patch the PLL reports locked after one or two polls (10–20 µs), CMU_STATUS = 0xc902c902, bit 15 set — the exit is on the right condition. Fisher exact, two-tailed: p = 4.4e-07.

Why I went looking

This machine hard-hangs — screen frozen, no oops, nothing in pstore, automatic reset — when the module is reloaded after the firmware has stopped responding. Two ingredients, both ordinary: a firmware timeout, which happens on its own (that is the premise of #332), and a module reload afterwards. I could reach the timeout reliably with an out-of-spec crop, which is what made it measurable.

With a black box writing /dev/kmsg to disk with fsync on every line, the last thing before death is always inside the DDR40 VDL configuration — the code that runs right after the skipped bypass:

facetimehd: deinit failed!
facetimehd: Found FaceTime HD camera with device id: 1570
facetimehd: Waiting for S2 PLL to lock at 450 MHz
facetimehd: Failed to lock S2 PLL: 0xc902c902     <- i.e. it had locked
facetimehd: DDR40 PLL is locked after 0 us
   (nothing. dead.)

On a healthy load the next line comes 0.02 ms later.

              hangs   reloads after a firmware timeout
master          2              18
patched         0             120

The 120 include 60 with iommu.strict=1 and 60 without. Fisher exact: p = 0.016.

I would not claim that on its own — two incidents is thin, and this machine has a history of unexplained deaths. The inverted condition stands on its own regardless: it is wrong against the semantics in fthd_reg.h, and it is what puts the driver on the no-bypass path in the first place.

The second patch changes nothing on this machine, since the error path is unreachable once the loop is fixed. It matters where the PLL genuinely does not lock: refusing to load is better than continuing into the DDR40 setup without bypass.

The loop spins while S2_PLL_CMU_STATUS_LOCKED is set, but fthd_reg.h
documents that bit as "1 = PLL locked", so it waits for the PLL to lose
lock instead of to acquire it. Both of its outcomes are then reported
backwards:

  PLL locks         -> 10000 iterations, ~256 ms, "Failed to lock S2 PLL"
  PLL does not lock -> exits on the first read, "S2 PLL is locked after 10 us"

Which one happens is decided by whether the PLL had settled by the time
of the very first read, so the driver takes one of two paths at random.
That matters because the error return leaves before putting the PLL into
bypass, and the caller ignores it, so the DDR40 setup that follows runs
against a clock configuration nothing intended.

The 0xff00 mask is dropped with it: bit 15 is inside it, so it never
selected anything.

Measured on a MacBookPro14,1, 36 loads of each: master reports "Failed to
lock" -- that is, the PLL had locked -- on 18 of 36, and skips bypass on
those. With this patch, 0 of 36; the PLL reports locked after one or two
polls, 10-20 us, with CMU_STATUS = 0xc902c902 and bit 15 set, and bypass
is entered every time.

Signed-off-by: Viorel Cernateanu <vrilutza@gmail.com>
fthd_hw_s2_pll_init() returns -EINVAL when the PLL does not lock, and on
that path it returns before putting the PLL into bypass. The caller drops
the return value and carries on into the DDR40 setup regardless, driving
the DDR PHY from a clock the error path left half configured.

Every other failure in fthd_hw_s2_init_ddr_controller_soc() aborts, so do
the same here.

With the previous patch the error path is not reached on this machine --
the PLL locks on every load -- so this changes nothing here. It matters
where the PLL genuinely does not lock: refusing to load is better than
continuing into the DDR40 setup without bypass.

Signed-off-by: Viorel Cernateanu <vrilutza@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant