Repository navigation
Conversation
ipts_mei_search() looks up a matching response under the read side of message_lock, drops the lock, and then re-acquires it for writing to unlink the entry. Two concurrent callers waiting for the same command code (typically the receiver poll thread and a userspace hidraw raw request, both inside ipts_control_send_feedback()) can each find the same entry during their read-locked walk and then both unlink and free it. The second list_del() hits LIST_POISON and, with CONFIG_DEBUG_LIST, oopses the poll kthread: list_del corruption, ...->next is LIST_POISON1 (dead000000000100) kernel BUG at lib/list_debug.c:56! RIP: __list_del_entry_valid_or_report.cold Call Trace: ipts_mei_search+0x83/0x150 [ipts] ipts_mei_recv+0x61/0x190 [ipts] ipts_cmd_recv_timeout+0x2f/0x90 [ipts] ipts_control_send_feedback+0x81/0xc0 [ipts] ipts_receiver_poll_loop.cold+0x89/0x11c [ipts] ipts_thread_runner+0x21/0x40 [ipts] Without CONFIG_DEBUG_LIST the same race shows up as a general protection fault on 0xdead000000000108 in ipts_mei_search(). The other caller is then left blocked in D state inside the driver, which makes every later suspend fail with "tasks refusing to freeze". Seen on resume from s2idle on a Surface Pro 7 while iptsd was (re)started. Hold the write lock across the search and the unlink so that a message can only be claimed by one caller. Signed-off-by: Josh Spaulding <git@spaulding.app>
The context is allocated with devm_kzalloc(), so devres frees it when the device is released, whoever still uses it. The following changes bound the stop path, and a stop that gives up must keep everything that a receiver thread or a HID device which is still there can touch. With a devm context, devres frees it anyway, and the next touch is a use-after-free. Reference-count the context instead, and free it only when the last reference is dropped: - Probe holds one reference, dropped by a devres action. Remove and every later probe failure disable the MEI device before that, which cancels its receive callback, so no message arrives while the context is freed. If adding the devres action fails, the context is freed at once, before the MEI device is disabled; the receive callback finds no context then, because drvdata is not set yet. - The receiver thread and the HID device hold one each, for their whole life. - The context holds a reference to the MEI device until it is freed: a user that is still there after remove keeps sending through it and logging against it. ipts_remove() then waits, bounded, for the remaining users to return. That narrows the window in which the module code is freed under one that is about to return, but does not close it: the last user wakes remove from inside its own put, before it has left the module code. After a failed stop the receiver thread or the HID device is still there on purpose, and remove does not wait for it. A failed probe used to return without disabling the MEI device, so its receive callback stayed registered after the context was gone. Every probe failure after mei_cldev_enable() now disables it. The messages are now plain kzalloc() instead of devm: they are freed on the error paths of the receive callback, and any left over is freed with the context. Fixes: 820c589 ("hid: Add support for Intel Precise Touch and Stylus") Patchset: ipts Assisted-by: LLM Signed-off-by: Tony Garcia <180770769+loafecho@users.noreply.github.com>
ipts_remove() runs on rmmod, and also on every reboot, power-off and suspend: mei_me_shutdown() and mei_me_pci_suspend() both call mei_stop(), which removes every ME client device. ipts_thread_stop() waited for the receiver thread with wait_for_completion() and no timeout. If the thread died in an oops, or never got out of a wait, the stop never returned, and reboot or suspend hung with it. Wait for the thread with a timeout instead, which the caller passes in. While waiting, check whether the thread is exiting: a thread that died runs no driver code anymore and counts as stopped, without calling kthread_stop() on it, which could wait for its exit forever. A thread that is still running after the timeout is reported to the caller, which then frees none of the driver's memory that the thread can still touch: the receiver keeps the thread's context reference, so the context stays, and the context keeps the MEI device. Only a later stop drops that reference. After remove there is none, so the context then stays for good. Until later changes in this series remove the HID device on those paths, a stop that gives up here, or finds the thread dead, still returns before the HID device is removed: rmmod with a stuck or dead receiver then leaves the HID device registered, and opening its hidraw node runs freed module code. Before this change, that rmmod hung for good instead. Keep a reference on the task_struct from creation until the stop is done, so that a thread which exits on its own early can still be looked at, and never store a kthread_create() error as the thread. The return value of the thread function moves to thread->exit_code; the receiver reports it as before. Fixes: 820c589 ("hid: Add support for Intel Precise Touch and Stylus") Patchset: ipts Assisted-by: LLM Signed-off-by: Tony Garcia <180770769+loafecho@users.noreply.github.com>
Every command waits for its reply in ipts_mei_recv(), which slept in wait_event() with ipts_mei_search() as the wait condition. The search takes message_lock, so a waiter blocked on the lock inside the wait condition, and a task that held the lock without releasing it (one that died while holding it) blocked every later command for good: the receiver thread, the HID requests, and the shutdown handshake on the stop path. The lock only ever covers a list add or delete, so nobody has a reason to hold it for long. Never block on it: - ipts_mei_search() only tries the lock, and reports a busy lock to the caller. - ipts_mei_recv() searches with the task running and sleeps until a new message is queued (a sequence counter bumped under the lock) or the time is up. While the lock is busy, it tries again after a short sleep. The time limit covers both. - ipts_mei_incoming() gives up after a bounded wait for the lock and drops the response. - Sends use mei_cldev_send_timeout(). An interrupted send is not retried, because the message is already queued and would go out twice. The timeout bounds the wait for the write to complete; the MEI core can still wait without a bound for its bus lock, and for room in a full transmit queue, which the driver cannot change. Holding the lock also holds a context reference, taken and dropped by the new lock helpers, so the context is never freed under a holder that never lets go. The lock is released before the reference, because dropping the last reference frees the context, and the release function takes the lock itself to free the messages. The "negative timeout waits forever" case could never be reached with a u64 timeout and is gone; a zero timeout still checks once. Fixes: 820c589 ("hid: Add support for Intel Precise Touch and Stylus") Patchset: ipts Assisted-by: LLM Signed-off-by: Tony Garcia <180770769+loafecho@users.noreply.github.com>
When the receiver thread returned an error from its shutdown handshake, _ipts_control_stop() and ipts_control_stop() returned early and never called ipts_hid_free(). The module then unloaded with its hid_device still registered, and the next open or ioctl of its hidraw node ran freed module code: a page fault in hidraw_open() or inside __hid_hw_raw_request(), with "[last unloaded: ipts]" in the oops. Remove the HID device on every path once the receiver thread is gone. The only exception is a receiver thread that is still running after the bounded stop: it still feeds the HID device and uses the buffers, so nothing is freed and the error is passed on. Until a later change in this series counts the receiver's HID input, that exception still lets rmmod with a stuck receiver leave the HID device registered, with the hidraw problem above. And a receiver that died inside hid_input_report() may hold HID core locks, so destroying the HID device after it can block; the same later change makes the stop wait for running HID input first, and keep the HID device in that case. If the thread died, or the device did not confirm the stop, the device may still write to its DMA buffers. Leak them instead of freeing them; the next start allocates new ones. The stop now reads the thread's exit code itself. A stop that finds no thread, because a start failed before it started one, reports a clean exit instead of the exit code of the thread before. Fixes: 820c589 ("hid: Add support for Intel Precise Touch and Stylus") Patchset: ipts Assisted-by: LLM Signed-off-by: Tony Garcia <180770769+loafecho@users.noreply.github.com>
…firmation Once a stop is requested, the receiver loops still finished every buffer of the current round before they looked at the stop flag again, and the poll loop walked through every doorbell value it had missed, however far the doorbell had moved. Check the stop flag inside the inner loops too, and never walk back further than the buffers that can hold new data. The poll loop sent QUIESCE_IO only when it saw the stop flag at the top of a round. A stop requested while it handled buffers ended the loop at the bottom check without sending it, and the device was never told to stop. Send it after the loop instead, as the event loop does, and then give back the buffers the loop did not get to, as before: in poll mode the device confirms the stop only once every filled buffer was given back. At the end of a loop, the stop handshake gave up on the QUIESCE_IO confirmation as soon as the last READY_FOR_DATA reply was missing, and reported a missing reply as success. Starting the device again before it has stopped can leave its firmware unresponsive until the next boot, and freeing buffers it still writes to corrupts memory. Collect the last READY_FOR_DATA reply, then wait for the QUIESCE_IO confirmation for a bounded time in any case, and return success only when the device confirmed the stop. Otherwise the stop keeps the DMA buffers. That is a trade-off: a device that never confirms its stop leaks one set of DMA buffers per stop, and that repeats on every suspend and every restart, with a warning each time. Memory the device may still write to is never freed. Every step after the stop request is bounded now, apart from the waits inside the MEI core that a send can still make, so the time a stop waits for the receiver thread is the sum of those bounds instead of a guess. A slow device then still gets its stop confirmed, instead of leaving a thread behind that keeps the buffers. The bounds move to receiver.h, so that the waits of the stop path can be derived from them too. Fixes: 820c589 ("hid: Add support for Intel Precise Touch and Stylus") Patchset: ipts Assisted-by: LLM Signed-off-by: Tony Garcia <180770769+loafecho@users.noreply.github.com>
On EDS1 devices, a HID request that switches the mode restarts the device from inside the HID callback, and nothing serialized that restart with ipts_remove(): both could stop the receiver thread and free the buffers at the same time. A stop that waits for running HID callbacks, as the next change makes it do, would also wait for the very callback that called it. Run the restart from a work item instead, under a mutex that remove takes for its own stop. A queued restart holds a context reference until it has run. Remove marks the context as going away first, so no new restart is scheduled, and a restart that is already running checks that mark again before it starts the device. Remove never waits for the work without a bound: the worker can be the task that is stuck, or that died with the mutex held, and reboot and suspend must still finish. It cancels a queued restart without waiting, before and after its stop, and takes the mutex with a bounded wait. The bound is derived from the bounds of a start and a stop, the way the receiver's stop timeout is: it covers whatever a running restart can still do once remove has marked the context. If a restart holds the mutex past that bound, it is stuck, and remove skips the stop and frees nothing that restart can still use: the restart holds a context reference, and the context holds the MEI device. A restart whose stop was not confirmed by the device, so that the DMA buffers were leaked, no longer starts the device again: starting a device that has not stopped can leave its firmware unresponsive until the next boot. The stop reports that case with a return value of its own. Touch input then stays off until the driver is probed again. Fixes: 820c589 ("hid: Add support for Intel Precise Touch and Stylus") Patchset: ipts Assisted-by: LLM Signed-off-by: Tony Garcia <180770769+loafecho@users.noreply.github.com>
Stopping only cleared hid_active, which new HID callbacks check on entry. A callback that had already passed that check kept running while the stop freed the buffers it used: a SET_FEATURE request writing the hid2me buffer, or a GET_FEATURE request waiting for a reply in a buffer that was about to go away. Count the callbacks that run driver code, and make the stop wait for the count to drop to zero, with a timeout. A pending GET_FEATURE request is released at once instead of waiting for its own timeout, and one that starts while the stop runs returns at once. The receiver thread's HID input is counted the same way. Once the count has dropped to zero, the receiver can no longer reach the HID device, so the HID device is destroyed even when the receiver thread does not stop. Before, a receiver that did not stop kept the HID device registered, and after rmmod its hidraw node ran freed module code. The receiver thread is asked to stop in any case. A callback that returns while it stops was only slow, and the stop goes on as usual, except that the buffers are leaked instead of freed: the callback may have sent the device a command after the device confirmed its stop, and the device may answer it by writing to them. A callback that still runs after the timeout and the receiver stop is most likely stuck for good, in a task that died inside it, possibly with HID core locks held. That includes a receiver that died inside hid_input_report(). It can still touch the buffers, and destroying the HID device could block forever on those locks, so in that case nothing is freed and the error is passed on. A later stop then leaks the buffers instead of freeing them, although it finds no receiver thread anymore. So rmmod after a HID callback died, or while one never returns, still leaves the HID device registered, and its hidraw node runs freed module code. Destroying it would trade that for a remove that can block, and with it reboot and suspend. Fixes: 820c589 ("hid: Add support for Intel Precise Touch and Stylus") Patchset: ipts Assisted-by: LLM Signed-off-by: Tony Garcia <180770769+loafecho@users.noreply.github.com>
When a step of ipts_control_start() failed, it returned at once and left behind whatever the earlier steps had set up: the buffers, and a receiver thread that kept running against a device that was never told to stop. An allocation error from hid_allocate_device() was even kept in ipts->hid, where later code took it for a device. Tear down on failure what the start set up. Buffers the device was never given are freed. Buffers it may have been given are leaked, since it was never told to stop: that covers a GET_DESCRIPTOR request that got no reply, because the device may still write the descriptor. Once the receiver thread runs, the normal stop path stops it, and keeps what a thread that does not stop can still use. On an EDS1 restart, that stop also removes the HID device that probe created. A HID device that failed to register is no longer destroyed in ipts_hid_init(), while the receiver may already use it, but by that stop, once HID input has stopped. The receiver thread already runs while ipts_hid_init() sets up the HID device, so the device is now set up in a local variable and published in ipts->hid only once it is complete. The receiver never sees an error pointer or a half set-up device. Fixes: 820c589 ("hid: Add support for Intel Precise Touch and Stylus") Patchset: ipts Assisted-by: LLM Signed-off-by: Tony Garcia <180770769+loafecho@users.noreply.github.com>
|
@antoniovazquezblanco thanks for running #178 on the SP4. Saw you closed intel-precise-touch#35 in favour of this PR. The code does cover it, ipts_thread_start() holds a reference on the task from creation until the stop is done and it doesn't leave an error pointer behind, same two changes as yours. That's in 224f261. But I only have a Surface Pro 7, so none of this has ever run on EDS1, which is where you actually hit that use-after-free. So I can say it should be fixed, I can't say it is. Would you be able to try this branch on your SP4? Mainly I'd want to know if the refcount warnings from your trace are gone when iptsd switches modes. That restart runs from a work item now, so more changed on that path than just the reference. Also worth watching dmesg for "Device did not confirm the stop" or "not restarting IPTS". If the device takes longer than about 3 s to confirm a stop, touch stays off until the next probe, and I honestly don't know how often that happens on EDS1. You also might not want to drop yours because of this. Yours is one file and you tested it on the hardware that hits the bug, this is 9 commits and could sit for a while. If you reopen it against v7.3-surface-devel and it lands first I'll just rebase on it, no problem. And since you reported this back in August, lmk if you want a Reported-by on that commit, I'd just need the name and address you want on it. One more thing, I saw my comment on #178 got linked from a lot of the intel-precise-touch threads. That was only my read from the commit history, not something a maintainer told me, so qzed or StollD would be the ones to say if that repo should be archived. |
|
Looks like my claude ended up duplicating some of this in #188 - so that's independent confirmation I guess. I've been testing on a Surface Book 1 / ESD, so I'll pull your branch here and see how it goes. I've got three more fixes that aren't in your branch though, how would you feel about including those with your PR? I can also do a followup if you think that better, but I'd rather test the full patch set rather than a subset. The three additional fixes are:
|
|
@jaydenmilne nice, that's good confirmation, especially the "one reply behind" part, because I've been chasing the same thing on my SP7 (EDS2) and found at least one way it gets there. FEEDBACK replies are matched to their sender by command code only. The receiver loop (giving back data buffers) and HID feature requests (through the one shared hid2me buffer, and SET_FEATURE isn't locked at all) can have FEEDBACKs in flight at the same time. In my traced runs, once three or more were in flight at once, a sender took another one's reply and returned early, and from then on the ME stayed one reply behind, only letting the newest reply out when the next command arrived. A QUIESCE_IO sent in that state never gets confirmed. I traced it with bpftrace on the MEI send/recv calls: every unconfirmed stop had exactly one reply owed when QUIESCE_IO went out, every confirmed stop had none, and in 25 of 25 idle gaps the owed reply never came out on its own. My fix serializes each FEEDBACK with its reply under one lock (which also covers the hid2me buffer) and drops a FEEDBACK reply that's already queued before sending, since under the lock that can only be a late one. Reloading the driver while 4 processes send feature requests, 200 cycles: 108 unconfirmed stops before, 0 after. On a KASAN + lockdep build, 160 before, 3 after, no reports. The FEEDBACK reply also echoes the buffer index in its first 4 bytes, so replies could be matched exactly if we want to go further. Your last three look complementary to that. Mine keeps the ME from getting one reply behind, your reconnect gets it out if it happens anyway, and your mode switch retry covers the EDS1 case #187 still leaves open (touch stays off until the next probe). I only have the SP7, so I can't test EDS1 myself. Your first three overlap with #187 (no retry on -EINTR, the thread reference, tearing down after a failed stop). One difference there: when the device didn't confirm the stop, #187 removes the HID device but keeps the DMA buffers instead of freeing them, since the device may still write to them. Since you want to test the full set, I can put one branch together: #187, my FEEDBACK fix, and your last three rebased onto v7.3, with your authorship and sign-off kept on yours. We both run it, then decide with the maintainers whether it goes into #187 or a follow-up. I'd lean follow-up so #187 doesn't grow, but I'm fine either way. Separate question: do you see "QUIESCE_IO: recv failed" on the SB1 when it suspends? I still get a rare unconfirmed stop on suspend, about 1 in 100, with or without my fix, and nothing owed in the ones I traced. Haven't found why yet. |
|
@jaydenmilne the combined branch is up: https://github.com/loafecho/kernel/tree/ipts-combined-test (branch It's #187 as it is now (6e15adc on v7.3-surface-devel) plus four commits:
Nothing was dropped. Results on my SP7 (EDS2, Ubuntu 24.04, 7.3.0-rc7 from v7.3-surface-devel), same tests as before:
It's installed as my daily driver now. The suspend one is the rare unconfirmed stop I asked you about before; it's the same with or without the fix, so it's something else. What I can't test: your mode-switch retry and the ME reconnect only run on EDS1, so on my SP7 they compile and stay out of the way, nothing more. Could you run it on the SB1? Two things I'd look for:
My commit only has an |
|
@jaydenmilne follow-up PR is up: #196. It's the combined branch you saw (my FEEDBACK fix plus your three, with your authorship and sign-off kept) plus one new commit for the rare unconfirmed stop on suspend. Cause: IPTS is a fixed-address MEI client, so on a stop the ME sends the READY_FOR_DATA and QUIESCE_IO replies back to back. When both land before the bus's rx work runs, the bus calls our callback once, we read one reply, and the QUIESCE_IO reply just sits in the bus queue. The fix keeps reading until nothing's ready. 300 suspends each, alternating: 3 unconfirmed without it, 0 with it. Your reconnect commit describes the ME getting stuck one response behind with the cause unknown. That might be this same thing (a reply stranded in the bus queue until the next one arrives), though I haven't proven it. If you can still hit that state on the SB1, it'd be worth seeing whether it still happens with the new commit. I still can't test the EDS1 paths here, so a run on the SB1 before merge would be great. |
|
OK - ran the soak test. Claude found a regression and says it fixed a few things, they're 4 new commits on the tip of my branch. The fully patched series ran very well. I can paste more of claude's notes if that helps. I can't reproduce the "one response behind" state with the complete branch, so it's possible we can drop the recovery when it gets behind
|
|
@jaydenmilne thanks for running it, and for catching the iptsd restart one. That's from my "Run the EDS1 mode-switch restart from a work item" in #187, as your commit message says, and the SP7 is EDS2 so I never saw it. I read your four commits and built them on top of #196: no new warnings, and checkpatch only flags the Co-Authored-By form. On the SP7 they don't change anything, since the restart path only runs on EDS1. Three notes:
Since #187 alone breaks iptsd restarts on EDS1, it shouldn't go in by itself. Happy to pull your commits into #196 with your authorship and sign-off once you've settled 1 and 3, or you can open them on top, whichever you prefer. |
The first commit here is #178 itself, unchanged and with its author kept, because this series depends on it and it is not merged yet. It drops out of this PR once #178 lands.
The IPTS driver's stop path had no bound: it waited for the receiver
thread with wait_for_completion(), and a thread that died in an oops or
slept forever on a stuck message lock hung every later stop, and with it
reboot and suspend, since ipts_remove() runs on both. When a stop failed
it also unloaded with its HID device still registered, so the next open
of its hidraw node ran freed module code. And the context was allocated
with devm, so any stop that gave up turned into a use-after-free.
This series reworks the stop path on top of linux-surface/kernel #178:
what a stuck task still uses without devres freeing it, and it holds
the MEI device for as long as it lives;
for a reply is woken on stop, and the other waiters run to their own
timeouts;
across a sleep;
the receiver is running, even if the receiver thread does not stop;
and no HID request outlived the stop's wait for it; otherwise they are
leaked, and an EDS1 device is not restarted;
the bounds of a start and a stop.
AI-assisted: Claude Code found the bugs, wrote these patches and their
messages, and ran the plain device tests. OpenCode wrote the test-only
fault-injection builds and ran those tests. AI models from several labs
reviewed the series. I directed the work on my own device and take
responsibility for it.
Only one device is available for testing, a Surface Pro 7 (EDS2, poll
mode) on Ubuntu 24.04, running linux-surface 7.3.0-rc7 with unrelated
camera patches. The EDS1 paths, including the mode-switch restart, are
untested. It needs linux-surface/kernel #178 applied first. Every patch
builds on its own and checkpatch --strict is clean.
Tested on that Surface Pro 7 with the final series. The touch check
passed at the end of every test run:
make the stop path stall or fail in five ways, 231 rmmod runs and 9
reboots: no rmmod and no reboot hung, and the 225 runs that checked
the kernel log found no oops;
and a use-after-free detector in the test build never fired. On an
earlier, unfixed version of this rework the same test fired it in 8
of 12 runs;
rmmod runs took 5-6 s (about 0.15 s without it) and leaked the DMA
buffers as designed, with no oops, and touch came back after each
reload.
The test patches and scripts are available on request.
Known limits:
a receiver thread that never returns, a HID callback that never
returns or dies, a task that holds the message lock forever, or an
EDS1 restart that never finishes. The driver frees none of its own
memory that such a task can still use, the context keeps the MEI
device for it, and reboot and suspend still finish. But after rmmod, a
receiver thread that wakes up runs freed module code. A HID callback
that never returned keeps the HID device registered, because
destroying it could block on HID core locks the stuck task holds, so
opening its hidraw node after rmmod runs freed module code too.
Closing this needs module pinning, which this series does not do.
never freed, even if the thread exits later: nothing stops it again.
That narrows the window in which a user still runs module code after
rmmod, but does not close it: the last user wakes remove from inside
its own put.
mei_cldev_send_timeout() can still wait without a bound for the MEI
bus lock, and for room in a full transmit queue.
hangs in the MEI core: mei_cldev_disable() waits for that work with
cancel_work_sync(), and the driver cannot bound that wait.
stop, on every suspend and every restart. An EDS1 restart then does
not start the device again, so touch stays off until the next probe.
A HID request that outlives the stop's wait for it leaks one set too,
with the same effect on an EDS1 restart.