lora/sx1262: fix intermittent false negatives and hangs on shared SPI bus - #222
Conversation
SPItransfer() asserted CS, then polled the BUSY GPIO for up to `timeout` (5000ms by default) before sending a single byte. On boards where this SPI bus is shared with another device (e.g. a display), CS being held low for that whole span -- which can repeat twice per command (once before sending, once after, via waitForBusy) -- widens the window in which the other device's traffic on the shared bus gets shifted into this chip while it's selected. That corrupts the transaction (a clean read comes back looking like "no chip answering") or leaves the chip mid-command, which can make it stop responding to BUSY entirely until power-cycled. Reordering so the BUSY-wait happens before CS is asserted means CS is now only held low for the actual (short) byte transfer, cutting the window from "up to 5-10s of idle CS-low" down to the transfer time itself. This does not fully eliminate the underlying race (the transfer loop itself is still several separate SPI calls while CS is held low, and this board's shared-bus arbitration only protects each individual machine.SPI call, not the whole CS-low span) -- see PR description for the full analysis and a suggested follow-up.
The one-shot LoRa probe could return a false "no rsp" (or hang) because the display and LoRa chip share one SPI bus on this board, and the LoRa driver holds chip-select low across a busy-wait that a display flush can collide with. Delay the first check until the initial screen paint settles, then retry a few times before giving up, since one clean read is enough to prove the module is present. Retries are capped rather than indefinite -- a failed attempt can itself block for several seconds at the driver level, so retrying forever on a badge with no LoRa module would leave the app sluggish for as long as it runs. Root cause writeup and a driver-level mitigation proposed upstream: MicroPythonOS/MicroPythonOS#222
… (v0.5.1) - Rename app to "Hardware Test" for clarity in the launcher. - Double-clicking X now reliably quits to the launcher: the splash and test screens each pushed their own activity-stack entry via setContentView(), so a single finish() only revealed this app's own earlier splash instead of the launcher. Pop both layers on quit. - Fix the launcher icon overlapping the (now two-line) app name: use a transparent background instead of opaque black, and top-align the shrunk artwork so the label has clear space to wrap into. - Batch NeoPixel writes: multiple buttons rising in the same poll tick now share a single LightsManager.write() instead of one call per button fired back-to-back, since that tight timing could upset the NeoPixel driver and crash the badge. - Rework the LoRa presence check into a single minimal standby() probe run in onCreate(), before this app has built any screen of its own, instead of the previous standby()+begin()+getPacketType() sequence retried in a loop. Confirmed on hardware that a negative reading is not reliable (the same badge with a module installed reads "no response" right after a reset, every time, and "OK" on every later reopen in the same session) -- shown as "???" rather than a confident-looking "no rsp", since this app cannot fully tell a missing module from this timing issue. Root cause and a proposed driver-level fix filed upstream: MicroPythonOS/MicroPythonOS#222
|
Absolutely genius! I don't know if you were inspired by our discussion on fri3d2020.slack.com today but we were just listing the issues with the LoRa, how it's a bit jittery, and speculating about the root cause(s). I think your work here goes a long way towards stabilizing it! Note that @lucid-void made a MeshCore app, I haven't tried it yet because I'm traveling, but someone else did and says it works. Which is surprising, considering the instability. Perhaps they are able to retry until it succeeds? Or perhaps they disable the display? I noticed that while scrolling, it's much less stable, so maybe it works fine with just a static display. Anyway, shout out to https://badgehub.eu/page/project/org.fri3d.meshcore
I'm all up for the complete fix too! Perhaps as a minimal .patch file to lvgl_micropython/ like we do for One "sword of Damocles" that I'm not sure when we should tackle, is that we currently have 2 different SX1262 drivers:
Ideally, we would switch both (or at least the fri3d_2026.py's lora driver) to the "official" upstream https://github.com/micropython/micropython-lib/tree/master/micropython/lora / https://github.com/micropython/micropython-lib/tree/master/micropython/lora/lora-sx126x (with possibly a minimal patch or two), so we can easily keep tracking upstream. The reasons we have different drivers are... messy, probably not fundamentally needed:
@cheops requested a reset_callback in the LoRa constructor to handle the reset pin, which is driven using the CH32 I/O Expander, because we currently pass a random, rarly used pin (because it's mandatory) which then gets toggled during reset. The prototype badges did not have a drivable reset pin, they were just connected to the global reset line, and that also worked, so the current behavior with the CH32 just resetting the LoRa pin at boot (IIRC @bertouttier ) might also suffice for now. |
|
@steemandavid would you like me to merge this one already, or take a stab at the other stuff as well? More separate merges means more separate testing, but it's as you prefer. |
|
I already merged it, since it's a small incremental improvement, and I'll try to test it soon to make sure no regressions. But I'd still love @steemandavid's feedback on #222 (comment) |
|
Thanks for the mention @ThomasFarstrike, and great fix @steemandavid — that matches what I was seeing. Short version of how MeshCore gets away with it: it doesn't avoid the race, it survives it. RX runs continuously on a worker thread with all radio SPI behind one lock (the UI thread never does raw radio SPI), and a ~2s watchdog recovers wedges by resetting the LoRa via the CH32 expander with the LCD kept powered — so the full hangs self-heal without a power-cycle. Lost packets get resent. A one-shot probe reads a transient as "no chip"; here it's just one missed poll. My lock is intra-app only, so your PR is complementary — a shorter CS-low window means my watchdog should have to reset far less. Happy to test. |
|
I am on holidays abroad (in the mountains) so unfortunately I can't help much any more with this issue. But I'm sure if someone points a Claude Code session at void's and my github repositories that the bug can be worked out, fixed and integrated into MicroPythonOS. I am partial to integrating the BLE functionality as a core feature of the OS, with the apps using it in a non-contentious way. |
Aha, that explains how you were able to recover! The prototype badge did not have a dedicated LoRa reset pin connected, so we weren't able to recover from these situations at all, other than physically pressing the reset button. So we added that LoRa reset pin on the CH32 at the last minute, exactly to be able to recover from these types of scenario's. It was a gamble, changing the hardware without doing a small prototype run, risking that somehow this would break something unexpected on those 700 devices... not great... but happy it already helped! |
|
@steemandavid @lucid-void FYI I'm working on this at #229 |
Use SPI.Device.lock()/unlock() (C-level spi_device_acquire_bus / spi_device_release_bus) in write_readinto() and readinto() so the shared SPI bus can't be preempted between per-byte calls while CS is held low. Closes the remaining gap from PR #222.
Problem
On the Fri3d Camp 2026 badge (and likely any board where
drivers/lora/sx1262.py's SPI bus is shared with another device, e.g. a display), the SX1262 LoRa chip can:getPacketType()reads back0xFF) even when a module is physically installed and wired correctly, andmachine.reset()) is not enough, since it doesn't power-cycle the external LoRa module.Both were reproduced independently while testing an app that probes the LoRa chip once per launch (a hardware self-test tool for this badge) — a fresh reboot would sometimes read the chip correctly, and other times fail, with no change to the wiring or module in between.
Root cause
internal_filesystem/lib/drivers/lora/sx1262.py'sSPItransfer():Chip-select is asserted, and then the code enters a busy-wait loop before a single command byte is sent — and this happens twice per logical command (once here, once again after the transfer via
waitForBusy). On the Fri3d 2026 board, this LoRa SPI device and the display share one physicalmachine.SPI.Bus(seeinternal_filesystem/lib/mpos/board/fri3d_2026.py): the LoRa device is created withcs=-1(hardware CS disabled) and manages its own chip-select via a plain GPIO, while the display runs on the same bus at a much higher clock speed.I traced the SPI transport itself (
lvgl_micropython,micropy_updates/esp32/machine_hw_spi.c,machine_hw_spi_device_transfer()): every individualmachine.SPI.Deviceread/write call is wrapped inspi_device_acquire_bus()/spi_device_release_bus(), which correctly serializes that one call against other devices on the same host. But that protection is scoped per-call, not across the whole logical SX126x command (assert CS → wait for BUSY → send bytes → deassert CS). Nothing stops the display's own SPI device from acquiring the bus and running a transaction while the LoRa driver's CS line is still asserted low from a plain GPIO the framework doesn't know about — from the SX1262's point of view, the display's clock/data looks like more command bytes while it's still selected, corrupting the exchange.This explains both symptoms:
0x00/0xFFreads back as "no chip" even though one is present (there's already a comment in this file'sgetPacketType()caller acknowledging0xFFis ambiguous between "absent" and "failed").timeout=5000busy-waits per command (one before sending, one after) are both spent with CS held low, and are hit twice more if a caller retries.This fix
Reorder
SPItransfer()so the BUSY wait happens beforeself.cs.value(0), not after. This means CS is only held low for the actual byte transfer (fast), rather than for up totimeoutmilliseconds of idle polling. It meaningfully shrinks the collision window from "up to several seconds of idle CS-low" down to "the transfer itself," without changing the command protocol or touching any C code.This is not a complete fix. The byte-transfer loop right after CS is asserted is still several separate
machine.SPIcalls, each individually bus-arbitrated but not as a single atomic unit — so the shared bus can still be preempted between those calls while CS is held low, just for a much shorter window than before. A complete fix needs the LoRa driver to hold the entire transfer (CS-low → last byte → CS-high) under one exclusive bus lock shared with every other device on the samemachine.SPI.Bus. The primitives for that already exist and are already used in this codebase —spi_device_acquire_bus()/spi_device_release_bus()inlvgl_micropython'smachine_hw_spi_device_transfer()— they're just not currently exposed to Python at a granularity a driver like this one can use (only implicitly, once per call). Exposing something likemachine.SPI.Device.lock()/.unlock()(thin wrappers around the same two ESP-IDF calls) inlvgl_micropython, and havingSPItransfer()hold that lock for the whole CS-low span, would close the remaining gap. Happy to follow up with that change (inlvgl_micropython) if it's a direction maintainers want — flagging it here rather than bundling an unreviewed cross-repo C change into this PR.Testing
Verified the failure modes (intermittent false-negative reads, and full hangs requiring a power-cycle) on real Fri3d Camp 2026 hardware across many repeated launches. The reorder in this PR is a straightforward, low-risk control-flow change (pure Python, no new APIs) that I'm confident is correct by inspection, but I have not yet been able to get a clean, repeated before/after comparison on hardware in the same session (the same physical chip became increasingly unreliable after a lot of rapid repeated probing during testing — power-cycling recovers it, which is itself consistent with the "chip left mid-command" theory above). Flagging that so this doesn't read as more thoroughly hardware-validated than it is — would appreciate a second pair of eyes/hardware to confirm the improvement, especially for the full-hang case.