Skip to content

lora/sx1262: fix intermittent false negatives and hangs on shared SPI bus - #222

Merged
ThomasFarstrike merged 1 commit into
MicroPythonOS:mainfrom
steemandavid:fix-lora-sx1262-shared-spi-bus-hang
Aug 1, 2026
Merged

ThomasFarstrike merged 1 commit into
MicroPythonOS:mainfrom
steemandavid:fix-lora-sx1262-shared-spi-bus-hang

Conversation

@steemandavid

Copy link
Copy Markdown
Contributor

Problem

On the Fri3d Camp 2026 badge (and likely any board where drivers/lora/sx1262.py's SPI bus is shared with another device, e.g. a display), the SX1262 LoRa chip can:

  1. Intermittently report as absent (getPacketType() reads back 0xFF) even when a module is physically installed and wired correctly, and
  2. Occasionally stop responding entirely (BUSY never clears), requiring a full power-cycle of the board to recover — a software reset (machine.reset()) is not enough, since it doesn't power-cycle the external LoRa module.

Both were reproduced independently while testing an app that probes the LoRa chip once per launch (a hardware self-test tool for this badge) — a fresh reboot would sometimes read the chip correctly, and other times fail, with no change to the wiring or module in between.

Root cause

internal_filesystem/lib/drivers/lora/sx1262.py's SPItransfer():

self.cs.value(0)                     # chip selected

start = time.ticks_ms()
while self.gpio.value():             # wait for BUSY, up to `timeout` (default 5000ms)
    yield_()
    if time.ticks_diff(time.ticks_ms(), start) >= timeout:
        self.cs.value(1)
        return _ERR_SPI_CMD_TIMEOUT

for i in range(cmdLen):
    self.spi.write(bytes([cmd[i]]))  # actual command bytes sent only now
...

Chip-select is asserted, and then the code enters a busy-wait loop before a single command byte is sent — and this happens twice per logical command (once here, once again after the transfer via waitForBusy). On the Fri3d 2026 board, this LoRa SPI device and the display share one physical machine.SPI.Bus (see internal_filesystem/lib/mpos/board/fri3d_2026.py): the LoRa device is created with cs=-1 (hardware CS disabled) and manages its own chip-select via a plain GPIO, while the display runs on the same bus at a much higher clock speed.

I traced the SPI transport itself (lvgl_micropython, micropy_updates/esp32/machine_hw_spi.c, machine_hw_spi_device_transfer()): every individual machine.SPI.Device read/write call is wrapped in spi_device_acquire_bus() / spi_device_release_bus(), which correctly serializes that one call against other devices on the same host. But that protection is scoped per-call, not across the whole logical SX126x command (assert CS → wait for BUSY → send bytes → deassert CS). Nothing stops the display's own SPI device from acquiring the bus and running a transaction while the LoRa driver's CS line is still asserted low from a plain GPIO the framework doesn't know about — from the SX1262's point of view, the display's clock/data looks like more command bytes while it's still selected, corrupting the exchange.

This explains both symptoms:

  • A corrupted read that happens to land on 0x00/0xFF reads back as "no chip" even though one is present (there's already a comment in this file's getPacketType() caller acknowledging 0xFF is ambiguous between "absent" and "failed").
  • A transaction that gets corrupted mid-command can leave the chip waiting for bytes that never arrive in the expected sequence, which can leave BUSY asserted indefinitely — hence the full hangs, since the two timeout=5000 busy-waits per command (one before sending, one after) are both spent with CS held low, and are hit twice more if a caller retries.

This fix

Reorder SPItransfer() so the BUSY wait happens before self.cs.value(0), not after. This means CS is only held low for the actual byte transfer (fast), rather than for up to timeout milliseconds of idle polling. It meaningfully shrinks the collision window from "up to several seconds of idle CS-low" down to "the transfer itself," without changing the command protocol or touching any C code.

This is not a complete fix. The byte-transfer loop right after CS is asserted is still several separate machine.SPI calls, each individually bus-arbitrated but not as a single atomic unit — so the shared bus can still be preempted between those calls while CS is held low, just for a much shorter window than before. A complete fix needs the LoRa driver to hold the entire transfer (CS-low → last byte → CS-high) under one exclusive bus lock shared with every other device on the same machine.SPI.Bus. The primitives for that already exist and are already used in this codebase — spi_device_acquire_bus() / spi_device_release_bus() in lvgl_micropython's machine_hw_spi_device_transfer() — they're just not currently exposed to Python at a granularity a driver like this one can use (only implicitly, once per call). Exposing something like machine.SPI.Device.lock() / .unlock() (thin wrappers around the same two ESP-IDF calls) in lvgl_micropython, and having SPItransfer() hold that lock for the whole CS-low span, would close the remaining gap. Happy to follow up with that change (in lvgl_micropython) if it's a direction maintainers want — flagging it here rather than bundling an unreviewed cross-repo C change into this PR.

Testing

Verified the failure modes (intermittent false-negative reads, and full hangs requiring a power-cycle) on real Fri3d Camp 2026 hardware across many repeated launches. The reorder in this PR is a straightforward, low-risk control-flow change (pure Python, no new APIs) that I'm confident is correct by inspection, but I have not yet been able to get a clean, repeated before/after comparison on hardware in the same session (the same physical chip became increasingly unreliable after a lot of rapid repeated probing during testing — power-cycling recovers it, which is itself consistent with the "chip left mid-command" theory above). Flagging that so this doesn't read as more thoroughly hardware-validated than it is — would appreciate a second pair of eyes/hardware to confirm the improvement, especially for the full-hang case.

SPItransfer() asserted CS, then polled the BUSY GPIO for up to
`timeout` (5000ms by default) before sending a single byte. On boards
where this SPI bus is shared with another device (e.g. a display),
CS being held low for that whole span -- which can repeat twice per
command (once before sending, once after, via waitForBusy) -- widens
the window in which the other device's traffic on the shared bus gets
shifted into this chip while it's selected. That corrupts the
transaction (a clean read comes back looking like "no chip answering")
or leaves the chip mid-command, which can make it stop responding to
BUSY entirely until power-cycled.

Reordering so the BUSY-wait happens before CS is asserted means CS is
now only held low for the actual (short) byte transfer, cutting the
window from "up to 5-10s of idle CS-low" down to the transfer time
itself.

This does not fully eliminate the underlying race (the transfer loop
itself is still several separate SPI calls while CS is held low, and
this board's shared-bus arbitration only protects each individual
machine.SPI call, not the whole CS-low span) -- see PR description for
the full analysis and a suggested follow-up.
steemandavid added a commit to steemandavid/fri3d-badge-hwtest that referenced this pull request Jul 30, 2026
The one-shot LoRa probe could return a false "no rsp" (or hang) because
the display and LoRa chip share one SPI bus on this board, and the
LoRa driver holds chip-select low across a busy-wait that a display
flush can collide with. Delay the first check until the initial
screen paint settles, then retry a few times before giving up, since
one clean read is enough to prove the module is present. Retries are
capped rather than indefinite -- a failed attempt can itself block for
several seconds at the driver level, so retrying forever on a badge
with no LoRa module would leave the app sluggish for as long as it
runs. Root cause writeup and a driver-level mitigation proposed
upstream: MicroPythonOS/MicroPythonOS#222
steemandavid added a commit to steemandavid/fri3d-badge-hwtest that referenced this pull request Jul 30, 2026
… (v0.5.1)

- Rename app to "Hardware Test" for clarity in the launcher.
- Double-clicking X now reliably quits to the launcher: the splash and
  test screens each pushed their own activity-stack entry via
  setContentView(), so a single finish() only revealed this app's own
  earlier splash instead of the launcher. Pop both layers on quit.
- Fix the launcher icon overlapping the (now two-line) app name: use a
  transparent background instead of opaque black, and top-align the
  shrunk artwork so the label has clear space to wrap into.
- Batch NeoPixel writes: multiple buttons rising in the same poll tick
  now share a single LightsManager.write() instead of one call per
  button fired back-to-back, since that tight timing could upset the
  NeoPixel driver and crash the badge.
- Rework the LoRa presence check into a single minimal standby() probe
  run in onCreate(), before this app has built any screen of its own,
  instead of the previous standby()+begin()+getPacketType() sequence
  retried in a loop. Confirmed on hardware that a negative reading is
  not reliable (the same badge with a module installed reads "no
  response" right after a reset, every time, and "OK" on every later
  reopen in the same session) -- shown as "???" rather than a
  confident-looking "no rsp", since this app cannot fully tell a
  missing module from this timing issue. Root cause and a proposed
  driver-level fix filed upstream:
  MicroPythonOS/MicroPythonOS#222
@ThomasFarstrike

Copy link
Copy Markdown
Contributor

Absolutely genius!

I don't know if you were inspired by our discussion on fri3d2020.slack.com today but we were just listing the issues with the LoRa, how it's a bit jittery, and speculating about the root cause(s). I think your work here goes a long way towards stabilizing it!

Note that @lucid-void made a MeshCore app, I haven't tried it yet because I'm traveling, but someone else did and says it works. Which is surprising, considering the instability. Perhaps they are able to retry until it succeeds? Or perhaps they disable the display? I noticed that while scrolling, it's much less stable, so maybe it works fine with just a static display. Anyway, shout out to https://badgehub.eu/page/project/org.fri3d.meshcore

A complete fix needs the LoRa driver to hold the entire transfer (CS-low → last byte → CS-high) under one exclusive bus lock shared with every other device on the same machine.SPI.Bus

I'm all up for the complete fix too! Perhaps as a minimal .patch file to lvgl_micropython/ like we do for unix_autoimport_main.patch, esp32_inisetup_readsize_progsize.patch, lib_lvgl_lv_bmp.c.patch and lib_lvgl_src_libs_tjpgd_fix_scaling.patch so that way it's clear what we carry.

One "sword of Damocles" that I'm not sure when we should tackle, is that we currently have 2 different SX1262 drivers:

  • fri3d_2026.py board uses internal_filesystem/lib/drivers/lora/sx1262.py
  • lilygo_t_watch_s3_plus.py uses internal_filesystem/lib/drivers/lora/micropySX126X/sx1262.py (patched to fallback to SoftSPI)

Ideally, we would switch both (or at least the fri3d_2026.py's lora driver) to the "official" upstream https://github.com/micropython/micropython-lib/tree/master/micropython/lora / https://github.com/micropython/micropython-lib/tree/master/micropython/lora/lora-sx126x (with possibly a minimal patch or two), so we can easily keep tracking upstream.

The reasons we have different drivers are... messy, probably not fundamentally needed:

  • in part, it's because of the observed instability, which you probably fixed here
  • the 2 different SPI device APIs; one with the split SPI.Bus and SPI.Device which is used by lvgl_micropython which differs from
    https://docs.micropython.org/en/latest/library/machine.SPI.html (that's the reason SoftSPI is used for the watch, iirc)
  • I initially tried to port and use the same driver for both but it hung on the lilygo_t_watch_s3_plus.py and didn't have time to troubleshoot further so I went with the setup that worked

@cheops requested a reset_callback in the LoRa constructor to handle the reset pin, which is driven using the CH32 I/O Expander, because we currently pass a random, rarly used pin (because it's mandatory) which then gets toggled during reset. The prototype badges did not have a drivable reset pin, they were just connected to the global reset line, and that also worked, so the current behavior with the CH32 just resetting the LoRa pin at boot (IIRC @bertouttier ) might also suffice for now.

@ThomasFarstrike

Copy link
Copy Markdown
Contributor

@steemandavid would you like me to merge this one already, or take a stab at the other stuff as well?

More separate merges means more separate testing, but it's as you prefer.

@ThomasFarstrike
ThomasFarstrike merged commit e1804c1 into MicroPythonOS:main Aug 1, 2026
6 of 9 checks passed
@ThomasFarstrike

Copy link
Copy Markdown
Contributor

I already merged it, since it's a small incremental improvement, and I'll try to test it soon to make sure no regressions.

But I'd still love @steemandavid's feedback on #222 (comment)

@lucid-void

Copy link
Copy Markdown

Thanks for the mention @ThomasFarstrike, and great fix @steemandavid — that matches what I was seeing.

Short version of how MeshCore gets away with it: it doesn't avoid the race, it survives it. RX runs continuously on a worker thread with all radio SPI behind one lock (the UI thread never does raw radio SPI), and a ~2s watchdog recovers wedges by resetting the LoRa via the CH32 expander with the LCD kept powered — so the full hangs self-heal without a power-cycle. Lost packets get resent. A one-shot probe reads a transient as "no chip"; here it's just one missed poll.

My lock is intra-app only, so your PR is complementary — a shorter CS-low window means my watchdog should have to reset far less. Happy to test.

@steemandavid

Copy link
Copy Markdown
Contributor Author

I am on holidays abroad (in the mountains) so unfortunately I can't help much any more with this issue. But I'm sure if someone points a Claude Code session at void's and my github repositories that the bug can be worked out, fixed and integrated into MicroPythonOS. I am partial to integrating the BLE functionality as a core feature of the OS, with the apps using it in a non-contentious way.

@ThomasFarstrike

Copy link
Copy Markdown
Contributor

a ~2s watchdog recovers wedges by resetting the LoRa via the CH32 expander with the LCD kept powered

Aha, that explains how you were able to recover!

The prototype badge did not have a dedicated LoRa reset pin connected, so we weren't able to recover from these situations at all, other than physically pressing the reset button.

So we added that LoRa reset pin on the CH32 at the last minute, exactly to be able to recover from these types of scenario's. It was a gamble, changing the hardware without doing a small prototype run, risking that somehow this would break something unexpected on those 700 devices... not great... but happy it already helped!

@ThomasFarstrike

Copy link
Copy Markdown
Contributor

@steemandavid @lucid-void FYI I'm working on this at #229
with the code in this branch: https://github.com/MicroPythonOS/MicroPythonOS/tree/refs/heads/lora-upstream
pull request (needs testing before merge): #231

ThomasFarstrike added a commit that referenced this pull request Aug 9, 2026
Use SPI.Device.lock()/unlock() (C-level spi_device_acquire_bus /
spi_device_release_bus) in write_readinto() and readinto() so the shared
SPI bus can't be preempted between per-byte calls while CS is held low.

Closes the remaining gap from PR #222.
@lucid-void lucid-void mentioned this pull request Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants