Describe the thing
We've gone through some robustness improvements lately with the timing of signals/flags/etc in HTTPSpooler, which has helped a lot, but I still timed out 2/96 of the last detections on the first run through (they auto-retriggered fresh overviews, which is great, but obviously we'd be even more robustness if we didn't need that) because the recipes were looking for events files before they were there.
Notably I connect to the onSpoolStop signal to push rules, which currently gets sent before the new finalise. One of the things I was going for in #850, which as @David-Baddeley mentioned had it's own potential flaw, was to send the spoolstop signal after the events have hit the cluster which is our current mark of completion. There's now a TODO in Spooler about whether to call finalise before the signal instead, which I'll PR shortly as I think that would be grand.
I did additionally want to just toss out there a couple other thoughts about events though, without taking control of the pyramid distribution discussion. If we aren't OK pushing events in batches as we go in the same way we push frames in groups as we do, I think we'll limit ourselves in terms of how 'smart' we can be. For example, I want to get the z height right at every ROI because I messed it up last week and spoiled a dataset (shouldn't have been imaging so late in the evening I guess). Anyway, in our currently framework, my options are to a) run a widefield protocol chained to a height-determining recipe which also queues back SMLM, b) image one z cycle in SMLM chained to a recipe to figure out whether we got the right height and requeue the remaining cycles with different stack settings (which notably would also take some stacksettings infrastructure) or c) just hack that stacksettings infrastructure into my initscript, i.e. don't really do anything in any framework. If we pushed events as we went though, and some recipe/protocol when can i release this task magic, we could adjust on the fly based on where we actually get localizations. Obviously this sort of feedback ability would be helpful potentially for 405 / other laser intensities as well.
So:
Describe the thing
We've gone through some robustness improvements lately with the timing of signals/flags/etc in HTTPSpooler, which has helped a lot, but I still timed out 2/96 of the last detections on the first run through (they auto-retriggered fresh overviews, which is great, but obviously we'd be even more robustness if we didn't need that) because the recipes were looking for events files before they were there.
Notably I connect to the onSpoolStop signal to push rules, which currently gets sent before the new
finalise. One of the things I was going for in #850, which as @David-Baddeley mentioned had it's own potential flaw, was to send the spoolstop signal after the events have hit the cluster which is our current mark of completion. There's now a TODO in Spooler about whether to call finalise before the signal instead, which I'll PR shortly as I think that would be grand.I did additionally want to just toss out there a couple other thoughts about events though, without taking control of the pyramid distribution discussion. If we aren't OK pushing events in batches as we go in the same way we push frames in groups as we do, I think we'll limit ourselves in terms of how 'smart' we can be. For example, I want to get the z height right at every ROI because I messed it up last week and spoiled a dataset (shouldn't have been imaging so late in the evening I guess). Anyway, in our currently framework, my options are to a) run a widefield protocol chained to a height-determining recipe which also queues back SMLM, b) image one z cycle in SMLM chained to a recipe to figure out whether we got the right height and requeue the remaining cycles with different stack settings (which notably would also take some stacksettings infrastructure) or c) just hack that stacksettings infrastructure into my initscript, i.e. don't really do anything in any framework. If we pushed events as we went though, and some recipe/protocol when can i release this task magic, we could adjust on the fly based on where we actually get localizations. Obviously this sort of feedback ability would be helpful potentially for 405 / other laser intensities as well.
So: