note that this also avoids the events/rule pushing race condition I ran into without any clusterpzfdatasource special casing

Unfortunately this might not work as intended as the existence of final_metadata.json / events.json is used in pushers to confirm whether spooling is complete (and, by inference, is assumed to occur after all the frames have been pushed).

Hard to be 100% certain if this will cause any issues, but there is a small chance that it will give rise to yet another race condition down the track. It might be safer to leave writing these files where it is and add a new spool_complete flag to check in the return from StartSpooling() instead. Will see if I can quickly pull together an alternative.

ah, I see your point. I do kind of think this is still the right place to push the events/metadata though, and wonder if it should just be combined with a spool_complete signal for the pushers to connect?

making the pushers aware of the spooler completion (rather than relying on file existence checks) is on the TODO list, and would be nice, but will only ever work for pushers which are in-process - and not for pushers which are running elsewhere - e.g. triggered from ClusterUI, or from the ruleserver, e.g., with a rule that monitors directories for changes (If you notice the rule structure, IntegerIDRule is currently the only rule type implemented on the ruleserver, but it derives from a Rule base class for a reason- the long term plan has always been to add GLOB based rules - i.e. generate tasks for all files matching this pattern - as well).

I guess I don't see how we're answering anything other than in-process pushers right now? A non in-process pusher e.g. on clusterUI will still check for the events file

--edit-- sorry, this was completely missing your last point. I'm there now!

Closing as issue addressed in #857 in a slightly safer / more paranoid way