Following up from my earlier post, I’ve been chipping away at this slowly. Another bad video first!
Most of the work I’ve done is on the software. I’ve rewritten most of it in a much more maintainable, logical manner as I’ve been learning CircuitPython (and Python generally). Whereas before I threw everything together in one file and hoped for the best (like most Arduino projects) I’ve made distinct modules for the display, switches, jacks and audio engine. The samples can be changed on the fly (although I’ve not written a way to do that directly on the module yet - at the moment it requires using the serial terminal) and I’ve put in support for one-shot samples and loops, either triggered or gated in both cases. The status of the sort of sample is reflected on the LED display, one character for each voice.
Oh, and I’ve added SD card support 
just in case 16MB of Flash isn’t enough (or more likely I use it as the basis for a different project down the line).
I’ve started working out a menu for loading samples, loading/saving sample sets, adjusting volumes etc. That’s my next push.
CircuitPython’s sample handling isn’t perfect - there’s no changing playback speed or seeking a position in a sample, for example - but for my needs that’s fine!