Are you sure it doesn’t get triggered, even briefly, by the act of connecting it?
I do know that typically in this sort of envelope generator, once it’s triggered it takes a long time (several seconds) to get back down to near 0 V, no matter how fast a decay time you set, because the diodes’ conductivity drops when you get below ~1 V. The Kassutronics Precision ADSR uses op amps to get around that problem, mostly. Even so there is a slight nonzero voltage because the gate produced by the input transistor doesn’t go quite to zero volts, but it’s a matter of ~10 mV or so I think.