That’s exactly what’s done in the reverse avalanche oscillator as shown at the above link:
The pot is on the supply voltage, and it affects the pitch. By limiting the current it increases the time needed to charge the pot, so the frequency decreases.
In the picture posted above one pot is added to the supply voltage for multiple VCOs, changing all their pitches at the same time.
Be careful not to put too much current through the pot or it’ll burn out.