We propose SPT, a semi-parametric prompt tuning method that improves multitask generalization for parameter-efficient fine-tuning. SPT demonstrates cross-task zero-shot generalization at scale through large-scale soft prompt tuning experiments.