A Scalable and Flexible Framework for Gaussian Processes via Matrix-Vector Multiplication