Leveraging Heterogeneity in Time-to-Event Predictions