We are working on a different pivot at Spark and its query optimizer / executor seems stuttering under pressure of concurrent requests, particularly when requests are prepared ( .cache() + .persist(MEMORY_ONLY) and executed against indexed data using Hyperspace https://github.com/microsoft/hyperspace.
With cached dataset the gap is even further, since memory cached data is out of grasp of query optimizer which currently resorts to brute force scan, which in turn is extremely slow, when it comes to queries targeting slices using "where" clause or filter.
Those are the sweet spots that lend themselves most easily to service like module. Where faster hardware, such as GPU, makes a lot of sense.