You are looking at it wrong by taking the names too literally. GPUs are simply processing units optimized for parallel computation and CPUs processing units optimized for general purpose sequential computation. And these optimizations require architectural trade-offs, so to be efficient at both types you'll need to have both a CPU and GPU.

So think of it this way, a CPU is actually a General-purpose Sequential Processing Unit and a GPU is a Parallel Processing Unit, but renaming them would only add to the confusion.