How is CPU different from GPU?

A central processing unit offers to handle various operations like calculating, watching movies, making presentation etc. While a graphics processing unit is majorly used for the purpose of video rendering or playing of high-quality graphical games. I found the above statement on a website listing the differences between CPU and GPU. What I am not able to understand is how making PPTs,listening to songs from graphics in terms of mathematical operations that the brain of computer applies! All these operations handled in terms of bits,so why treat them differently? Also why cant we have a PC completely based on GPUs if they so good at executing parallel instructions and handling large amount of bits for graphics?Though i have heard about GPGPUs but why cant traditional GPUs be used?