Can you trace a VPN user by matching up traffics when it exits the server?

If one were to monitor traffic exiting a VPN server, could you identify traffic streams of a user by marching return port numbers? Eg you identity traffic of interest returning to port number x. You then look for other traffic with return port number x. You find some of this traffic is for (eg Dropbox). Dropbox can resolve that traffic to a user. Hence you identify the user of the first traffic stream?

Basically I guess I’m asking if one user of a VPN can have all associated traffic identified.