Grasp of Hadar: can using it while flying cause falling damage?

We got into a battle with some berserkers. I was already flying (fly spell) so decided to use eldritch blast and the Grasp of Hadar invocation.

Once on each of your turns when you hit a creature with your eldritch blast, you can move that creature in a straight line 10 feet closer to you. (XGtE, p. 57)

As I flew over one of the Berserkers, I zapped him with EB twice (at 5th level this cantrip emits two blasts) and used one of them to lay Grasp of Hadar on him. As my DM ruled it, I pulled him toward me ten feet, straight up. Once once I had pulled him up, he didn’t stay up; the DM ruled that he had to fall since the Grasp doesn’t grapple him nor hold him in place.

I recommended to the DM that he take an additional 1d6 falling damage; I rolled it and he agreed.

Is this correct, or did we misunderstand how Grasp of Hadar works?
And if he falls, is he now prone?

Imbedded question if the answer to the above is “yes, d6 was correct” –
Something that had us scratching our heads was how far offset do I need to be before the fall won’t be enough to cause damage? I can be as high as 90′ up and EB will still reach, but if I am too far offset the vertical height of the Grasped creature won’t be 10 feet (for example at a 45 degree angle they would be about 7 feet off the ground). For the time being the DM is going to wing it, but I thought I’d ask so that I have something more concrete to offer him.

A related question about thorn whip is here and I suspect that the Four Elements Monk Water Whip feature has a similar behavior.