Playing online without a VTT, and is it worth it?

New DM here. I have dabbled a bit with D&D in the past, including playing on FG back in 2010. I’m now looking to get back in to it and start up an adventure with my wife and a few friends. My friends are remote, so it would necessitate playing online. I really enjoy rolling actual dice though, and I fear that using a VTT like Roll20 or Fantasy Grounds would kind of cheapen the experience. I’d like to run a pre-written adventure, most likely Mines of Phandelver just to ease the DMing burden rather than create my own adventure right off the bat.

How much more work would it be to just use Discord and basically play without any VTT, doing everything manually? Would this slow down the game too much? I’m eventually shooting for like a 70%/30% RP to combat ratio because I really enjoy theater of the mind vs. tactical maps/dungeon crawls. Is this insane? Should I just use a VTT and get on with it?