How do SRP (and PAKE generally) protect against verifier leak

I have been reading into PAKE protocols, specifically starting with SRP RFC2945

The gist of the requirement on the server is that the server saves triplet (username, verifier (v), salt (s)) in the credentials table.

Where verifier v = g^x % N (the ^ operator is the exponentiation operation) and x = SHA(<salt> | SHA(<username> | ":" | <raw password>))

Now during the authentication dance, the client obtains the salt (s) from the server and computes the same verifier (v) that the server is using. It can compute this value because it collects the username and password from the user themselves.

In the next few steps the client can subtract the component v from the servers challenge B = (v + g^b) % N and arrive at a key derived from S.

Client: S = (B - g^x) ^ (a + u * x) % N

My question is that if someone hacks my database and dumps the credential table with (username, verifier (v), salt (s)) they immediately have access to all verifiers for each username. Then what stops them from using the obtained verifiers to imitate a client and complete the client side authentication steps? So instead of computing the verifier (v) from the real username and password, they can simply use the verifier obtained maliciously from the server to continue the client side computation and arrive at the same key as the server.

i.e. In my reasoning if my server is hacked and credentials leaked the end result is no better than if I were saving plain text passwords.

Disclaimer: I admit I do not understand fully the maths but generally the concept that such crypto protocols rely on the property of exponents that (g^a)^b = (g^b)^a = g^ab