No way of restricting public access to Firestore/API

Just glancing at GCP offerings for storing data, I noticed that while using Firestore, the only control for restricting public access is via security rules. However, in case of mis-configuration of security rules or compromise on access tokens/keys the data store becomes absolutely public available at:

https://firestore.googleapis.com/v1/projects/<YOUR_PROJECT_ID?/databases/(default)/documents/*/** 

What’s the way of completely blocking public access here (or restrict access to certain whitelisted IPs)? I am aware that we cannot put managed services inside a VPC.