Right now we have a highly analytical system which collects tons of various metrics. We need to display this data in a web ui for users to consume.
Since we are in early stages of development we are throwing all data into google bigquery, when a users visits the app we run a cloud function to request the data from bigquery, and store/cache it in firebase.
When a user visits the same route for the same view we will show them the data from firebase (old data) whilst we fetch any updated results from bigquery.
The data itself is similar to web event logs for the most part, but some of it is more complex, and we are storing it as a
json string in bigquery.
It feels like a complex and hard to maintain setup for a small team, as we are still building the MVP and keep iterating on the views presented to the user. My question is how to simplify our database choices, to have to maintain less DBs, but still have flexibility around changing the final views and an acceptable level of latency for queries <2secs.