I have decided to improve my well-being and in case being successful write a note about this and share it with my peers for free and try to help them improve there well-being as well. But on this road, I need foundational advice from the database practitioners.
In my research, I need to collect a lot of different types of time-series (GDP, Real GDP, Nominal GDP, Consumption, Investment, Percapita GDP, Number of Hours Work, Unit Labor Cost and many more). Right now, I am following and collecting about 155 time-series. Until now, I have been updating them manually as soon as an update was published on statistical web pages. I have been doing this using Excel spreadsheet (like, downloading new excel and then copying and pasting the necessary data point into my excel). However, this is a very daunting task, and it is prone to a lot of errors. Checking and finding the error is another nightmare. So, in total, maintaining this Excel database (updating checking and documenting) takes about 10-12 hours per week approximately.
- I want to create a database where I will have all these and many more variables (I want each variable to have a name, description, source description (ideally with URL included), date of last update etc.) stored.
- I also want to be able to update the database automatically from downloaded Excel or CSV file (These files come in different formats from various webpages. Some are structure vertically some horizontally).
- I also want to have some error checkers for easier debugging the base.
- The series are of different frequencies (some are daily, some weekly, some monthly and some quarterly and yet some annual). I want to be able to easily convert from daily to monthly or from monthly to quarterly or vice versa using my own predetermined formula. (being it average or median or whatever the current task requires)
- I also want to then later to be able to easily query any variable or several of the variables and create a dashboard in various programs like MATLAB or Python or Julia to visualize dynamics and use them for regression or model estimations in these programs.
For all the tasks described above which database management program would you recommend (ideally, it would be better to be it free and open source)
P.S. I tried writing MATLAB script for automatic updates, but it’s very inconvenient.