Which database to use? Storing one string every second for several days

I would like to observe some cryptocurrency prices and for that, I want to store a string, everey second in a database for ca. 3 days. The string will vary in length, it will be 2500 characters max. (complete orderbook), sometimes just 50 (prices). I want to analyze the prices later in Python.

Several questions arise for me: 1) Which database would you recommend? I have worked with SQLite and like it so far, but I am unsure whether it’s the most efficient solution, as file sizes get large and reading might be slow. Are there more appropriate solutions? 2) If the string has a special format, e.g. JSON, should my choice from quesiton 1) change – are there special databases which fit well with my use case?

Thank you very much in advance!

Storing dates and their history of changes and emailing – Does my database design pose any problems?

Background

I’m designing some tables to allow dates to be assigned to an object so progress can be tracked on that object, along with functionality to store emails sent pertaining to these contracts and their dates. For example lets say that the Object is a Contract (A contracted task to be completed).

This means we would have a contract table

ContractTable

Now for this contract we can have Date fields such as a obtained date, start date, review date, estimated finish date, finish date etc. Right now most contracts will contain the same columns but we don’t know in the future if more columns will be added or less will be used as requirements change quite often.

Because of this, I figured to separate the dates from the contract table. Each date in this ContractDate table would have a foreign key to a FieldHeading table that describes what aspect of the contract the date is. And finally, a AuditTrail type table called ContractDateHistory that would keep track of changes of a Date.

Contract and Date table seperate with a history of dates table.

This way we can also selectively choose what dates go to a contract. One contract may have 4 dates and another may have 6.

These contracts can have emails sent to people that inform them of how these dates are going or if a date has changed. Because multiple contracts can be contained in one email I have a ContractsForEmail table which contains the ID of the email of each contract that is contained in an email.

Contracts with emailing tables

A row would be input for each email sent to a user, needsResolution is there if the user has to take action with the email (depends on the type of email) and resolved is there if the user has taken action, a program will check if the user has taken action after a set amount of time, and if not, the program will send a reminder email (which could possibly also show up in this table)

The template can have a list of users that the emails derived from the template sends to (an email will be sent for each user assigned to the mailing list of a template, additional could be sent too but users that without fail get an email sent are kept here.

My Question

I have many questions about if my design is okay but in specific I’d like to know, is what I am doing with contracts and contractDates is okay?

Instead of having the Contract and Dates separate should I just add Date columns to the Contracts table?

If they get used then good and if they don’t for a specific contract then its just kept as NULL.

This could lead to columns in the contract table that get added and never used again (almost always NULL). Like so:

Contract table with dates inside

AddedDateInFuture is there to illustrate that a column may be added in the future, it wouldn’t literally be there.

  • What are the trade-offs between the two options?
  • Is there anything I’m missing with the emailing tables? Would this kind of design work?

Currently the system would be catering to under 200 users but I’d like it to be built a little future proof for if many more use it.

As a last note some database tables such as the usual “User” table has been left out.

Storing search based database in excel [on hold]

I am searching for a program to download all results that can acquired from searching a database that only accepts numbers within a certain range to search. This one for Example: http://natega1.youm7.com/ Once you enter a number roughly between (110000-700000) I think. the website displays the corresponding data to the code. The site is hosting exam results if you are wondering.

So is there a program that can search for all results within a certain range and store them in a file like excel perhaps? I am no programmer and don’t even know how to begin search google for this. Any help is appreciated.

Btw, sorry for any misuse of words.

How to change default directory for storing screenshots via hotkeys?

I want to change the autosave directory for screenshots taken with hotkeys like Print Screen. I tried to update configs of gnome-screenshot, something like:

gsettings set org.gnome.gnome-screenshot auto-save-directory "file:///home/$  USER/screenshots/" 

But after some tries I figured out that GNOME (triggered by hotkeys) doesn’t use gnome-screenshot for making pictures. So what shoult I do next?

OS: Ubuntu 19.04 Gnome-shell: 3.32.1

Storing earth observation derived attributes for polygons

I want to generate a number of vegetation related indices (NDVI, NDTI, etc., min/max/avg/std (pixel values)) from earth observation data for a large number of polygons during the entire growing season, roughly between March and November each year. The figures are about as follows: I have about one million polygons. Every day I will get new EO data (Sentinel-1/Sentinel-2) for about 20% of them. For each of these polygons I generate 10-20 indices based on the EO data. This gives me appr. 2-4 million records, every day. That makes appr. 500 – 1000 million during just one growing season (I´ll need to store at least 5 seasons).

The infrastructure within which I have to operate is predetermined and will have to be something based on either Oracle(Locator) or PostGIS. Personally I´d prefer PostGIS since OpenSource allows for much more flexibility.

My initial idea is to create a PostGIS database, which is partitioned based on year value. I thought about creating one attribute table where I create a new row for each date and each interpreted property (polygon geometries+id are stored in a separate table). It would look something like this:

table_structure

Since I have to do different interpretations depending on the geographical zone where the polygon lies, I also thought about creating a separate table for each zone. This will however make querying more difficult.

My questions are hence:

  1. Does Oracle (Locator) or PostGIS as base for all this make sense at all or do I need to start asking for an account at an ESA DIAS/Google Earth Engine/AWS in order to be able to use cloud solutions?
  2. If this indeed makes sense, wht is your opinion on my planned table structure?

All comments & help welcome.

Does PCI-DSS requirement 10 (“track and monitor all access to … card holder data”) apply if I am not storing card holder data?

Requirement 10 states: Track and monitor all access to network resources and card holder data

I find this a little vague and I have two questions.

If I don’t store card holder data – do I just need to monitor access to networking resources?

When they say “Card holder data” Do they mean, the Card holder data environment or specific records of card holder data? Can anyone clarify?

Storing a search string in VIM? [migrated]

I have a fairly elaborate regex string for finding IP addresses (25[0-5]|2[0-4][0-9]|[0-1][0-9]{2}|[0-9]{1,2}. etc.) If, for instance, I want to get all instances of the IP address, I have to delete everything in the line prior to the instance, and in a second call, I have to delete everything after the instance. Is there a way that I could store the search string as a variable to use in multiple replacements?

For instance:

    Store the string in register "a" :y a 25[0-5]|2[0-4][0-9]|[0-1][0-9]{2}|[0-9]{1,2}\. . .     Refer back to register "a" when deleting everything before the address :%s/^.*(\a)//g     And again when deleting everything after it :%s/(\a).*$  //g 

Best practice for storing confidential properties in a Spring Boot application

I’m currently storing an API key as a property in the YAML file of my Spring Boot application. This is obviously insecure and I’d like to know what is best practice in web services for storing confidential information like this. Is it best to house it in a database or is there a secure way of storing it in a YAML or Properties file that can be hidden if it’s hosted on a public repository like BitBucket or GitHub?

Criteria: I’d like a means of storing confidential properties for a Spring Boot application (API keys, passwords, etc.) in a format that can be stored in a public git repository but also in such a way that they are not visible to public – i.e. I’d still like them to be under source control

Can a Shadow Sorcerer see through Darkness cast into a Ring of Spell Storing via Eyes of the Dark?

A sorcerer with the Shadow Magic origin has a feature called Eyes of the Dark, which states the following (XGTE, p. 51):

When you reach 3rd level in this class, you learn the darkness spell, which doesn’t count against your number of sorcerer spells known. In addition, you can cast it by spending 2 sorcery points or by expending a spell slot. If you cast it with sorcery points, you can see through the darkness created by the spell.

A Ring of Spell Storing (DMG, p. 192) states:

This ring stores spells cast into it, holding them until the attuned wearer uses them.

Any creature can cast a spell of 1st through 5th level into the ring by touching the ring as the spell is cast. The spell has no effect, other than to be stored in the ring.

While wearing this ring, you can cast any spell stored in it. The spell uses the slot level, spell save DC, spell attack bonus, and spellcasting ability of the original caster, but is otherwise treated as if you cast the spell.

If a sorcerer cast darkness using sorcery points as per their Eyes of the Dark feature into the Ring of Spell Storing, then when they later “released” the spell from the ring, would they be able to see through the darkness as per their Eyes of the Dark feature?

(Bonus question: what if the ring was used by someone else; would the sorcerer who cast the spell originally still see though it, even though someone else “released” the spell? This is just out of interest, though; the main question I care about is in the case where the sorcerer uses the ring.)

I imagine one could argue yes, because it was originally cast via sorcery points, but one could also argue no, because of the technicality of the ring’s description saying “but is otherwise treated as if you cast the spell”, or something like that. Which is it?

Storing files in database through encryption and Viewing them only on providing key for the file

I have this use case:

  1. User1 uploads his file into the database which should be encrypted.
  2. Any other users who request that file, user1 will be notified for approval, on approving other users should be able to view the file but can’t download the file.
  3. The file should be encrypted in the database so that it cannot be viewed if the database is hacked.