Using Temp Tables in Azure Data Studio Notebooks

tl;dr I want to use temp tables across multiple cells in a Jupyter Notebook to save CPU time on our SQL Server instances.

I’m trying to modernize a bunch of the monitoring queries that I run daily as a DBA. We use a real monitoring tool for almost all of our server level stuff, but we’re a small shop, so monitoring the actual application logs falls on the DBA team as well (we’re trying to fix that). Currently we just have a pile of mostly undocumented stored procedures we run every morning, but I want something a little less arcane, so I am looking into Jupyter Notebooks in Azure SQL Data Studio.

One of our standard practices is take all of the logs from the past day and drop them into a temp table, filtering out all of the noise. After that we run a dozen or so aggregate queries on the filtered temp table to produce meaningful results. I want to do something like this:

Cell 1

Markdown description of the loading process, with details on available variables 

Cell 2

T SQL statements to populate temp table(s) 

Cell 3

Markdown description of next aggregate 

Cell 4

T SQL to produce aggregate 

The problem is that, it seems, each cell is run in an independent session, so the temp tables from cell 2 are all gone by the time I run any later cells (even if I use the “Run cells” button to run everything in order).

I could simply create staging tables in the user database and write my filtered logs there, but eventually I’d like to be able to pass off the notebooks to the dev teams and have them run the monitoring queries themselves. We don’t give write access on any prod reporting replicas, and it would not be feasible to create a separate schema which devs can write to (for several reasons, not the least of which being that I am nowhere near qualified to recreate tempdb in a user database).

notebooks couple, double trouble

I’d really like a solution for my problem. I have two notebooks: one is an Asus f540sa-xx220T an the other is a Compaq 15-s004nl. The compaq was working very well but doesn’t start anymore because of a hard drive recent problem. The Asus starts but it’s working so slow that you can’t do anything anymore. The thing is I don’t want to throw them away. I really want to save at least one. So the idea was to open them and maybe exchange the inside pieces. Do like a super notebook 2.0 taking like the hard drive to change the broken one and maybe upgrading the ram, you know. The thing is the pieces doesn’t really seem compatible. For example I didn’t find a place for the ram and stuff in the other computer and all the stuff is so different. My idea is even possible? It’s really different from the PC work. If upgade is not possible than what could I do? The notebooks have been open on the table for one day already. Some advice? P. S. I’d like not to spend money if it’s possible =))

How to chain unix commands in jupyter notebooks while on windows 10?

My Os is windows 10 and I am using a jupyter notebook from anaconda prompt.

When I type !pwd it works, When I type !ls -l it works, But when I want to chain the commands and output them to an output file e.g. !(pwd; ls-l) > out.txt it says: “pwd: unknown option — l Try ‘pwd –help’ for more information.”

Does anyone know how to chain these commands and output them to a file?

Thank you in advance

Groups vs notebooks

I’m working on a note taking app. One of the feature is that you can group the notes in a folder like manner in which one can create. subgroups inside groups and more subgroups and put notes/notebooks inside it. The app has also a feature where the user can make notebooks in which the user can add notes (You cannot put a notebook inside a notebook). But doesn’t that make the notebook same as a group without subgroups? How useful this feature can be? Can having both confuse people? Thank You.

Nvidia driver breaks some notebooks LAN connection, huh?

ISo, I have an ASUS ROG G551JW and I’ve installed half-a-dozen Linus distros when I’ve realized, I wasn’t really the one f*cked it up. So nearly on all OS, I was trying to set up the laptop for gaming, so I’ve installed NVidia drivers. All wen’t well, always, until the 1st reboot. Right after that, I have lost my LAN connection. The OS is trying to connect but no luck and I’m also unable to make it work with static IP addresses. I’ve came to the point to install POP OS Nvidia edition, and now I’ve realized, there is no connection on the live system either. What could cause this habit of Arch, Ubuntu, Fedora and other OSs based on these, to act like what I’ve just described? Any workaround?

How compatible are data science notebooks with clean architecture?

Clean architecture decouples an app’s core from the presentation/UI layer. The UI is just a plugin, replaceable (eg, web-based to desktop) without impacting the core.

Many data science apps mix code, user inputs, text, graphics and other outputs in one notebook, eg, Jupyter. Everything seems coupled: the domain, UI, presentation, persistence.

Q: How to design such an app cleanly, with the notebook maximally decoupled? Or are notebooks inherently incompatible with clean architecture?

Perhaps I could have an independent module with core functionality. The notebook would call this module, without defining any non-trivial functionality. Would this, however, allow enough decoupling or even fit with a notebook?

Why:

I’ll be developing an app for a client who’s only used Excel. The app will predict cost effectiveness of medical treatments and will need MCMC simulations, regression and other stats.

I plan to implement it in Python with Jupyter or the nteract notebook, pushed by Netflix https://medium.com/netflix-techblog/tagged/nteract. However, this may eventually prove unsuitable for the client, as Jupyter is mainly used by those who program it themselves. There’re other potential pitfalls, eg, https://docs.google.com/presentation/d/1n2RlMdmv1p25Xy5thJUhkKGvjtV-dkAIsUXP-AL4ffI/edit#slide=id.g362da58057_0_1. Ideally, I could easily swap between notebook types or change over to a desktop GUI.

How can a SSD freeze a notebook’s boot screen for half a minute?

My wife has a Toshiba Satellite C855D-11U and I got an idea that I’ll enhance its performance, so I bought a Kingston SUV500 120GB SSD. I put it into the notebook then after power on the TOSHIBA logo was shown for almost one minute. During this time I couldn’t do anything but when I pressed the F2 or F12 after one minute, the notebook opened the boot menu / BIOS.

When I put back the original Toshiba HDD, the bootlogo appeared for 1-2 seconds and everything went fine. I noticed that the problem also exist when I don’t use any SATA device.

The SATA mode in BIOS is AHCI, the USB Legacy is enabled.

What could be the problem?