Simple Web Crawler for Small Sites

This script is my web crawlerfor small sites. Can you possibly review it for best coding practices?


import bs4 as bs import urllib.request from urllib.parse import urlparse,urljoin import pprint   class Page(object):     def __init__(self,base_url,url):         self.url = url         self.base_url = base_url     def soup(self):         sauce = urllib.request.urlopen(self.url).read()         return (bs.BeautifulSoup(sauce,'lxml'))     def title(self):         soup = self.soup()         return soup.title.string     def links(self):         urls = []         soup = self.soup()         href = [i.get('href') for i in soup.findAll('a') ]         links = [i for i in (list(map((lambda url : url if bool(urlparse(url).netloc) == True else urljoin (self.base_url, url)),href))) if i.startswith(self.base_url)]         return links     def map_page(self):         map = {self.url:{'title':self.title(),'links':set(self.links())}}         return map  def site_map(base_url):     map_pages = {}     links_to_map = [base_url]      def check_and_add(url):         if url not in map_pages:             [links_to_map.append(i) for i in Page(base_url,url).links()]             (map_pages.update(Page(base_url,url).map_page()))             links_to_map.remove(url)         else:             links_to_map.remove(url)      while links_to_map != []:         url = links_to_map[0]         check_and_add(url)      pprint.pprint(map_pages) 

I have a small Open Source project I’d like reviewed

Is this the proper place? It’s a node.js module intended as a possible alternative to Helmet.

If this is not the place, any suggestions? Thanks.

Hmm, apparently I must include at least three lines of code. Here’s how you would use it:

// declarative style let kepi = Kepi({   'X-Powered-By': 'super duper system',   'Content-Type': ['text/html', 'charset=utf-8']   'Feature-Policy': {     vibrate: "'none'",     geolocation: "'self'",   }, });  // can also be programmatic kepi.header('Expires').set( + 60*60*1000);  // good for one hour  // assuming you use Express app.use(kepi.middleware()); 

Note, to mimic Hemlet you’d just go


Compared to Hemlet, its much smaller, IMO simpler, and also allows you to modify headers on the fly, and not just security headers. OTOH, I’m not a security expert and there may be bugs or flaws.

Github repo here

Growth assumption and example of finite (arbitrarily small) time blow up for ODE

Consider the following ODE initial value problem \begin{align*} &\frac{d}{dt}\Phi(t,x) = \boldsymbol{F}(t,\Phi(t,x)), & t \in [0,T], \ \ x \in \mathbb{R}^N,\ &\Phi(0,x) = x, & x \in \mathbb{R}^N. \end{align*}

We say that $ \Phi: [0,T] \times \mathbb{R}^N \to \mathbb{R}^N$ is the flow of the ODE (as in this paper) if it solves it in some sense.

We assume that the vector field $ \boldsymbol{F}:[0,T]\times \mathbb{R}^N \to \mathbb{R}^N$ is Sobolev and such that that $ $ (*) \qquad \frac{|\boldsymbol{F}|}{1+|x|} \in L^1\left([0,T]; L^1(\mathbb{R}^N) \right) + L^1\left([0,T]; L^\infty(\mathbb{R}^N) \right),$ $ that is, there exist \begin{align*} &\boldsymbol{F}_1 \in L^1\left([0,T]; L^1(\mathbb{R}^N) \right)\ &\boldsymbol{F}_2 \in L^1\left([0,T]; L^\infty(\mathbb{R}^N) \right) \end{align*} such that $ $ \frac{\boldsymbol{F}}{1+|x|} = \boldsymbol{F}_1 + \boldsymbol{F}_2.$ $

In an answer to Quantitative finite speed of propagation property for ODE (cone of dependence), it has been remarked that the flow $ \Phi$ can blow up in finite (and arbitrarily small) time if the $ F_1\neq 0$ .

  1. Can you provide an example of such flow that blows up in finite (and arbitrarily small) time?

  2. Why is this not in contrast with the fact that assumption (*) is used in the existence and uniqueness result of Theorem 30 (page 23) of this paper?

  3. In the theorem cited in the previous point, is assumption (*) key for existence or uniqueness?

Small step vs big step semantics for static analysis?

To be an honest question poster I do not yet fully grasp the difference between small step and big step semantics. There was a good discussion here

My question is if I were to do static analysis on a file to determine if it is malware, what would be better to use: small step or big step semantics?

To take this a step further if I am doing dynamic analysis which would be best: small step or big step semantics?

Preferably a simple example as to why would be much appreciated.

How to reduce a big ViewModel into small classes

I’m currently developing an android application using the Androidx Lifedata and data binding components to apply the MVVM pattern.

In the application we have a MapFragment quite similar to google maps. But you can plan routes directly on the map instead of the extra window with the locations on top that you get in google maps. This means the user is most of the time in the same MapFragment, whether he just looks at the map, plans a route, looks at a location, looks at a gpx file or uses active guiding.

Now the ViewModel for this MapFragment has become very large. We are trying to move all the navigation logic etc into UseCases and similar other classes, but the ViewModel still has to consider what happens to the View once these did their calculations. As they depend on the View state I don’t see how I could move them out of the ViewModel. One example:

When long pressing the map, what happens depends on:

  • Whether you have the map for this area: If not it proposes to download it
  • Whether you have nothing selected: It selects the place and shows a loading icon until more information about the location (name, address) are loaded
  • Whether you just tried to select something which is still loading: In this case selecting is not possible
  • Whether already something selected: It removes the old selection and selects the new
  • Whether you are planning a route: It is added as a waypoint if possible, but maybe the route is not possible, then it has to return
  • Whether you are looking at a gpx file: In which case the location is shown but the gpx route stays in view
  • Whether you are currently using the active guidance: In which case it is not possible

These are just small cases, but each of them need to check at least one LiveData value and there are many more ways to select something on the map. Be it from favourites or the search etc. All these need handling for every state of the MapFragment And selecting something is just a small part of the view functionality. Planning a route is quite complicated, furthermore the map should be centered upon whatever the user currently uses. Also it shows some overlays with extra information. These are Views with their own ViewModel, but it still needs to be managed which is currently shown and synchronise the content with the MapFragment, if for example during route planning the user rearranges the Waypoints.

So I wonder what is the best pattern to make this ViewModel class more readable. I assume splitting it into some smaller classes is the best approach. But I’m having trouble doing that, because all these classes again need access to a bunch of view states which are in the ViewModel. Should I just make everything public and basically move single functions into other classes? It seems wrong to me, smells very much like the feature envy anti-pattern. I have one other idea which I will put into an answer to this question, but I am not sure if it is such a good one.

Note: I cannot provide any code, as it is a confidential work project. I am not asking you to do my job, the app works and I am not getting paid to redo my own work (although I might be able to talk them into that if I see a better solution). But first and foremost the software architect in me wants to learn from the experience. So the answer doesn’t have to be specific to this case but generally to how to break a big ViewModel.

Is it possible to mount very small PC-like computers on an electric socket?

I’ve been thinking of getting something like Raspberry Pi (or similar small computer) to use as a print server, but am irked by the prospect of having to find a shelf for it near the electric socket and have a cable dangle to to reach it. Then I got to thinking: Can’t the computer just live on top of the socket? Weak computers these days are pretty small. Even with an AC-to-DC converter, it’s not inconceivable to make one light and small enough to stably be mounted onto a socket.

Is this something realistic? Does such hardware exist? And if so – are these mounting adapters for non-socket-mounted computers, or perhaps computers pre-engineered so that their power converters and case are designed from the get-go to fit a power socket?

Small business owners’ go-to resources for technical advice

Hope you are doing well.

I want to learn from the community that what are the top resources used by small business owners whenever they need any technical advice related to business applications? For this, I have created a small survey, which will not take more than 3 mins of your time, but it will help me immensely to learn technical habits of small business owners.

How do I partition my Xubuntu install to maximize my small hard drive?

I’m installing Xubuntu on a laptop with a 32GB SSD. I want to have my /root partition be the largest possible size. Letting the installer configure partitions automatically leaves a portion of the hard drive unused as “unallocated space” and other partitions oversized so I want to learn how to partition manually and maximize available hard drive space to keep my programs and files. I just want to know which partitions are necessary for installation, flags, etc.

A term for a set of vertices with a small set of neighbors

Given a graph $ G(V,E)$ ans a subset $ S\subseteq V$ , define $ N(S) = $ the set of neighbors of $ S$ .

Is there a standard term for a subset $ S\subseteq V$ for which $ |N(S)|\leq |S|$ ? Is there a standard term for $ |N(S)|< |S|$ ?

I remember once seeing the name expander used for a subset $ S$ for which $ |N(S)|>|S|$ , but I do not find the reference now. Then when $ |N(S)|\leq |S|$ , $ S$ can be called a non-expander, and when $ |N(S)|<|S|$ , $ S$ can be called a contractor. But these are all my inventions. What are the standard terms?