‘n’ number of CAML queries in a single executeQueryAsync function

How to execute ‘n’ number of CAML queries to get a different list of objects? We can use multiple clientContext.load() if we have a defined number of queries like clientContext.load(collListItem); clientContext.load(collListItem1);

In my scenario, the queries are generated and cannot determine the number of CAML queries needs to be executed. Is this possible?

clientContext.load(collListItem1); clientContext.load(collListItem2); ... ... clientContext.load(collListItemN); 

Data structure & algorithms for super-interval queries on intervals with small integer ends

I would like to have an online data structure that supports inserting an interval, and given a query interval $ I_q=[l_q,h_q]$ answer if $ I_q$ is contained at some interval of the data structure, i.e. if $ I_q$ is a super-interval of some interval of the structure (so the answer of the query is just boolean, no need to output all such intervals on the structure) at the fastest possible time complexity.

I have searched for such a combination, and found out that probably an Interval Tree would be appropriate for my situation, with $ O(\log n)$ interval insertion and overlapping intervals query (so it’s not exactly my desired query, but I think that it could possibly be turned to it. Also I can avoid the output complexity dependence, since the desired output is boolean and therefore on the first match I would know the answer is true).

Furthermore, here (https://en.wikipedia.org/wiki/Interval_tree) it is stated that:

If the endpoints of intervals are within a small integer range (e.g., in the range [1,…,O(n)]), faster data structures exist with preprocessing time O(n) and query time O(1+m) for reporting m intervals containing a given query point.

Since I also can guarantee that both interval ends are going to be small integers (i.e. not floats, but natural integers up to $ \approx 10^6$ ), what would be the best data-structure/algorithmic way (considering time-complexity of the above two operations) to implement those two operations I would like to have?

If the fastest time-complexity would be an Interval Tree, then how can I modify the overlapping-intervals query to support my query in $ O(\log n)$ time, i.e. not $ O(k\cdot\log n)$ where $ k$ is $ I_q$ ‘s range? However, I am quite interested in the above quoted passage, and how I could possibly (with another data structure, maybe?) manage such a fast complexity, so in that case Interval Trees wouldn’t matter.

Note: In my attempt to test such an algorithm and speed on an Interval Tree, I have found out the following library: https://github.com/chaimleib/intervaltree/blob/master/intervaltree/intervaltree.py where a similar query seems to be implemented on the envelop query with a time-complexity of $ O(m+k\cdot\log n)$ , where “$ n$ = size of the tree, $ m$ = number of matches, $ k$ = size of the search range”, which however is not as fast as I would like (especially considering the $ k$ multiplying factor).

Constructing a directed graph for O(1) queries

I need help to design a data structure for a directed graph with the following properties:

  1. Initialization should be done in O(1) time.
  2. AddVertex(id1,id2,…idK) – Add a new vertex to the graph. The new vertex is added with id=N+1 where N is the number of existing vertices. id1…idK are the neighbors of the new vertex, such that there is an outgoing edge from the new added vertex to each of the vertices id1 .. idK. This should be done in O(1) time.
  3. GetNumberOfNeighbors(id) – Return the number of outgoing edges for a vertex with a given id (id is in 1..N). This should be done in O(1) time. All O(1) times are for worse case.

Thanks allot

How to decide if two conjunctive queries cannot have any result in common

Consider the following queries:

Q1: age > 18 & age < 24 & gender = ‘male’

Q2: age > 25 & gender = ‘male’ & major = ‘CS’

Q3: age = 20 & major = ‘CS’

Clearly, given any database instance $ D$ , we can decide $ Q1(D) \cap Q2(D) = \emptyset$ and $ Q2(D) \cap Q3(D) = \emptyset$ . However, we cannot decide about the result of $ Q1(D) \cap Q3(D)$ .

I’m aware of query containment, in where the result of one query always is a subset of the results of another query. Also, I’m aware of query equivalence, in where the results of two queries should be exactly the same given any database instance.

However, I’m looking for an algorithm/terminology for my case, where before evaluating the two queries, I can decide the intersection of their results are always empty. Indeed, due to conflicting selection conditions, they cannot have any common result.

How to stop continuous concurrent local DNS queries to dnsmasq

I set up a DNS server using dnsmasq, but it seems that it doesn’t work properly. The networking delay is up to hundreds of millisecs.

PING [server] ([server]) 56(84) bytes of data. 64 bytes from [server]: icmp_seq=1 ttl=50 time=583 ms 64 bytes from [server]: icmp_seq=2 ttl=50 time=583 ms 64 bytes from [server]: icmp_seq=3 ttl=50 time=583 ms 64 bytes from [server]: icmp_seq=4 ttl=50 time=583 ms 64 bytes from [server]: icmp_seq=5 ttl=50 time=583 ms 64 bytes from [server]: icmp_seq=6 ttl=50 time=583 ms 64 bytes from [server]: icmp_seq=7 ttl=50 time=583 ms 64 bytes from [server]: icmp_seq=8 ttl=50 time=583 ms 

Then soon I discovered that it’ll turn all right with using the default resolver systemd-resolved. By checking the log, I got the messages here below:

Jul 27 13:32:53 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:53 dnsmasq[3780]: forwarded ntp.ubuntu.com to 127.0.0.53 Jul 27 13:32:53 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:53 dnsmasq[3780]: forwarded ntp.ubuntu.com to 127.0.0.53 [countless records repeating these above...] Jul 27 13:32:53 dnsmasq[3780]: Maximum number of concurrent DNS queries reached (max: 150) Jul 27 13:32:54 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:54 dnsmasq[3780]: forwarded ntp.ubuntu.com to 127.0.0.53 [...] [probably the sigterm was sent here?] Jul 27 13:32:57 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:57 dnsmasq[3780]: forwarded ntp.ubuntu.com to 127.0.0.53 Jul 27 13:32:58 dnsmasq[3780]: no servers found in /run/dnsmasq/resolv.conf, will retry Jul 27 13:32:58 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:58 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:58 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:58 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:58 dnsmasq[3780]: query[A] ntp.ubuntu.com from 127.0.0.1 Jul 27 13:32:58 dnsmasq[3780]: exiting on receipt of SIGTERM 

So.. How to resolve this problem which makes me crazy, plz 🙁

The content from dnsmasq.conf is here:

strict-order resolv-file=/etc/resolv.conf.dnsmasq listen-address=0.0.0.0 server=/aliyuncs.com/100.100.2.136 server=/aliyuncs.com/100.100.2.138 server=/*.cn/100.100.2.136 server=/*.cn/100.100.2.138  bogus-nxdomain=100.100.2.136 bogus-nxdomain=100.100.2.138  log-queries log-facility=/var/log/dnsmasq/dnsmasq.log log-async=50  #EOF 

And resolv.conf.dnsmasq:

nameserver 100.100.2.136 nameserver 100.100.2.138 nameserver 208.67.222.123 nameserver 208.67.220.123 nameserver 8.8.8.8 nameserver 8.8.4.4 nameserver 1.2.4.8 nameserver 210.2.4.8 nameserver 208.67.222.222 nameserver 208.67.220.220 

resolv.conf:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8) #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN # 127.0.0.53 is the systemd-resolved stub resolver. # run "systemd-resolve --status" to see details about the actual nameservers.  nameserver 127.0.0.53 options timeout:2 attempts:3 rotate single-request-reopen 

Magento 2 generates thousands of temp table queries on product update

I have a Magento 2.2.5 store with ~20k products and ~250 categories arranged in a tree of maximum 2 sub categories.

One of the top categories has 5k products and when I try to remove a product from the category, or remove the category for a specific product the server takes more than 2 minutes and times out.

After reading the general_log, I saw thousands of queries like this:

SELECT `e`.* FROM `tmp_select_e5e424ef98e4a8a4026e18df9d8088e0` AS `e` WHERE (hash_key = '1_44083') 

peaking the database server to 100%: enter image description here

I think this has something to do with url_rewrites, but I can’t seem to understand how is this a problem with only 5k products.

Are there any configurations I can set maybe loosing functionalities, to be able to do something as simple as removing a product from a category?

Thanks a lot.

Is it better to have a single query that returns all related entities, or separate queries for each case?


Summary

When writing methods to query for related entities, is it better to have a single method for all related entities or separate methods for each combination?

Full Problem

I’m writing repository classes for a .NET MVC project where each repository is responsible for querying for a specific entity type. I’m using Dapper, so all of the queries are written by hand. The repositories all implement an IRepository<TEntity> interface, which has method TEntity FindById(id).

Fetching just the entity is obviously very straightforward, but I’m faced with a dilemma when it comes to fetching the entity’s children and grandchildren and so forth. Often times, for an entity to be usable, it needs some or all of its descendants. I initially wrote FindById to return the entity with all descendants, which satisfies all use cases, but is also wasteful in many where I only need several descendants.

I wrote out individual methods for fetching various combinations. FindByIdWithChildA or FindByIdWithChildBAndGrandChildC and such. The problem here is that the method names keep getting longer and longer, which makes them a pain to read, and also wastes a lot of space in code.

My next idea was to accept various enum flag values designating which descendants to return. This worked fine for the immediate children, but by the time I got to grandchildren and beyond I had so many possible combinations to implement, and many of them didn’t make sense or wouldn’t be used.

By the time I have all this boilerplate written out, it almost feels like I’m writing a bad ORM instead of repositories. This has certainly slowed down development dramatically, and it has taken focus away from actually implementing new features.

The obvious wrong solution is to not use SQL joins and have the repositories splice together entities and collections of entities using LINQ. Using LINQ with EntityFramework could be an option, but I don’t think that the solution is to switch to another library.

Should I just simplify and have the single method/query return everything? If not, what is the best way to concisely implement separate methods?

This answer indicated that a FindByIdWithChild type method is normal, but I can’t imagine that being good practice for complex queries.

Clarification

So my entities are Sport, Division, SkillLevel, and TeamType. Each Sport has a list of Divisions, SkillLevels, and TeamTypes. Each Division has a single TeamType and a list of SkillLevels.

Suppose I want a Division with its parent Sport, its TeamType, and all of its SkillLevels. Should I do:

  1. DivisionRepository.FindById(id) which returns all of these properties, ignoring any case where I want just a Division or a Division and Sport, Division and TeamType, etc.

  2. DivisionRepository.FindById(id, BitFlagQueryScope) which has a big mess of an implementation to resolve all the possible flags, some of which might not be needed.

  3. DivisionRepository.FindByIdWithSportAndTeamTypeAndSkillLevels(id) and have one of these methods for every combination I need.

In all three cases, I am making one trip to the database. It’s just a matter of how I structure my code.

Additional example, suppose I want a Sport with just its TeamTypes. Elsewhere I will need a Sport with its SkillLevels, etc. Same questions as above. I’m only making one database trip, but should I use different code for these different queries, or just grab everything and discard/not use the excess?

Additional Clarification

In case there’s something wrong with my design overall, here’s how I’m using the entities:

Action(string id) {     var sport = SportRepository.FindById(id); // This needs to either return everything associated with the entity or be changed to allow specification     var model = new SportDisplayModel(sport);     return View(model); } 

I have two models, $ Entity$ DisplayModel, and $ Entity$ EditModel, and I use them for every view.

Suspicious SMTP server client activity – non-ASCII queries

I’ve noticed some strange activities from some remote host connected to our SMTP mail server. Queries contains non-ASCII characters, below is the copy of one of them (to each query our server responded with “command not implemented”):

..._)....pz4.H.T7.&...4..rШ./.+.... ..

The hex representation is:

20 00 00 00 5F 29 00 00 00 00 70 7A 34 00 48 00 54 37 00 26 00 00 00 34 00 00 72 D8 00 2F 00 2B 00 00 00 00 20 20 20 20 00 00 

What is the goal of such queries? Is this some kind of security attack / searching for “black doors” or something?