Value flow (and economics) in stacked reinforcement learning systems: agent as reinforcement environment for other agents?

There is evolving notion of stacked reinforcement learning systems, e.g. https://www.ijcai.org/proceedings/2018/0103.pdf – where one RL systems executes actions of the second RL system and it itself executes action of the thrid and the reward and value flows back.

So, one can consider the RL system RLn with:

  • S – space of states;
  • A={Ai, Ainner, Ao} – space of actions which can be divided into 3 subspaces:
    • Ai – actions that RLn exposes to other RL systems (like RLn-1, RLn-2, etc.) and for whome the RLn provides reward to those other systems;
    • Ainner – actions that RLn executes internally under its own will and life;
    • Ao – actions that RLn imports from other RL systems (like RLn+1, RLn+2, etc.) and that RLn executes on the other systems and form which RLn gets some rewards, that can be used for providing rewards for Ai actions.

So, one can consider the network (possibly hierarchical) of RL systems . My questions are:

  • is there some literature that consider such stacks/networks of reinforcement learning systems/environments?
  • is there economic research about value flow, about accumulation of wealth, about starvation and survival proceses and evolution in such stacks/networks of RL systems/environments

Essentially – my question is about RL environments that can function in the role of agents for some other environments and about agents that can function in the role of environemnts for some other agents. How computer science/machine learning/artificial intelligence name such systems, what methods are used for research and how the concepts of economics/governance/preference/utility are used in the research of evolution of such systems?

Finding the nearest agents to a customer using IComparable

I got a C# class:

public class Agent  {     public string AgentId { set; get; }     public double Longitude { set; get; }     public double Latitude { set; get; } } 

There are several agents in several different locations. During the program’s lifetime, they can receive a call to another (latitude, longitude) point which is unknown during runtime. I got an API to calculate the distance in meters between their location and the given point. What I need eventually is to find the 5 closest agents to the given point.

What I did in order to achieve this is adding another class:

public class AgentDistance : Agent, IComparable<AgentDistance> {      public AgentDistance(Agent agent)     {         if(agent == null)         {             throw new Exception("Can't initalize agent with distance since agent is null");         }         this.AgentId = agent.AgentId;         this.Latitude = agent.Latitude;         this.Longitude = agent.Longitude;     }      public double Distance { set; get; }      public int CompareTo(AgentDistance other)     {         if(other == null)         {             return 1;         }         return Distance.CompareTo(other.Distance);     } } 

And the usage:

var agents = db.GetAgents(); if(agents == null || agents.Count == 0) {     Console.WriteLine("No Results"); } List<AgentDistance> agentsWithDistance = new List<AgentDistance>(); foreach(Agent agent in agents) {     double res = ws.GetDistanceMeters(agent.Latitude, agent.Longitude, customerLatitude, customerLongitude);     agentsWithDistance.Add(new AgentDistance(agent) { Distance = res }); } agentsWithDistance.Sort(); for(int i = 0; i < 5; i++) {     Console.WriteLine(string.Format("agent_id:  {0}  distance: {1} m", agentsWithDistance[i].AgentId, agentsWithDistance[i].Distance)); } 

It works, but is there a more elegant way to do it? I’m not sure if adding another class might be a bit redundant since all it does is just adding a property for sorting, but adding the distance property to the Agent class, doesn’t make so much sense.

What are approved French travel agents whose organised trip grants Visa-free entry to French overseas territories

As per wikipedia, citizens from a group of developing countries (China, India etc) enjoy Visa-free entry of up to 15 days for each stay in New Caledonia and French Polynesia, for trips organized by an approved travel agency.

The problem is such travel agencies are nowhere to be found. Wikipedia sourced information from the pages of French government websites. For example, regarding rules of la Nouvelle-Calédonie (New Caledonia), this web page:

https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000024389411

Quote the text from Chine (China) part:

  • pour les séjours d’une durée inférieure à quinze jours pour les ressortissants dont le voyage et le séjour sont organisés par l’intermédiaire d’une agence agréée.

The word “approved”, here agréée, is not to be found again in the same page. It’s left undefined.

To put this discussion to context, I’m searching for an organised tour to New Caledonia for my Chinese parents.

OTRS 6 – Users and Agents authenticate on AD, but now I lost mi Admin user

I am testing OTRS 6 with the ITSM module. Right now I’m only testing it on our DEV environment, so it would not be a problem to ditch everything and start from scratch if it’s the easiest way. I intend to put it to production soon, tho.

I installed it “by the book” using the official docs and it worked like a charm! I have all users authenticating locally using the User Database on MySQL. I had Customers, Agents and Admins all able to authenticate and being presented with the correct user panel.

After that, I have been able to successfully integrate OTRS with my AD, but with a catch: all my AD users are mapped as Customers, all my Domain Admins (who also belong to the OTRS_Admins AD Group) are Agents, and…. I’m left with no account to manage OTRS. No Admins at all.

What should I do? How can I map my Domain Admins to be OTRS Admins and not Agents? How to map some of my Domain Users to be Agents? Am I doing something wrong? I’m completely lost.

The oficial docs don´t help that much, and I couldn’t find on Google anyone with my specific need.

My (redacted) Config.pm:

     $  Self->{'AuthModule'} = 'Kernel::System::Auth::LDAP';           ### OTRS Admin Auth         ###          $  Self->{'AuthModule::LDAP::Host'} = '192.168.179.2';     # AD Server         $  Self->{'AuthModule::LDAP::BaseDN'} = 'dc=test,DC=local'; # Domain         $  Self->{'AuthModule::LDAP::UID'} = 'sAMAccountName';         $  Self->{'AuthModule::LDAP::GroupDN'} = 'CN=OTRS_Admins,CN=Users,DC=test,DC=local';   #OTRS Admin group         $  Self->{'AuthModule::LDAP::AccessAttr'} = 'member';         $  Self->{'AuthModule::LDAP::UserAttr'} = 'DN';         $  Self->{'AuthModule::LDAP::SearchUserDN'} = 'svc_otrs'; #OTRS service user         $  Self->{'AuthModule::LDAP::SearchUserPw'} = 'Passw0rd'; #And its passwird         $  Self->{'AuthModule::LDAP::AlwaysFilter'} = '';         $  Self->{'AuthModule::LDAP::Params'} = {                           port => 389,                           timeout => 120,                           async => 0,                           version => 3,                           sscope => 'sub'                         };          ### User Sync         ### AD <==> DB OTRS (MySQL)         $  Self->{'AuthSyncModule'} = 'Kernel::System::Auth::Sync::LDAP';         $  Self->{'AuthSyncModule::LDAP::Host'} = '192.168.179.2';      # AD SRV         $  Self->{'AuthSyncModule::LDAP::BaseDN'} = 'dc=test,DC=local'; # Domain         $  Self->{'AuthSyncModule::LDAP::UID'} = 'sAMAccountName';         $  Self->{'AuthSyncModule::LDAP::SearchUserDN'} = 'svc_otrs';                  $  Self->{'AuthSyncModule::LDAP::SearchUserPw'} = 'Passw0rd';             $  Self->{'AuthSyncModule::LDAP::UserSyncMap'} = {         # DB -> LDAP         UserFirstname => 'givenName',         UserLastname => 'sn',         UserEmail => 'mail',         };          $  Self->{'AuthSyncModule::LDAP::UserSyncInitialGroups'} = [         'users', 'basic_admin',         ];   $  Self->{'Customer::AuthModule'} = 'Kernel::System::CustomerAuth::LDAP';     $  Self->{'Customer::AuthModule::LDAP::Host'} = '192.168.179.2';     $  Self->{'Customer::AuthModule::LDAP::BaseDN'} = 'dc=test,DC=local';         $  Self->{'Customer::AuthModule::LDAP::UID'} = 'sAMAccountName';     $  Self->{'Customer::AuthModule::LDAP::SearchUserDN'} = 'svc_otrs';          $  Self->{'Customer::AuthModule::LDAP::SearchUserPw'} = 'Passw0rd';          $  Self->{CustomerUser} = {     Module => 'Kernel::System::CustomerUser::LDAP',     Params => {     Host => '192.168.179.2',     # AD Server     BaseDN => 'dc=test,DC=local',      #Domain     SSCOPE => 'sub',     UserDN =>'svc_otrs',     #OTRS Service User     UserPw => 'Passw0rd',    #its password     AlwaysFilter => '(&(samAccountType=805306368)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))',     SourceCharset => 'utf-8',     DestCharset => 'utf-8',     },      CustomerKey => 'sAMAccountName',     CustomerID => 'mail',     CustomerUserListFields => ['sAMAccountName', 'cn', 'mail'],     CustomerUserSearchFields => ['sAMAccountName', 'cn', 'mail'],     CustomerUserSearchPrefix => '',     CustomerUserSearchSuffix => '*',     CustomerUserSearchListLimit => 10000,     CustomerUserPostMasterSearchFields => ['mail'],     CustomerUserNameFields => ['givenname', 'sn'],     Map => [     # note: Login, Email and CustomerID needed!     #[ 'UserSalutation', 'Title', 'title', 1, 0, 'var' ],     [ 'UserFirstname', 'Firstname', 'givenname', 1, 1, 'var' ],     [ 'UserLastname', 'Lastname', 'sn', 1, 1, 'var' ],     [ 'UserLogin', 'Login', 'sAMAccountName', 1, 1, 'var' ],     [ 'UserEmail', 'Email', 'mail', 1, 1, 'var' ],     [ 'UserCustomerID', 'CustomerID', 'mail', 0, 1, 'var' ],     [ 'UserPhone', 'Phone', 'telephonenumber', 1, 0, 'var' ],     #[ 'UserAddress', 'Address', 'postaladdress', 1, 0, 'var' ],     #[ 'UserComment', 'Comment', 'description', 1, 0, 'var' ],     ],     }; 

What Exploit Are These User Agents Trying to Use?

I just looked at my user agent tracking page on my site (archived on Yandex) and I noticed these user agents. I believe they are an attempt to exploit my server (NGinx with PHP). The 1 in front of it is just how many times the user agent was seen in the NGinx log. These are also shortened user agents and not long ones like Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36. I no longer have access to the logs as I presume this occurred sometime in January or February (my oldest logs are in March and I created the site in January).

1 Mozilla/5.9}print(238947899389478923-34567343546345);{ 1 Mozilla/5.9{$  {print(238947899389478923-34567343546345)}} 1 Mozilla/5.9\x22{$  {print(238947899389478923-34567343546345)}}\x22 1 Mozilla/5.9\x22];print(238947899389478923-34567343546345);// 1 Mozilla/5.9\x22 

What exploit was attempted and how can I test to ensure these exploits are not usable?

Multiple Tasks and Multiple Agents problem in Linear Programming

The assignment is definted as follows:

We have a set of 15 tasks. We are given a set number of workers needed for each task and the time they need to complete them. Every worker can work on only one task at the time and once a task starts it cant stop.

We have 24 weeks to complete all tasks, no order specified. We can’t do them one after another since that excedes the 24 weeks.

We need to find the minimum number of new workers to hire required to complete all tasks in the given timeframe. We start with some workers already but they are not enough for our workload.

The problem needs to be solved using Linear Programming. Any insight would be appreciated! I am totally confused about what to set as parameters, objective function, constraints etc…

Whitelisting browser agents in ModSecurrity

I’m using ModSecurity WAF with Apache. I wanted to know if there’s a way I can restrict to only few browser agent type? I want to do this for two reasons:

  1. Force the users to use an updated browser.
  2. Deny reconnaissance. Thus most bots on the internet use useragent that doesn’t reflect a browser or an useragent that uses an outdated browser identifier.

Hence for example, I want to block chrome 69 and below, is there a way that i can use ModSecurity (including the CRS ruleset) to block anything but the following:

AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36  

I have removed the OS part to ensure any OS can work.

Survey structure for Live Chat Transfer between Agents

I’m developing a live chat application for my site and want to include a post-chat survey to gather feedback on my Agents.

However, I have two types of Agents, for dealing with different queries, and a person engaging in the Chat may be transferred at any time to another Agent from the pool.

I’m concious of the fact that if a person is engaged in the Chat, speaking to Agent A, and then gets transferred to Agent B, that their answers in the Post-Chat Survey may reflect only one or the other, depending on how the Chat Survey is worded.

From doing research, the optimum amount of questions to ask is three. My question to you is, how do I structure the survey to ensure the User answers it with respect to both Agents?

Should I refer to “1st Agent”, “2nd Agent”, should I double up on the questions, should I include a Survey in between the transfer?