Does the leader election problem only apply to process replication/redundancy?

The leader election problem is said to be

In distributed computing, leader election is the process of designating a single process as the organizer of some task distributed among several computers (nodes). Before the task is begun, all network nodes are either unaware which node will serve as the “leader” (or coordinator) of the task, or unable to communicate with the current coordinator. After a leader election algorithm has been run, however, each node throughout the network recognizes a particular, unique node as the task leader.

As far as the leader’s responsibility is concerned, are the candidates identical?

Does the leader election problem apply only to processes which are replicas, i.e. to process redundancy? In other words, does the leader election problem not apply to processes which are not replicas, i.e. processes without redundancy?

My confusions come from:

  • Design Data Intensive Programs by Kleppmann introduces the concept of “leader” in Chapter 5 Replication, while the election problem in Chapter 9: Consistency and Consensus.

  • Distributed Systems by Coulouris introduces the concept of “primary replica manager” and “backup replica manager” in Chapter 18 Replication, while the election problem in Chapter 15: Coordination and Agreement.

So I wonder if the election problem applies only to replication (more specifically, process replication), or also to cases which don’t involve replication?

Thanks.

Service temporarily unavailable due to an ongoing leader election. Please refresh

I have three physical nodes with docker installed on each of them. I configured Mesos,Flink,Zookeeper,Marathon and Hadoop on docker of each node. When ever I changed a docker, unfortunately, Flink UI ended up with this error:

Service temporarily unavailable due to an ongoing leader election. Please refresh.

I have searched a lot and not understood what is wrong with configurations.

Would you please guide me how to solve the problem?

Thank you in advance.

PHP to cast votes on an open-to-public election

I am creating an online voting platform where users could vote on publicly-held polls. It uses Kreait Firebase library (as Google didn’t create a PHP library for Firebase Admin).

I am very new to PHP and am not sure about how I should format my PHP code. It is messy. Please help me improve the code.

<?php  $  request = json_decode(file_get_contents("php://input"));  if (!isset($  request->pollPath) || !isset($  request->votes)) {   print_r($  request);   echo "pollPath and/or votes have not been sent!";   exit; }  require __DIR__.'/../vendor/autoload.php';  use Kreait\Firebase\Factory; use Kreait\Firebase\ServiceAccount;  $  pollPath = $  request->pollPath; $  votes = $  request->votes;  $  serviceAccount = ServiceAccount::fromJsonFile(__DIR__.'/../service.credentials.json'); $  firebase = (new Factory)   ->withServiceAccount($  serviceAccount)   ->asUser('brokerServer')   ->create();  $  pollsDatabase = $  firebase->getDatabase(); $  electionRef = $  pollsDatabase->getReference('publicPolls/' . $  request->pollPath);  $  postsRef = $  electionRef->getChild('posts'); $  postsCount = $  postsRef->getSnapshot()->numChildren();  if ($  postsCount != count($  request->votes)) {   echo "Array of votes doesn't match the election's position count";   exit; }  $  votesRef = $  electionRef->getChild('votes'); $  electionVotes = $  votesRef->getValue();  if (!isset($  electionVotes))   $  electionVotes = [];  $  postsValue = $  postsRef->getValue(); for ($  i = 0; $  i  < $  postsCount; $  i++) {   $  postValue = &$  postsValue[$  i];   $  postVotes = &$  electionVotes[$  i];    if (!isset($  postVotes)) {     $  postVotes = [];     $  postCands = count($  postValue['candidates']);      for ($  j = 0; $  j < $  postCands; $  j++)       $  postVotes[$  postValue['candidates'][$  j]] = 0;   }    if (isset($  request->votes[$  i]))     $  postVotes[$  request->votes[$  i]] += 1; }  $  votesRef->set($  electionVotes); print_r($  electionVotes); echo "done"; 

The code basically gets a JSON fetch request from the client with the path of the election (in Firebase) and an array of votes (mapping index of positions in election to the candidates chosen).

What is the difference between Consensus and Leader Election problems?

According to this paper written by Lamport, `selecting a unique leader is equivalent to solving the consensus problem’.

Based on the above quote, my question is: What is the difference between consensus and leader election problems?

My answer: I think that there is no theoretical difference between them. They are two terms with the same meaning, but they are used in different context. When the context is related to data stores, replication, or related subjects, ‘consensus’ is used, otherwise, ‘leader election’ is used.


Note: In the above paragraph, ‘theoretical‘ means the way of defining a true consensus and leader election algorithm. A true consensus and leader election algorithm should satisfy three properties– termination, agreement, and integrity.

Architecting a distributed file processing system with leadership election

I’m in the planning phases of trying to build a distributed file processing system in Java and I’m looking for feedback and advice:

Problem : There are a large number of files continuously posted on an FTP server that we need to grab, process and pass on.

Solution Idea: A master node will look for new files on an FTP server, and assign processing work to child nodes. The master node will send a JMS message to the child telling them which file to process and the child will send a response back when it is done, and ask for more work.

If the master node goes down for some reason, one of the child nodes should presume the role of master. My idea for implementing this was to have a “lock” collection in a MongoDB that contains info about the master node, as well as a lock expiration time. Every 15 seconds or so the master node will refresh their lock and update the expiration time to 30 seconds in the future. If the child nodes sees the lock is expired, one of them will assign itself as the master node.

I’m looking for feedback on this design, and wondering if anyone has advice on improvements/java frameworks or tools that already exist that I can leverage for something like this.

Thanks!