computing sum from buffer in C pthread producer consumer

I want to calculate the sum of the elements in a buffer, basicly one thread compute a parcial sum, and the other thread reads the buffer and sum the elements. Also, is possible to terminate the consumer thread other way?

but I cannot get it to work.

# include <stdio.h> #include <stdlib.h> #include <unistd.h> # include <pthread.h>  void *tarefa_calculadora(); void *tarefa_somadora(); int nt; int tamanho_buffer; int items_buffer=0; char *buffer; int sinal_saida; int matriz_soma_global; int soma_global;  pthread_cond_t buffer_nao_cheio=PTHREAD_COND_INITIALIZER; pthread_cond_t buffer_nao_vazio=PTHREAD_COND_INITIALIZER; pthread_mutex_t mVar=PTHREAD_MUTEX_INITIALIZER; 

the main gunctions seems ok for me. No problems here.

int main(int argc, char ** argv) {          pthread_t calculadora_id,somadora_id;      nt=atoi(argv[1]);     tamanho_buffer=nt;      buffer=(char *) malloc(sizeof(char) * tamanho_buffer);                  pthread_create(&calculadora_id,NULL,tarefa_calculadora,NULL);     pthread_create(&somadora_id,NULL,tarefa_somadora,NULL);       pthread_join(calculadora_id,NULL);     pthread_join(somadora_id,NULL);        return 0; } 

This is the producer function, it also seems ok for me. the only question is im thinking in a way to signal the consumer thread that the consumer threads are finished. I have used the return to add 1 to an intiger and then if the consumer thread sees that the intiger is one the consumer can end.

void *tarefa_calculadora() {          int i, parcial_sum;      for(i=0;i<11;i++)     {         pthread_mutex_lock(&mVar);          if(items_buffer==tamanho_buffer){                                     pthread_cond_wait(&buffer_nao_cheio,&mVar);         }          parcial_sum=i*i;          buffer[items_buffer++]= parcial_sum;          printf("Soma parcial : %d \n",parcial_sum);          pthread_mutex_unlock(&mVar);         pthread_cond_signal(&buffer_nao_vazio);             }      return sinal_saida=1;    } 

this is the consumert thread , im having problems here I cant calculate the sum of the parcial sumas that are in the buffer it seems logical to me to do this but the programs is not working. Also i would like to use sched_yield to free the thread when is done.

    void *tarefa_somadora() {     int i;       for(i=0;i<12;i++)     {         pthread_mutex_lock(&mVar);          if(items_buffer==-1){                pthread_cond_wait(&buffer_nao_vazio,&mVar);         }          for(int z=0;i<nt;i++){             soma_global+=buffer[z];         }          buffer[0]=0;          printf("Soma Global : %d \n",soma_global);                 pthread_mutex_unlock(&mVar);                 pthread_cond_signal(&buffer_nao_cheio);                          if(sinal_saida==1){pthread_exit(NULL);}      }     return NULL;  } 

RabbitMQ Consumer Architecture

I have been investigating using RabbitMQ for a publish/subscribe pattern across micro-services where aysnc calls are appropriate but I am having trouble understanding some concepts and could not find any info in the manual.

I’ll outline a basic example of how I wish to use it below.

I have a core service for user management. When a user is created I wish to publish an event such as user_created I would like this event to be consumed by multiple different services. For example

  • A communications service which handles the sending of the welcome email
  • A marketing service which takes the user data and publishes it to a third party data bucket
  • A finance service which sets up a new customer statement/account

Each of the services above are their own separate codebase in a few different languages/frameworks.

From what I can tell of the docs for this kind of I will need an exchange where I publish a message like user_created along with any data I wish to send (like the user object)

On each of my services I then set up a consumer script which connects to the exchange and creates a queue and consumes any messages sent.

If my understanding so far is correct my main question comes from how to handle these consumers on the services end. If I am using acknowledgement my understanding is that even if one of my consumers dies the queue and message will not be deleted. Both will persist and wait for the a new consumer indefinitely.

Any number of services will presumably want to consume lots of different messages. Is it okay to have lots of different consumers constantly running?

Primarily the services use PHP so these consumers would maybe look like the below (for the comms service)

sendWelcomeEmail.php sendGoodbyeEmail.php sendInvoice.php 

So presumably I set up each of these scripts as directed on the tutorials (like here and run the scripts as part of the deployment process. I would then need something like supervisor to make sure the consumer is running and if it is not restart it. This could be quite cumbersome having to set a supervisor up for each consumer. Are there better ways to handle things?

[GET] Digital Marketing Analytics: Making Sense of Consumer Data in a Digital World (Que Biz-Tech)

Good news: neither have your competitors. It's hard! But digital marketing analytics is 100% doable, it offers colossal opportunities, and all of the data is accessible to you. Chuck Hemann and Ken Burbary will help you chop the problem down to size, solve every piece of the puzzle, and integrate a virtually frictionless system for moving from data to decision, action to results! Scope it out, pick your tools, learn to listen, get the metrics right, and then distill your digital data…

[GET] Digital Marketing Analytics: Making Sense of Consumer Data in a Digital World (Que Biz-Tech)

How to show a consumer products in a eCommerce Site with only filters?

I have an eCommerce site that sells baby clothes. When i think of the different ways i can arrange/organize the main menu i get frustrated on trying to make sense. I start to think of how a physical store is organized. Usually they are separated by gender and age simultaneously:

  • Women (Gender Criteria)
  • Men (Gender Criteria)
  • Kids (Age Criteria)

This really translates in this sections:

  • Adult – Women (Gender and Age Criteria)
  • Adult – Men (Gender and Age Criteria)
  • Kids – Newborns – Boy (Gender and Age Criteria)
  • Kids – Newborns – Girl (Gender and Age Criteria)
  • Kids – Babies – Boy (Gender and Age Criteria)
  • Kids – Babies – Girl (Gender and Age Criteria)
  • Kids – Adolescence – Boy (Gender and Age Criteria)
  • Kids – Adolescence – Girl (Gender and Age Criteria)

So age and gender are the two main filters that people use to start browsing clothes on a physical store and also on the web. I understand this, they are in fact the two most probable mutually exclusive criteria: “Someone who is looking for women clothes most probably isn’t interested in simultaneously looking for man clothes”, “Someone who looks for adult clothes isn’t interested in simultaneously looking for kids clothes”.

But in fact they are just two characteristics as any other like size, color, price or even like season, rating, style, etc. In a eCommerce site usually there is a main characteristic that is present on a menu and then a set of filters appear after the first selection. Example:

Main Menu Ladies | Men | Kids | Home

After selecting Men we get something like this, a product area which is already filtered for men clothes and a set of additional filters at the left side bar.


Other Examples are just age like:

Main Menu

Adult | Kids | Babies

But this isn’t really necessary in a web environment, people could just enter the store and see all products and then have the filters at their disposal, including the typically “man” and “women” filter that is commonly showed on a main menu (in my case “girl”, “boy” and “unisex”). This is helpful in my case where i think soon to be mothers would like to browse kids clothes without having to limit the browsing to a gender or age. Why? Because the gender of the baby may be unknown and although they could browse the “unisex” department they may also like to see the options for boy and girl simultaneous. The may have twins, a boy and a girl, and would rather shop for both of them at a time. They may also like to browse clothes for 3 month old and 6 month old at the same time, babies grow very fast and mothers sometimes purchase clothes for different ages. In conclusion, age and gender are not as mutually exclusive in babies.

My options of doing this?

Option 1 Create a menu with a link to the “Store”. In this case i would have a main menu with links like this: Store | Contacts | Help

Then all the filters, including the typical “boy”, “girl” and “unisex” would appear on the side bar.

Option 2 Create a menu with the typical “boy”, “girl” and “unisex” and then have the same filter appear on the left sidebar for filtering. Maybe this would be the more UX friendly option because i would present this in a standard way but i would leave the option for browsing two or more of the gender options (“boy”, “girl”, “unisex”)

None of the options seems very clean, i don’t know if there are any other standard ways of doing this. So how do you think is the best option, that doesn’t confuse the user and adds value, to take the costumer to an all filter based navigation without the need for him to select any criteria at first? Thanks

Consumer “async.operations.all” skipped as required connection “amqp” is not configured. Unknown connection name amqp

Since i upgraded from Magento 2.2.7 to Magento 2.3.1 i am getting this message in the system.log file:

main.INFO: Consumer “async.operations.all” skipped as required connection “amqp” is not configured. Unknown connection name amqp [] []

where in the settings this could be configured or disabled?

Configure spark as kafka consumer issue SCALA

hello everybody here is my code i am trying to configure spark as kafka consumer, but i got error exception first issue that web UI bind to or a nat ip:4040 which i can’t find in browser sencond issue i will write it in the bottom thanks for your help:


import org.apache.spark.sql.functions._ import org.apache.spark.sql.SparkSession object Teal extends App{ val spark = SparkSession .builder() .master(“local[]”) .appName(“teal”) .getOrCreate() import spark.implicits._ val df = spark.readStream .format(“kafka”) //.setMaster(“local[]”) .option(“kafka.bootstrap.servers”, “”) .option(“subscribe”, “test”) .load() df.selectExpr(“CAST(key AS STRING)”, “CAST(value AS STRING)”) .as[(String, String)] val query = df.writeStream .outputMode(“complete”) .format(“console”) .start() }


issue is :Exception in thread “main” org.apache.spark.sql.AnalysisException: Failed to find data source: kafka. Please deploy the application as per the deployment section of “Structured Streaming + Kafka Integration Guide”.; at org.apache.spark.sql.execution.datasources.DataSource$ .lookupDataSource(DataSource.scala:652) at org.apache.spark.sql.streaming.DataStreamReader.load(DataStreamReader.scala:161)

C++ Producer / Consumer multi threads pattern with maximum container size

I’ve just written a Producer / Consumer pattern templatized that should work in multi threads environment (IE producer(s) and consumer(s) in separate threads).

The original use case is one producer and one consumer in separate thread with a limit of container size.

It should work for a multi producers / consumers case though.

Standard used is C++ 14

My question is about the correctness of the std::unique_lock<std::mutex> instructions and synchronizer.wait ones.

I have also a question about the forward declaration of templates, is there a way to avoid it without having a compiler error in the linking step ?

And last question about the handle of the container (the list), it must become the owner of the elements and handle its life time so what’s the method signature I have to use (const T &e, T e, T &&e, …) ? Or even std::move and implementation of perfect forwarding, I don’t know.


#ifndef CUDA_HARDWARE_VIDEO_READER_PRODUCERCONSUMER_HPP #define CUDA_HARDWARE_VIDEO_READER_PRODUCERCONSUMER_HPP  #include <condition_variable> #include <mutex> #include <thread> #include <queue>  template<typename T> class ProducerConsumer {     public:         explicit ProducerConsumer(ulong maxSizeInBytes);          void produce(const T &element);         T    consume();         void producerIsOver();         bool isFinished();      private:         std::queue<T>           queue;         std::condition_variable synchronizer;         std::mutex              mutex;         bool                    producerOver;         ulong                   maxSizeInBytes; };  #endif //CUDA_HARDWARE_VIDEO_READER_PRODUCERCONSUMER_HPP  


#include "ProducerConsumer.hpp"  /**  * @brief Constructor  *  * @tparam T - The type of element's container  * @param maxSizeInMB - The maximum size the container can have in bytes  */ template<typename T> ProducerConsumer<T>::ProducerConsumer(ulong maxSizeInBytes): maxSizeInBytes(maxSizeInBytes), producerOver(false) { }  /**  * @brief Producer - add element in the list  *  * @tparam T - The type of the element  * @param element - The element to add  */ template<typename T> void ProducerConsumer<T>::produce(const T &element) {     std::unique_lock<std::mutex> lock(mutex);      // Block the thread until the size of the queue allows to add more elements in it     if (sizeof(T) * (queue.size() + 1) > maxSizeInBytes) {         synchronizer.wait(lock, [&]() {             return sizeof(T) * (queue.size() + 1) <= maxSizeInBytes;         });     }      queue.push(element);     synchronizer.notify_one(); }  /**  * @brief Consumer - Get an element of the list and wait for it if the list is empty  *  * @tparam T - The type of the element  *  * @return The element  */ template<typename T> T ProducerConsumer<T>::consume() {     std::unique_lock<std::mutex> lock(mutex);      // Block the thread until the queue is empty and the producer has finished to fill the queue     if (queue.empty() && !producerOver) {         synchronizer.wait(lock, [&]() {             return !queue.empty() || producerOver;         });     }      T element = queue.front();      queue.pop();     // Wake up the producer if he's waiting     synchronizer.notify_all();      return element; }  /**  * @brief Tells the producer has finished to fill the container  *  * @tparam T - The type of element's container  */ template<typename T> void ProducerConsumer<T>::producerIsOver() {     // @todo Useless ?     std::unique_lock<std::mutex> lock(mutex);      producerOver = true; }  /**  * @brief Tells if the whole process is over (IE producer has finished to fill the container and the container is empty)  *  * @tparam T - The type of element's container  *  * @return True if the whole process is over, false otherwise  */ template<typename T> bool ProducerConsumer<T>::isFinished() {     std::unique_lock<std::mutex> lock(mutex);      return producerOver && queue.empty(); }  // @todo Annoying forward declaration there ... template class ProducerConsumer<long double>;  


#include <iostream> #include <random> #include <ctime> #include <unistd.h> #include "ProducerConsumer.hpp"  /**  * @brief Test the producer / consumer pattern that preserves order and is limited in size  *        Required output is :  *        Consume 1,  *        Consume 2,  *        Consume 3,  *        ...,  *        Consume 20  *  * @return EXIT_SUCCESS if no error occurred  */ int main() {     ProducerConsumer<long double>           container(sizeof(long double));     std::random_device                      randomDevice;     std::mt19937                            randomGenerator(randomDevice());     std::uniform_int_distribution<uint32_t> distribution(10000, 1000000);      randomGenerator.seed(std::time(nullptr));      std::thread producer([&]() {         for (int i = 1; i < 21; ++i) {             usleep(distribution(randomGenerator));             container.produce(i);         }          container.producerIsOver();     });      std::thread consumer([&]() {         do {             usleep(distribution(randomGenerator));             std::cout << "Consume " << container.consume() << std::endl;         } while (!container.isFinished());     });      producer.join();     consumer.join();      return EXIT_SUCCESS; }  

How to deploy a Dynamic Policy Engine on a Consumer Router by any Manufacturer?

Specifically I am working on a security product for ISP’s that has a custom policy engine per individual Router, but in theory this should run on a Router from any Manufacturer.

Considering each Manufacturer (TP-Link, Actiontec, Cisco, Greenwave Systems, Linksys) are running their own OS along with differing chipsets/CPU/Flash/Ram brands with varying resource capabilities…what is the best method of running/deploying a policy engine that can be dynamically updated?

I’ve found a few suggestions pointing at developing an SDK for ISP’s to utilize or developing an Agent the ISP can deploy across all customers running their equipment but hit a roadblock on determining which/if either are best to pursue.

An SDK seems scalable but I am not sure what facets to consider on implementing an SDK to conform to differing chipsets/CPU/Flash/Ram.

An Agent seems more feasible but not scalable as it will require a full reboot of a ISP customers Router.

In both cases please take into consideration the resource constrained device will have minimum specs of:

2 Cores, 600 MHz 128 Mb Flash 512 Mb RAM

Consumer is not authorized to access Magento_Sales::sales

I am creating an Invoice through Magento REST API with the below request:

POST https:///mg2/rest/V1/order/9/invoice

Headers: {“Content-Type”:”application/json”,”Accept”:”application/json”,”Authorization”:”Bearer jnsptpdgsvmk5mrbctisi3e0yml7k6iu”}

Body: {“capture”:true,”order_id”:9,”items”:[{“order_item_id”:33,”qty”:3,”extension_attributes”:{}}],”notify”:true,”appendComment”:true,”comment”:{“extension_attributes”:{},”comment”:”Invoice#: 22″,”is_visible_on_front”:0},”arguments”:{“extension_attributes”:{}}}

But I consistently get this error: {“message”:”Consumer is not authorized to access %resources”,”parameters”:{“resources”:”Magento_Sales::sales”}}

I am certain that the token I use works, because I am able to search orders and customers using the same token. The token is automatically renewed when I get a 401 error code in the response.

Also, the user I set up in Magento is an Administrator, which (I believe) has a full access on all objects (Sales Orders, Invoice, Shipments, etc.).

I have already configured the Integrations as well. So I am not sure what else I am missing?

Thanks in advance.

Consumer Producer problem solution using JDBI, MySQL, HikariCP

The app suppose to solve Producer–consumer problem. Would be great to have feedback on overall design, testability and general advice on how to improve further. I have doubts about choosing the right approach while testing a temporary database outage.

@Slf4j public class DataSource {      private static final String CONFIG_FILE = "src/main/resources/db/";     private static HikariConfig config = new HikariConfig(CONFIG_FILE);      private static HikariDataSource ds;      static{         try {             ds = new HikariDataSource(config);         } catch(CJCommunicationsException | HikariPool.PoolInitializationException e) {   ;         }     }      static class Holder {         static DataSource INSTANCE = new DataSource();     }      private DataSource() { }      public static DataSource getInstance() {         return Holder.INSTANCE;     }      public static Connection getConnection() throws SQLException{             return ds.getConnection();     }      public static HikariDataSource getDs() {         return ds;     } }