Include dependency on another microservice, or isolate the dependency in a third microservice?

New to the microservice arena, and working through some “guiding concepts” for architecture.

Assume a “User” microservice that identifies users. A second microservice, “System” identifies systems. To associate users/access to a system, there would typically be a join table, and it “seems reasonable” that this table would live in either “User” or “System”.

But, what happens when “Organization” is introduced? Does “User” (assuming it had the join to “System”) have to be updated to include a new join? Or, would it be cleaner to implement separate “System User” and “User Organization” services, creating a more hierarchical dependency, and avoiding direct awareness between the “User”, “System” and “Organization” microservices?

Is it a bad idea to make a microservice for every endpoint?

Many microservices commonly have only a single endpoint, so I know that a single-endpoint-service is not inherently a bad thing.

I’ve noticed that many of the advantages of a microservice over a monolith amplify in an inversely proportional manner to the size of the microservice. For instance, being able to scale up one service of an application independently of the others is an advantage of microservices; yet if every microservice was as small as possible (managing a single endpoint / command), this advantage simply becomes greater.

Other advantages which hold this property include:

  • Deployability
  • Team management (perhaps each team is assigned a group of “related” microservices, and perhaps each member is assigned a couple microservices within that group)
  • Emphasis of security where it counts; i.e. if one endpoint involves communication of more sensitive data, it can be singled out and given higher focus on security
  • Selection of tools / languages; i.e. if one endpoint involves some sort of processing or data retrieval which warrants using a different persistence format (NoSQL vs SQL vs in-memory database, etc) or programming language, it can be singled out and built using those tools and languages

Perhaps there are others as well. My point is, if these advantages simply grow the more you split your services, why don’t we make every microservice as small and granular as possible?

Using distributed locks in microservice environment

I’m developing distributed system and trying to use best practices of microservice architecture. I was faced with a situation when I think I need something like distributed locks. Since I have not so many experience in microservices and still not sure about the final solution. I need any suggestions, thoughts, best practices, possible articles that can help me (now I’m investigating the issue and already have possible solutions).

I’m using Azure Service Bus to organize pub \ sub communication and Redis as shared cache in my system.

Here is an abstract schema of what I actually trying to do . Explanation: I have queue with incoming data (dataFlow). The format is the following {dataOwnerId: “1”, dataOwnerSubId: “1”, data: “status”} The dataOwnerId is an aggregator for a 1-n sub owners. So it’s 1 to many relationship.

The owners periodically (let say each 1 minute) push new status (data) to the queue related to certain sub-owner. So if the dataOwnerId related to dataOwnerSubIds [1, 2, 3]. The message can be {1, 1, status} or {1, 2, status} or {1, 3, status} and after 1 minutes {1, 1, status2} or {1, 2, status2} or {1, 3, status3} etc

The queue has 1-n consumers (dataConsumers) which processed data from queue and storing to DB. Data order is non-deterministic, the first available consumer takes the first message from queue. So the dataConsumer1-N can handle the data from owner1-N and sub-owner1-N.

For example in one time the dataConsumer1 can handle the data from owner1 and sub-ownerN the next time he can handle the data from ownerN and sub-owner1.

Requirements: The possibility of auto adding new data owners and data sub-owners to the system during queue processing required.

Possible solution: I can have situation when the dataOwnerN + 1 (new customer) already push data to the queue but he is still not in the system. In this case the dataConsumers will skip this new owner (skip storing the data) and push message NewOwnerEvent to ServiceBus.

The messages will be handled by dataOwnerManagers which working with dataOwners. The dataOwnerManagers responsibilities are creating the new dataOwners and related dataSubOwners

Problem: In one time dataConsumers can process data of two (for example) the same NEW (should be added to the system before processing) dataOwners with same or different dataSubOwners for example: {1, 1, status} or {1, 2, status} . In this case the two NewOwnerEvent will be created and dataOwnerManagers can start creating the same NEW dataOwner record. Same situation can be for dataSubOwners but here we have at least 1 minute delay for creation. So if the first event will be {1, 1, status}, the second event {1, 1, status2} will be after 1 minute and it almost impossible that those 2 events will be handled in one time in dataConsumers.

So I need to prevent the resending NewOwnerEvent related to the same dataOwner – if the dataConsumer1 already send the NewOwnerEvent related to NEW owner1, I must notify (add possibility to check that the same event already exists) all other dataConsumers, that the owner1 have already been added to the queue and will be created soon. So I need some shared list of already sent/processed NewOwnerEvent events/dataOwners

Possible implementation: I tried using Redis to solve this problem. I’m adding the key each time I’m pushing the NewOwnerEvent event, so all other data processors can check this key to understand was the event already pushed or not. But the problem here is that in one time different dataConsumers can take the message from the same owner {1, 1, status} and {1, 2, status} and try to push event at the same time, because the key in Redis won’t be created yet(dataConsumer2 check faster then dataConsuer1 add the key in Redis). So I need distributed lock to be sure that the Redis key will be added only by one dataConsumers at one time. I know that Redis already have implementation for distributed lock but I still confusing and guess it will be very complex solution in my case.

Files resources for microservice

Hi I am using Adam Biens microprofile maven archetype and deploys and makes a restful api service boilerplate great.

I want one service to have a small list of Test files that is can use to pass to another. I have two sets of files that I want to update and one is test/training data that might change less often so I thought storing them as a resource would be good and they can get updated with a war compile. However, they do not seem to load however I try to load them, from web-inf or class or classloader. What is the recommended way for them to be stored and loaded in jax-rs? So is there a better design or making files available to microservices to use? Ideally I would use a solution that I could use for my scale data which might be up to 2 million small text files.

Can/should multiple application contexts share a common microservice in a single namespace?

My current employer has a single namespace into which all microservices for all projects are deployed.

Projects A, B and C all use microservice x in this common namespace:

enter image description here

Rather than each design and implement their own microservice that performs the same behaviour (for example for some common domain object CRUD operations against a database), Projects A, B and C all use and claim shared-ownership of a common implementation – microservice x – and service it in accordance with their project requirements.

Projects are upsetting the shared ecosystem for each other when deploying revised versions of x (downtime, breaking changes).

I am unclear if this is an example of what Kubernetes best practices considers to be an example of an “overcrowded namespace” anti-pattern – I am thrown by the fact that each project utilises a common microservice and I thought one of the points of microservices was to expose common things in a kind of public way like this?

Is it appropriate for this “public exposure” to manifest as a single service instance in a common namespace?

Or should the namespaces split apart and they all have their own deployment of microservice x like this?

enter image description here

Microservice Deomposition and Inter-service communication

enter image description here

I read theory but I need practical architectural advice on the actual implementation in java language for microservices. In the context of the attached screenshot. Here is my setup.

Order Project consisting of the following 4 Modules.

  1. Order History (Jar file, Docker Container)
  2. Order Placement (Jar file, Docker Container)
  3. Order Tracking (Jar file, Docker Container)
  4. Order Dispute (Jar file, Docker Container)
  5. Service (WAR file, Docker Container)

Service (5) sits on the front (order.example.com), receives HTTP request, then calls 1 or 2 or 3 or 4 via gRPC and returns the result.

Q1. Is 1, 2, 3, 4 above considered a Microservice or is OrderManagement or ShoppingCart a microservice?

Q2. Same setup as Order (sits behind cart.example.com). Communicate between Order and Shopping Cart via like Messaging (Kafka) or REST or gRPC?

Is this correct? Hopefully, I don’t have a nano anti-pattern here?

Moving a microservice architecture away from Elastic Beanstalk

I have an application architecture that looks like this:

Application architecture

Client traffic flows from the web to our CDN. Primarily, the CDN will route traffic to our frontend application’s Elastic Load Balancer. That ELB in turn sends the traffic to NGINX running on one of any number of EC2 instances. These EC2 instances are configured via Elastic Beanstalk to use a single-container Docker setup. The Docker container contains a Node.js web application that runs React.js, ultimately generating content that gets returned to the client.

This React.js application internally requests content from various REST API services we host that have very similar — save for using Restify vs. Express — architectures. However, the API requests originating from the frontend application point at the public-facing DNS at our CDN for each API.

There are two main reasons I say I would like to explore moving away from Elastic Beanstalk:

  1. As a service, Elastic Beanstalk is just a bit too heavy-handed for my tastes. Relying on extensions files and a somewhat opaque set of AWS-managed scripts to handle application/configuration deployment has bitten us a number of times.

    It’s also a bit too opinionated for our current application lifecycle. I realize this is by design, but it’s starting to feel like we’re running up against it and fighting the design more than it’s helping.

  2. Elastic Beanstalk also forces us to use Elastic Load Balancing, which has recently started causing us headaches: During normal load, ELB is generally fine. However, if we receive a (not even especially large) traffic spike, the frontend application’s ELB starts serving HTTP 503s as it scales up to handle the load.

    I’m aware we can ask AWS to pre-warm the ELB in anticipation of a spike, but for usual (e.g., weekly) operations that elevate traffic beyond normal conditions, manually requesting pre-warming becomes an annoying, and error-prone process.

Beyond those reasons, there are other things that are apparent that we’re losing out on with this approach:

  • Even though we have an almost completely static site (which is why we rely heavily on the CDN, even for REST API requests), going all the way out of the network for server-side requests from the frontend application to our APIs seems inefficient.

    If we were to instead look towards an approach that bypasses all of these various hops (is something like Consul or Eureka an option here? Forgive me, I’m not familiar yet with these approaches, but have been reading up on them recently), we could reduce the overall latency of our design.

    We still need to be able to have client requests hit our APIs directly during client-side routing, so we can’t make them completely private, but if it’s possible to have our cake and eat it, too (i.e., gain speed by doing frontend-to-API requests all internally, while still allowing client-to-API requests via the CDN), that would be great.

  • Since NGINX is simply passing requests between the ELB and the Node.js web servers, it isn’t really providing much utility for us right now. This seems like an opportunity to either rip it out entirely, or move the load balancing here (I’m also open to moving to HAProxy if there are other benefits, especially as they relate to a service discovery-based architecture).

As for questions I have swimming around my head (I’ve been in a kind of architecture soup the past week, so forgive me):

  1. Does it seem feasible to rip out Elastic Beanstalk and instead rely on a simpler Auto-scaling Group approach that manages EC2 instances with a custom AMI? I don’t want to get too crazy with this (e.g., Kubernetes, serverless, etc.), but I also don’t want to run up against a wall too quickly if we decide to move away from EB.

  2. Is a service discovery platform a potential benefit here? Say we were to use Consul. Would an example approach to that be to add the Consul agent alongside our Docker container (in a sibling container via docker-compose? As a standalone service on the EC2 instance?) and manage the server-to-server requests that way?

  3. What about load balancing? Does it make sense to put a cluster of HAProxy instances behind our CDN that know about the healthy nodes in each environment (e.g., app, API 1, API 2, etc.)? I realize this seems like reinventing the ELB wheel, but at least we can get around the scale-up HTTP 503 issue, or at least manage it ourselves based on our known usage patterns.

  4. I’ve seen approaches that run NGINX or HAProxy locally, alongside each application instance in a service mesh setup, presumably to route local traffic between services. Is this necessary for those types of setups, or is it more of an optimization?

Thanks for any help you can provide, I really appreciate it.

web module in a microservice architecture using consul

new to microservices and trying to implement one. Let’s say I’ve 2 microservice modules.

  1. Products
  2. Orders

I’m using Consul to register these services (service discovery)

How are these services should be accessed from a client (it’s a web page calling these services via XHR).

Do I need to setup a web microservice on the front that will intercept requests, get the endpoint from Consul followed by calling the service and sending back response and headers?

Basically the question is how the clients would be reaching out to Order and Product service?

How to prevent querying each user each time when having per microservice authorization? [on hold]

I am trying to find a good approach for authentication and authorization of a user in a microservice-application.

Currently I have the following idea:

authentication and authorization using JWT

In regards to the sequence diagram I have the following questions:

  1. [step 12] and [step 16] check for a permission, which is necessary on the server-side. Where are those permissions supposed to be stored?
  2. What about the note (after [step 20])? Possibilities:
    1. Issue an additional signed Token, that contains permissions / roles for the message service. This token can be acquired during [step 13].
    2. Re-issue existing JWT, include permissions / roles for the message service. This token can be acquired during [step 13].
    3. Issue a server-side request, that returns information about whether a certain GUI-area should be displayed.

If in 2.1 or 2.2 the client-token is modified, the user is able to access GUI-areas, which should not be available to him. However: any subsequent service calls will not succeed, because the modified (nor the original) token permits access to corresponding server-calls to the user.

In 2.3 the microservice is queried each time the GUI has to decide whether to display a GUI-area or not.

Serialization of messages on a servicebus and microservice architecture

In a microservice-based system, it exist some principles. Sam Newman often refers to them as these:

  1. You should always model your application around the domain
  2. A culture of automation. Automate whats possible to automate
  3. Hide your implementation details
  4. Decentralize everything
  5. Deploy indepentently
  6. Isolate your failures
  7. Highly observable

https://samnewman.io/talks/principles-of-microservices/

When building a system where you microservices reflects the domain, which basically means to me that you have one microservice for each defined bounded context, I find it quite difficult to deal with a servicebus architecture and events published.

An example:

Lets say you’re creating a system for a foodstore chain where the customers can sign up for a bonus program, which gives them x % back for every purchase they make.

So you and your team sits down with the customer to identify the domain and you identified at least two bounded contexts. The contexts identified are: Customer and Loyalty. You start out by making two microservices based on these two contexts, CustomerService and LoyaltyService.

The business defines a rule that, whenever a customer signs up, a 5 % discount coupon shall be sent to the customer’s email.

The CustomerService exposes an API to create the customer in the database and when the customer is created an event CustomerCreatedEvent is published on the servicebus. As the LoyaltyService is interested in this event to send the 5 % discount coupon, it has to subscribe to this message. In NServiceBus, which is my prefered servicebus, its done this way:

public class SomeClassName : IHandleMessage<CustomerCreatedEvent>  

You might already figured it out, but this is where I struggle to find a good solution to be “compatible” with the principles of microservices. How should the LoyaltyService be able to subscribe and deserialize this event without having a reference to CustomerService either over NuGet or direct reference? Should we duplicate our code so that the CustomerCreatedEvent are defined in both services?

Any suggestions, blog posts, books, pluralsight courses on how to create a domain driven microservice architecture application would be very helpful.

Thanks.