Communication between services is something that we cannot avoid when building microservices. We can either do it synchronously with for example REST or asynchronously with for example Messaging. A benefit with messaging is that we can decouple the sender from the receiver. This will avoid a runtime dependency between the services and the sender can just say what has happened rather than knowing the internals of the services it needs to call. These things help us with stability and keeping the services clean and not bloated with concepts from other teams services. But it also comes with a few challenges. For example, if I publish a message, when do I know that someone has received it and if someone asks us to resend messages, how do we accommodate that?
It is time to visit our squeeky toy company for the final time. We have previously helped them with two problems, one with distributed transactions and one with slow reads. This time they have stumbled across some issues with their asynchronous communication. Let’s explore their problem.
If you’re looking for Challenge 1, you can find it here. Challenge 2 is over here. And, don’t forget to grab a spot to our Microservices MeetUp with Chris Richardson at the 6th of May.
It is tuesday afternoon and suddenly a dashboard for one of the teams just went red and one of the team members sighs and shouts out.
“What now then?”
And starts to browse the logs. After a few minutes the whole team gathers to look at the inconsistencies found.
“Look, here. It looks like we are getting updates on a product that does not yet exist in our system.”
The team asks the other team about this and gets an interesting response.
“It looks good on our side. The product is in our database which means that we should have sent it.”
Another team members joins in.
“Wait! Look here. We actually got some strange exception when publishing the message. Which should mean that we stored it in the database, but did not publish. I wonder how many of these have happened.”
The ProductService is responsible for taking care of the product catalog and everything around it. When products are added/changed/removed they notify everyone who wants to listen to the change.
At the moment a part of their code does two things:
Could you help and guide the team to ensure that these are happening transactionally:
- We do not want to publish if the storage fails.
- We do not want to store if the publishing fails (unless there are other ways of re-sending the message).
- A bonus would be to be able to resend events if other teams are losing them somehow.