Since a few years until now, every two months I'm working on a different project of the same kind: one or more services with many CPU-bound tasks.
A common scenario is a RESTapi service for differents client, various pieces of software to interface ORM like Psycopg2, Pony, pyMongo (and of course her majesty, SQLAlchemist), with some heavy task like parsing pdf or querying many DB and launch a function to compare and shrink all data into a new JSON-like-object.
For many of us developer, the real question is only one: which framework choose to manage all queue of request and get easily responses?
At the first time, I'm looking on Google and read some answer on Stackoverflow. Of course, I find the common choice (if you mind it, it was Celery) and start to develop with it.
But now I'm going to ask how many and who are going to use this framework.
Looking around trends.google.com, I cannot find any kind of evidence except once: since a few years, more search contains keyword between celery framework and elements that aren't strictly correlated to Django.
After an exploit in 2017 on February, a date that put Celery-Dango correlation more researched than Celery-Python, many developers use Celery as a "good standalone framework" for various purpose. That is a good thing if you consider all the macro-argument "microservices"; an area characterized by a specific goal: satisfy all request as quickly as possible, using horizontal scalability.
Celery MQTT, Redis, AMQP: which is the standard de-facto?
The real question is: how to provide messages between worker? How to share a task? That's so clear, but not very simple.
After a few test, I have found so many contents using Celery with Redis, but less with amqp. I cannot explain the source of this design choice, but I'm going to share one of them, simply and clearly; like all frameworks I have seen around in this years, if many developer adopt a "standard" like Celery-Redis there is always an answer to explain that. The answer is documentation.
The documentation of https://docs.celeryproject.org/en/stable/ is awesome, if you need to deploy quickly many workers to assign on specific tasks, for a few moments. But is not so clear to explain how to buildup a microservice network using broker message like MQTT or AMQP.
At this point, All post like this would share a piece of code to solve this problem. But not this, and not this moment. Until I'm not sure I'm not going to reinvent the wheel, I keep testing and improve the few lines of code. I hope to share it asap.