Running Magento 2 Crons With Queue Consumers on Kubernetes or Docker

Submitted by Eric on Sun, 04/21/2019 - 16:01
Image
magento kubernetes

Let me start saying that this article will be quick and simple, but it was a headache to find out the issues we encountered with running Magento crons with queue consumers on Kubernetes (or even Docker).
Configuring the cron is pretty simple and Magento has a nice documentation here: https://devdocs.magento.com/guides/v2.3/config-guide/cli/config-cli-subcommands-cron.html

But the problem is not just running the crons, it's if you run crons while you have queues enabled using RabbitMQ, which will require to use the consumers. You can read more about the queues and consumers here: https://devdocs.magento.com/guides/v2.3/config-guide/mq/rabbitmq-overview.html

Let's say you added a new module that will consume the queues or you just added to your app/etc/env.php:

'queue' =>
  array (
    'amqp' =>
    array (
      'host' => 'rabbitmq.example.com',
      'port' => '11213',
      'user' => 'magento',
      'password' => 'magento',
      'virtualhost' => '/',
      'ssl' => true,
      'ssl_options' => [
            'cafile' =>  '/etc/pki/tls/certs/DigiCertCA.crt',
            'certfile' => '/path/to/magento/app/etc/ssl/test-rabbit.crt',
            'keyfile' => '/path/to/magento/app/etc/ssl/test-rabbit.key'
       ],
     ),
  ),

Or you added the piece of code below during installation:

--amqp-host="<hostname>" --amqp-port="5672" --amqp-user="<user_name>" --amqp-password="<password>" --amqp-virtualhost="/"

Good, now Magento will use consumers to work the queues. Now we encounter two problems. The first one is that every time a cron runs, it will create background processes with the consumers, for example:

$ kubectl exec -n my-store magento-75788af261-adwdc -c magento ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  18048  1932 ?        Ss   14:13   0:00 /bin/bash /setup.sh
root        16  0.0  0.0   4284   588 ?        S    14:13   0:00 /bin/sh /usr/sbin/apache2ctl -D FOREGROUND
root        18  0.0  0.0 274652 19168 ?        S    14:13   0:00 /usr/sbin/apache2 -D FOREGROUND
www-data    20  0.9  0.2 344228 90216 ?        S    14:13   1:50 /usr/sbin/apache2 -D FOREGROUND
www-data    24  0.9  0.2 344232 86344 ?        S    14:13   1:41 /usr/sbin/apache2 -D FOREGROUND
www-data    59  0.0  0.2 291524 94128 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start commonErrorManagement --pid-file-path=commonErrorManagement-magento76799dc865rglqw.pid --max-messages=10000
www-data    61  0.0  0.2 291524 94024 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start ping --pid-file-path=ping-magento76799dc865rglqw.pid --max-messages=10000
www-data    63  0.0  0.3 317288 118348 ?       S    14:15   0:05 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start productConsumer --pid-file-path=productConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data    65  0.0  0.2 293740 95408 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start stockConsumer --pid-file-path=stockConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data    67  0.0  0.2 293752 95652 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start productReturnUpdatedConsumer --pid-file-path=productReturnUpdatedConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data    69  0.0  0.3 293704 97224 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start refundUpdatedConsumer --pid-file-path=refundUpdatedConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data    71  0.0  0.3 317288 118240 ?       S    14:15   0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderCancellation --pid-file-path=orderCancellation-magento76799dc865rglqw.pid --max-messages=10000
www-data    73  0.0  0.3 317288 118116 ?       S    14:15   0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderGetNotifications --pid-file-path=orderGetNotifications-magento76799dc865rglqw.pid --max-messages=10000
www-data    75  0.0  0.3 317288 118440 ?       S    14:15   0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderResendNotification --pid-file-path=orderResendNotification-magento76799dc865rglqw.pid --max-messages=10000
www-data    77  0.0  0.3 317292 118508 ?       S    14:15   0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderConsumer --pid-file-path=orderConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data    79  0.0  0.2 291524 93944 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderUpdatedConsumer --pid-file-path=orderUpdatedConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data    82  0.0  0.3 295756 97992 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start customerShipmentDoneConsumer --pid-file-path=customerShipmentDoneConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data    84  0.0  0.2 291656 93784 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start orderLineCancelledConsumer --pid-file-path=orderLineCancelledConsumer-magento76799dc865rglqw.pid --max-messages=10000
www-data    86  0.0  0.2 291656 94968 ?        S    14:15   0:03 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start shippingAddressUpdated --pid-file-path=shippingAddressUpdated-magento76799dc865rglqw.pid --max-messages=10000
www-data    92  0.0  0.3 317292 119656 ?       S    14:15   0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start serviceBusDiscover --pid-file-path=serviceBusDiscover-magento76799dc865rglqw.pid --max-messages=10000
www-data    94  0.0  0.3 317292 118524 ?       S    14:15   0:04 /usr/local/bin/php /var/www/html/bin/magento queue:consumers:start async.operations.all --pid-file-path=async.operations.all-magento76799dc865rglqw.pid --max-messages=10000
www-data   178  1.6  0.2 336144 81796 ?        S    16:24   0:53 /usr/sbin/apache2 -D FOREGROUND
www-data   179  1.2  0.2 336220 83776 ?        S    16:24   0:42 /usr/sbin/apache2 -D FOREGROUND
www-data   195  1.3  0.2 336160 81964 ?        S    16:27   0:42 /usr/sbin/apache2 -D FOREGROUND
www-data   198  1.0  0.2 333196 74692 ?        S    16:30   0:31 /usr/sbin/apache2 -D FOREGROUND
www-data   199  1.0  0.2 333196 74696 ?        S    16:30   0:31 /usr/sbin/apache2 -D FOREGROUND
www-data   200  1.0  0.2 333212 74708 ?        S    16:30   0:29 /usr/sbin/apache2 -D FOREGROUND
www-data   201  1.3  0.2 333140 74632 ?        S    16:30   0:40 /usr/sbin/apache2 -D FOREGROUND
www-data   210  1.1  0.2 333140 74636 ?        S    16:30   0:34 /usr/sbin/apache2 -D FOREGROUND
root       440  0.0  0.0  36640  2712 ?        Rs   17:19   0:00 ps aux

As you can see from above there are several consumers that started, they are all from Magento. If you have any custom queue, it will add the consumer to the list as well. That's when the second issue comes up.
Magento has on its documentation that each consumer processes 1000 messages and then terminates: https://devdocs.magento.com/guides/v2.3/config-guide/mq/manage-message-queues.html#configuration

max_messages - the maximum number of messages for each consumer that must be processed before consumer terminate, by default is 1000. If it is 0, then the consumer never stops working.

But as you can see above it is set to 10000 and we can confirm on their repo: https://github.com/magento/magento2/blob/2.3-develop/app/code/Magento/MessageQueue/Model/Cron/ConsumersRunner.php#L114

All right, no big deal if you are running the cron on a single or multi servers. But once you start the cron, it has these background processes that will only end if the queue has completed 10000 messages (by default). The cron will run until Kubernetes timesout (and the default is one hour).
We could then set --max-messages to 1, but the way Magento works it will ALWAYS wait for that one, instead of just checking if queue is empty or not and finish if it's empty. There is an issue opened right now on their repository, but no sing that they will ever fix it: https://github.com/magento/magento2/issues/17951

So what to do? Well, for now we just decided to run Supervisord on its own pod and the consumers inside Supervisord. And don't remember to disable the consumers from running during cron:

...
    'cron_consumers_runner' => array(
        'cron_run' => false
    ),
...

I would just hope that Magento would consider building its application DevOps friendly.
If you managed to get this working a different way, don't forget to add your comment below.


FacebookTwitterGoogle+Share