Introduction

This article is about problems that can occur with Magento cronjobs. The standard way to configure crontab for Magento 1 has it’s limits. The more custom cronjobs  a Magento system has, the more probable the system will face problems considering cronjobs. The most common issues are:

• Indexer Cronjob (Magento Enterprise malways mode) takes longer than usual so that other cronjobs (mdefault mode) are skipped (not executed) for the time the indexer runs
• Some of the cronjobs in mdefault scope take a long time to run and block others

Second issue can be avoided if this rule is followed: create a shell script and make a separate crontab entry on the server for long running jobs, e.g. imports or exports.

Magento 1

Plain Magento 1

If  we have a plain Magento 1 we can split the malways and mdefault cronjob modes:

* * * * * /bin/sh /path/to/your/cron.sh cron.php -mdefault 
* * * * * /bin/sh /path/to/your/cron.sh cron.php -malways

This will prevent that the indexer blocks other mdefault jobs or an mdefault job blocks the indexer.

But there are much more options of parallelization if you use the Magento 1 extension AOE Scheduler.

Magento 1 with AOE Scheduler

The AOE Scheduler has multiple benefits for managing Magento Cronjobs. In this article I want to focus on the “cron groups” feature.

The instruction how to use cron groups can be found here.

The main idea is to split Magento cronjobs into groups. The execution of those groups can be triggered separately via the server crontab.

I recently introduced this feature in a project. These are the steps I needed to take:

1. Create a new module, e.g. Namespace_AoeSchedulerCronGroups
   This module contains only an empty helper and config.xml.
2. In the config.xml define the groups for each cronjob in the system like this:

   <crontab>
       <jobs>
           <job_code>
              <groups>your_group_name</groups>
           </job_code>
       </jobs>
   </crontab>

   To get a full list of cronjobs you can either use the backend grid of AOE Scheduler or use the following Magerun command:

   $ n98-magerun.phar sys:cron:list

   The splitting of cronjobs in groups should be based on project knowledge and experience. In my case the groups were something like this:

   • magento_core_general 
   • general
   • important_fast
   • important_long_running
   • projectspecific_general
   • projectspecific_important
   • erp
   • erp_long_running
3. After deploying the new code base with the new module to the server, edit the crontab, remove the standard cron.sh / cron.php call and add something like this (matches my example groups):

   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode always
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --includeGroups magento_core_general
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --includeGroups general
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --includeGroups important_fast
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --includeGroups important_long_running
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --includeGroups projectspecific_general
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --includeGroups projectspecific_important
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --includeGroups erp
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --includeGroups erp_long_running
   # For jobs not assigned to any group
   * * * * * ! test -e /path/to/your/magentoroot/maintenance.flag && /bin/sh /path/to/your/magentoroot/scheduler_cron.sh --mode default --excludeGroups magento_core_general,general,important_fast,important_long_running,projectspecific_general,projectspecific_important,erp,erp_long_running

   The last entry is pretty important: this executes jobs, which are not assigned to any group, e.g. for newly developed cronjobs which didn’t get any group assignment.

Magento 2

Magento 2 comes with the cron groups feature out of the box. The feature and how to configure multiple groups are explained in the magento devdocs:

• Custom cron job and cron group reference
• Configure a custom cron job and cron group (tutorial)

In Magento 2 there are more explicit options for cron groups than in Magento 1 including installed AOE Scheduler module:

Groups are defined in a cron_groups.xml file and each group may get its own configuration values:

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:module:Magento_Cron:etc/cron_groups.xsd">
   <group id="custom_crongroup">
     <schedule_generate_every>1</schedule_generate_every>
     <schedule_ahead_for>4</schedule_ahead_for>
     <schedule_lifetime>2</schedule_lifetime>
     <history_cleanup_every>10</history_cleanup_every>
     <history_success_lifetime>60</history_success_lifetime>
     <history_failure_lifetime>600</history_failure_lifetime>
  </group>
</config>

Conclusion

In this article we looked at the evolution of cronjob performance optimization beginning with Magento 1, over Magento 1 with installed AOE Scheduler extension, up to Magento 2. Here we have a good example, how community modules with nice features can be a benefit for Magento and also that Magento can implement those features in future releases.

Feel free to leave a comment.

Alexander Dite

Magento Certified Developer Plus
Magento 2 Professional Developer
Magento 2 Solution Specialist