Browsed by
Category: Magento 1

Cronjob performance optimization: Magento 1 vs. Magento 2

Cronjob performance optimization: Magento 1 vs. Magento 2

Introduction

This article is about problems that can occur with Magento cronjobs. The standard way to configure crontab for Magento 1 has it’s limits. The more custom cronjobs  a Magento system has, the more probable the system will face problems considering cronjobs. The most common issues are:

  • Indexer Cronjob (Magento Enterprise malways mode) takes longer than usual so that other cronjobs (mdefault mode) are skipped (not executed) for the time the indexer runs
  • Some of the cronjobs in mdefault scope take a long time to run and block others

Second issue can be avoided if this rule is followed: create a shell script and make a separate crontab entry on the server for long running jobs, e.g. imports or exports.

Magento 1

Plain Magento 1

If  we have a plain Magento 1 we can split the malways and mdefault cronjob modes:

This will prevent that the indexer blocks other mdefault jobs or an mdefault job blocks the indexer.

But there are much more options of parallelization if you use the Magento 1 extension AOE Scheduler.

Magento 1 with AOE Scheduler

The AOE Scheduler has multiple benefits for managing Magento Cronjobs. In this article I want to focus on the “cron groups” feature.

The instruction how to use cron groups can be found here.

The main idea is to split Magento cronjobs into groups. The execution of those groups can be triggered separately via the server crontab.

I recently introduced this feature in a project. These are the steps I needed to take:

  1. Create a new module, e.g. Namespace_AoeSchedulerCronGroups
    This module contains only an empty helper and config.xml.
  2. In the config.xml define the groups for each cronjob in the system like this:

    To get a full list of cronjobs you can either use the backend grid of AOE Scheduler or use the following Magerun command:

    The splitting of cronjobs in groups should be based on project knowledge and experience. In my case the groups were something like this:

    • magento_core_general 
    • general
    • important_fast
    • important_long_running
    • projectspecific_general
    • projectspecific_important
    • erp
    • erp_long_running
  3. After deploying the new code base with the new module to the server, edit the crontab, remove the standard cron.sh / cron.php call and add something like this (matches my example groups):

    The last entry is pretty important: this executes jobs, which are not assigned to any group, e.g. for newly developed cronjobs which didn’t get any group assignment.

Magento 2

Magento 2 comes with the cron groups feature out of the box. The feature and how to configure multiple groups are explained in the magento devdocs:

In Magento 2 there are more explicit options for cron groups than in Magento 1 including installed AOE Scheduler module:

Groups are defined in a cron_groups.xml file and each group may get its own configuration values:

Conclusion

In this article we looked at the evolution of cronjob performance optimization beginning with Magento 1, over Magento 1 with installed AOE Scheduler extension, up to Magento 2. Here we have a good example, how community modules with nice features can be a benefit for Magento and also that Magento can implement those features in future releases.

Feel free to leave a comment.

Magento Certified Developer Plus
Fixing issues after changing product attribute type from varchar to text

Fixing issues after changing product attribute type from varchar to text

In some cases there is a need to change the backend type of a catalog product attribute from varchar to text. The purpose of this change is to get more than 255 characters space for a string value.

In this article I will cover the situation when problems occur after changing the backend type of an attribute.

The Problem

If the backend type of an attribute is changed, e.g. via install/upgrade script, Magento does not automatically copy and clean up old values. The consequence of that is that there are rudiments in the EAV value tables which cause some side effects. One of the side effects I was facing is editing a product which had a value for the affected attribute before the backend type change (in admin area). No values are displayed and there is no possibility to set a new value.

So what to do if the change already happened and there is a mix between old value table rudiments and new value table entries?

The Solution

One possible solution to solve the issue are the following SQL Statements, here is an example for changing from varchar to text (you need to find out the id of the attribute from the eav_attribute table – here {attribute_id}):

1. Copy the “value” from varchar table to text table for the case an entry for a product entity exists in both tables, but only if the “value” in the text table is null:

 2. Copy entries which do not exist in text value table, but exist in the varchar table

3. Delete entries from the varchar table

Important note

Please verify the SQL, whether it is suitable for your purpose. Best practice is also to test it in a local / staging system and to back up the live database before applying the SQL on production. The solution is not perfect: I myself faced the issue, that the enterprise indexer cronjob took about 4h after applying the SQL, which blocked other cronjobs to be executed (about 50K products in DB). Possible way to avoid this is to separate “malways” (enterprise indexer) and “mdefault” cronjobs.

I hope this  can be helpful. Feel free to comment if you faced this issue too or if you have any additions or a better solution.

Magento Certified Developer Plus
Solving a 2006 MySQL error connection timeout in Magento1

Solving a 2006 MySQL error connection timeout in Magento1

In my recent task I was testing a web crawler script which uses Magento database information for the crawling requests. I have encountered the following error:

Fatal error: Uncaught exception ‘PDOException’ with message ‘SQLSTATE[HY000]: General error: 2006 MySQL server has gone away’ in lib/Zend/Db/Statement/Pdo.php:228

The Problem

This error occured after around 45 minutes of script runtime.
The script was written in a way that it was possible that there is no database interaction for a longer period.
In consequence to that when the script reached a point where it was trying to save or fetch something from the database, the mysql connection ran into a timeout – unnoticed by the script.
Thus resulting in the above mentioned MySQL error.

mysql wait_timeout

The variable controlling this timeout from MySQL is the wait_timeout system variable.

Definition from MySQL Reference Manual :

“The number of seconds the server waits for activity on a non-interactive connection before closing it.”

As it turns out we have already had this situation in another project – thanks to the netz98 developers for the hint.

The Solution

The solution is to close the connection before using it – in case of long running and un-interrupted code part that does no database communication.

We have added the following code snippet to our ResourceModel:

With this method the Connection within the Adapter can be closed. When it is closed the connection will be re-initialized automatically with the next Database Interaction that is triggered by our code. To do so we introduced the parameter $useNewConncetion which enforces this behaviour.

Each time we have reached a point in our script where it could be possible that the connection hit the wait_timeout we just call this method with useNewConnection set to true.

I hope this article is helpful for you, in case you face the same situation. Feel free to comment if you faced this issue too or if you have any additions.

Update: Keep alive implementation by Ivan Chepurnyi (@IvanChepurnyi)

 

Magento Certified Developer Plus
Thoughts about custom attributes and the way to manage them!

Thoughts about custom attributes and the way to manage them!

To add or update an EAV attribute in Magento, it is necessary to add setup scripts to your code base. If you don’t handle your custom attributes in setup scripts, you are not able to install your project from scratch, which could cause a lot of problems. For example, it makes it harder to run unit- and integration tests.

So lets assume that we are working in a team with 4 developers and everybody has a different task. Each task needs a custom attribute in a different entity_type.

I want to share my thoughts and summarize an overview of how the 4 developers could implement their setup scripts. Also, I want to show the advantages and disadvantages of each method.

Read More Read More