How to speed up your Magento import

Previously, we talked about how to automate your magento import using ssh, no browser needed. Today we will talk about speeding up this import by doing just a small modification. By default Magento's indexes are all set to "Update on Save". If we change this value to "Manual Update" our import will run about twice as fast, but the downside is that we are unable to update certain area's of the website and see the changes made immediately (update categories, remove categories, move categories, add/remove products from categories, update product stocks, etc.).

The solution is simple. Set the indexes to "Manual Update" when we start importing and set it to "Update on Save" when we are done. I thought about doing this by updating the database or something, but there is a much easier way. (note that this solution is 1.4+ only).

Set an indexer to manual update:

And change it back:

We can implement it by opening our import.sh file and disable it before the import and enable it after:

That's all. Easy improvement!

Update 24 Sep.

Since I setup to be notified every day about the cron results, I noticed that my import was unimaginably slow. My dayly productimport of about 15000 products took 10-15 hours! This is about 2.5 seconds per product (10*3600/15000), not acceptable. I also noticed that some imports were (relatively) fast. Those imports only took 1.5 to 2.5 hours, thats 0.5 seconds per product (2*3600/1500). After investigating a bit further it seemed that my cache directory was about 1 gig, maybe that could be it. Indeed it was, after flushing the cache my import was at its original speed.

I added the following line to the import.sh script right before the actual mag_import.php is executed:

Be absolutely sure that the path is correct! If you have a wrong path set here and that path is a folder as well, you risk deleting files you dont want.