How to speed up your Magento import

Previously, we talked about how to automate your magento import using ssh, no browser needed. Today we will talk about speeding up this import by doing just a small modification. By default Magento's indexes are all set to "Update on Save". If we change this value to "Manual Update" our import will run about twice as fast, but the downside is that we are unable to update certain area's of the website and see the changes made immediately (update categories, remove categories, move categories, add/remove products from categories, update product stocks, etc.).

The solution is simple. Set the indexes to "Manual Update" when we start importing and set it to "Update on Save" when we are done. I thought about doing this by updating the database or something, but there is a much easier way. (note that this solution is 1.4+ only).

Set an indexer to manual update:

And change it back:

We can implement it by opening our import.sh file and disable it before the import and enable it after:

That's all. Easy improvement!

Update 24 Sep.

Since I setup to be notified every day about the cron results, I noticed that my import was unimaginably slow. My dayly productimport of about 15000 products took 10-15 hours! This is about 2.5 seconds per product (10*3600/15000), not acceptable. I also noticed that some imports were (relatively) fast. Those imports only took 1.5 to 2.5 hours, thats 0.5 seconds per product (2*3600/1500). After investigating a bit further it seemed that my cache directory was about 1 gig, maybe that could be it. Indeed it was, after flushing the cache my import was at its original speed.

I added the following line to the import.sh script right before the actual mag_import.php is executed:

Be absolutely sure that the path is correct! If you have a wrong path set here and that path is a folder as well, you risk deleting files you dont want.

H&O is expert in het oplossen van complexe Magento vraagstukken, maatwerk grafisch ontwerp en het bouwen van custom Magento modules.

Als je op zoek bent naar een partner die je kan helpen met het beantwoorden van moeilijke Magento vragen, een maatwerk productimport voor je ontwikkeld of een custom Magento module voor je schrijft die jouw assortiment met een andere webshop koppelt, dan ben je aan het juiste adres. H&O heeft zich sinds 2007 verdiept in de techniek en ontwikkelingen van het e-commerce CMS, speelt een actieve rol in de community, heeft verschillende modules uitgebracht en heeft voor klanten uiteenlopende technische modules ontwikkeld. Ons kantoor is gevestigd in Roelofarendsveen, direct naast de A4 én slechts op 15 minuten afstand van Amsterdam. Neem contact met ons op en kom eens langs, dan bespreken we jouw vraagstuk!

Paul Hachmang

Paul Hachmang is a Back-end Developer from Roelofarendsveen (The Netherlands) and co-founder of H&O. His weapons of choice include PHP, MySQL, HTML, Javascript, ExpressionEngine, Magento and a good cup of coffee. Contact Paul Hachmang

Comments
  • dweeves

    (nov 08)
    Reply to this comment

    For importing products faster and with many neat features (like computed values from csv,custom options import) , i would suggest you to have a look at magmi.
    To know more about it: just go to magento forum,dataflow thread , the post with 1790+ responses is the magmi thread.
    Magmi also has a wiki and it’s free.
    Ah , of course, it can import & update products at the speed of light compared to dataSlow.

  • Roy

    (okt 09)
    Reply to this comment

    Thanks for the information. But I have a trouble…I’m a newbie and it is too difficult for me. I did everything which was mentioned above but got an error. “No path found_!”...
    the only thing that helped me was this service. I used it because it is error-fre and very fast. But it is paid…that’s why i’m asking you how to solve this problem.
    Thank you very much!

  • Bob

    (dec 12)
    Reply to this comment

    Indeed when a large number of goods or attributes need to be saved, backend works very slow, as each time you save data the system re-indexes the goods through the entire catalog.

    Asynchronous Re-indexing is a solution of this problem. When a product or category is saved it is not immediately re-indexed, but put into a queue. The queue is re-indexed in the background. This greatly speeds up the backend. This mechanism is implemented by using an extension of http://mirasvit.com/magento-extensions/magento-asynchronous-reindex.html.

    Even in case you place online shop even on a good hosting, sometimes the products, categories and attributes are saved slowly. This is due to the need to clear the cache and run re-indexing of stored items.

  • Tony

    (dec 31)
    Reply to this comment

    Hello !
    Thanks for your help in community Magento…. :)

    i try your script (mag_import) and others solutions for imports CSV in MAGENTO Catalog Products
    ALL => WORKS => PERFECTLY (no erreur, no notice….)

    My CSV’s rows are in database ‘dataflow_batch_import’
    Profil run completely…

    i run all index (auto / manual) with ssh command
    BUT NOTHING > ANY PRODUCT IN BACK OFFICE are available
    (for the moment i try just update price on two productID)

    so my csv is very simple and short :
    sku;price
    001;50
    002;100

    and…. when i try import the same profil on Magento Administration
    my products are updated…
    have you an idea ?!
    thx

  • Magento Extensions

    (jun 27)
    Reply to this comment

    Bedankt voor de handige article.I ben nieuwe ontwikkelaar van magento.This artic le hielp me uit.

Reageren

Contact

Bel mij terug

Vul uw telefoonnummer in en we bellen u zo spoedig mogelijk terug.

Contactgegevens

H&O
Veenderveld 20
2371 TV
Roelofarendsveen
info@h-o.nl
www.h-o.nl
071 744 0084
KvK Rijnland nr. 28119089
BTW nr. NL818554071B01

Nieuwe projecten

Wilt u weten wat H&O kan betekenen voor uw Magento project? Stuur ons een mailtje:

Paul Hachmang

p.hachmang@h-o.nl

Erwin Otten

e.otten@h-o.nl

Contactformulier