Automatic import with Magento using SSH, no browser needed

For one of our Magento webshops, one of the requirements was that the product feed was imported automatically overnight. This ment that I had to find a way to automate this process, but since Magento doesn't support a non-browser productimport this was easier said then done. After looking for a while, I found the following topic on the magento forums. This looked good, this meant I could use the regular Import/Export profiles and run them in the shell. After investigating a bit further, I discovered I could also use Advanced Import/Export profiles - what was just what I needed. So I set up my Advanced profile (one for importing products, one for importing product relations) and did my first first testrun. Everything went well, but after about 1000 - 2000 products It suddenly crashed: PHP Allowed Memory Size Exchausted Fatal Error. That is just great ...

Please note: This functionality was created for Magento 1.4.0.1 and is not tested for Magento 1.5+ and therefor may not work as expected. This script has been replaced with a new script (tested with 1.6.2 and 1.7.0), which can be found on H&O's GitHub Page.

Importing a large quantity of products via the shell.

I wondered what happened. After logging my memory usage a bit more, I noticed that after every product that was saved, the memory usage got a little bit higher every time until it eventually crashed. How could this happen, and why does the regular import doesn't have this problem? It seems that Magento doesn't free the used memory AFTER saving a product. So after a 1000-2000 product saves, the system is out of memory. One other thing I noticed, is that the import also gets much slower after imported a few hundred products.

Why does the browser import doesn't have this problem? Why doesn't it crash and why doesn't it get slower? That is because the browser import does something different:

  1. When the importer loads (when the user has pressed run) it parses the complete source file (csv, xml) and places it in a temporary database table (dataflow_batch_import).
  2. From that table it creates a list of all the rows that need to be imported and it sends that list to the browser.
  3. The browser creates a Javascript Ajax request back to magento for every batch (the number you configured during import).
  4. When Magento receives that request, magento starts, imports the batch, sends the results back to the page and closes.
  5. When the page receives the results back, it prints them, and starts over with step three.

This process solves the problem that Magento will run out of memory and the import getting slower. If we were to call the importpage from the shell in some way, this solution wouldn't work, since the page depends on Javascript to place it's Ajax requests. What we need to do, is recreate this step within the shell. Since the shell can't execute Javascript we'll have to recreate this in PHP (in the shell, PHP is allowed to run for an infinite amount of time). In PHP we create cURL requests instead of Ajax requests. We fully simulate the browser-import-situation.

The code consists of two files. mag_login.php and mag_import.php. Both are needed. Place them in /PATH/TO/YOUR/MAGENTOINSTALLATION/shell/

mag_import.php:

mag_login.php:

 

Copy the above code to your local Magento installation, put them in a secure folder, that isn't accesible by a browser and login to the shell. Fill in the correct information into the files. After that is done, start up your shell, browse to the folder. Run the file with:

In the line above, 7 should be replaced by your own profile Id, which can be found the first column of the profiles.

Making the Magento product import fully automatic.

If we got the above working, we need to get the working fully automatic. We want to import our products and do a complete reindex of magento every night. First we create a file that can be executed by the shell that imports the products and reindexes it. Call it something like import.sh (sh from shell):

Try and run the file with the following:

If that all works properly, we can setup our cronjob to do it automatically every night. Enter the following into you shell:

If you already have setup your Magento cron, you should see something like:

Now, if something happens to our import, somethings goes wrong, etc, we wan't to get notified. Add the following line at the top of the crontab file.

Below the Magento line we add our import cron:

Now save the crontab (depends on the editor how) and your all set.

Note: if you want to see error regarding the import, lines that could not be processed open /PATH/TO/YOUR/MAGENTOINSTALLATION/var/log/import.log or browse to the file in the shell and type:

Further reading:

Making your import faster is always great: How to speed up your magento import.

H&O is expert in het oplossen van complexe Magento vraagstukken, maatwerk grafisch ontwerp en het bouwen van custom Magento modules.

Als je op zoek bent naar een partner die je kan helpen met het beantwoorden van moeilijke Magento vragen, een maatwerk productimport voor je ontwikkeld of een custom Magento module voor je schrijft die jouw assortiment met een andere webshop koppelt, dan ben je aan het juiste adres. H&O heeft zich sinds 2007 verdiept in de techniek en ontwikkelingen van het e-commerce CMS, speelt een actieve rol in de community, heeft verschillende modules uitgebracht en heeft voor klanten uiteenlopende technische modules ontwikkeld. Ons kantoor is gevestigd in Roelofarendsveen, direct naast de A4 én slechts op 15 minuten afstand van Amsterdam. Neem contact met ons op en kom eens langs, dan bespreken we jouw vraagstuk!

Paul Hachmang

Paul Hachmang is a Back-end Developer from Roelofarendsveen (The Netherlands) and co-founder of H&O. His weapons of choice include PHP, MySQL, HTML, Javascript, ExpressionEngine, Magento and a good cup of coffee. Contact Paul Hachmang

Comments
  • Alta

    (okt 05)
    Reply to this comment

    What version Magento?

  • Paul Hachmang

    (okt 07)
    Reply to this comment

    This script is tested on Magento 1.4.0.1, but I think it will also work on 1.4.1.x and 1.3.3.0.

  • David O B

    (okt 07)
    Reply to this comment

    Hi there,

    I followed and tested your script in in magento 1.4.1 however the products
    don’t import and the import.log file show 403 permision dined errors.

    Any idea what might be causing this.

    This is an excellent feature and would really like to get in working.

    Log File Output

    2010
    -10-07T14:35:48+00:00 DEBUG (7):


    2010-10-07T14:35:48+00:00 DEBUG (7): 7.55 mb STARTING IMPORT
    2010
    -10-07T14:35:48+00:00 DEBUG (7): 9.03 mb Table dataflow_batch_import cleaned
    2010
    -10-07T14:35:48+00:00 DEBUG (7): 9.03 mb Preparing profile...
    2010-10-07T14:36:18+00:00 DEBUG (7): 12 mb - ...Done
    2010
    -10-07T14:36:18+00:00 DEBUG (7): 12 mb Loaded batch id 208
    2010
    -10-07T14:36:18+00:00 DEBUG (7): 17.12 mb 0/20428
    2010
    -10-07T14:36:18+00:00 DEBUG (7): 17.13 mb 50/20428 [<?xml version="1.0" encoding="ISO-8859-1"?>
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
      "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"
    >
    <
    html >
    <
    head>
    <
    title>Access forbidden!</title>
    <
    link rev="made" href="mailto:webmaster@localhost" />
    <
    style type="text/css"><!--/*--><![CDATA[/*><!--*/
        
    body { color#000000; background-color: #FFFFFF; }
        
    a:link { color#0000CC; }
        
    paddress {margin-left3em;}
        span {font
    -sizesmaller;}
    /*]]>*/--></style>
    </
    head>

    <
    body>
    <
    h1>Access forbidden!</h1>
    <
    p>




        
    You don't have permission to access the requested directory.
        There is either no index document or the directory is read-protected.



    </p>
    <p>
    If you think this is a server error, please contact
    the <a href="mailto:webmaster@localhost">webmaster</a>.

    </p>

    <h2>Error 403</h2>
    <address>
      <a href="/">dev.mydomain.com</a><br />

      <span>Thu Oct  7 15:36:18 2010<br /> 

    and

    2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load
    2010
    -10-07T14:43:18+00:00 DEBUG (7): 17.12 mb Loaded successfully"/var/www/mydomain/public/httpdocs/var/import/my_products.csv".
    2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
    2010
    -10-07T14:43:18+00:00 DEBUG (7): 17.12 mb Found 20428 rows.
    2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb Starting catalog/convert_adapter_product :: parse
    2010
    -10-07T14:43:18+00:00 DEBUG (7): 17.12 mb Completed!
    2010-10-07T15:04:03+00:00 DEBUG (7): 

     

  • David O B

    (okt 08)
    Reply to this comment

    When I run mag_login.php with “php mag_login.php” it just runs no error on screen, no inputs to import.log file

  • David O B

    (okt 08)
    Reply to this comment

    If I run it from the browser I get the message: The page isn’t redirecting properly

  • Paul Hachmang

    (okt 09)
    Reply to this comment

    Hi David,

    It seems that the login isn’t functioning properly. I can reproduce it here as well.

  • MagentoKnight

    (okt 21)
    Reply to this comment

    Same here. Can anyone get this to work?

  • Idealclic

    (nov 10)
    Reply to this comment

    I can’t make it work !
    Can you explain the difference between
    Path to the root of your magento installation
    and
    Url to your magento installation ?

    Also, it is said to put mag_login.php in shell and shell in the root, so is dataflow/shell/mag_login.php, the correct path for “relative path from the magento root to the login file.” ?

    Many thanks.

  • Paul Hachmang

    (nov 11)
    Reply to this comment

    Your URL is the webaddress you type in your browser, starts with http:// etc. Your Path is the location of your magento installation on the harddisk on your server. If you don’t know what it is, create a php file and put <?php echo $_SERVER[‘PHP_SELF’]; ?> inside, and you’ll see your server path.

    As for the relative path to the login file, i’ve changed it, the dataflow was something i used in my own application shouldn’t be there, changed it in the post.

  • Idealclic

    (nov 11)
    Reply to this comment

    Hi Paul,

    Thank you for your answer !

    Now I’m getting this error in ssh when I run php -f mycron/mag_import.php 11
    PHP Warning:  Unknown(): Unable to load dynamic library ‘/usr/lib/php/extensions/no-debug-non-zts-20020429/mcrypt.so’ - /usr/lib/php/extensions/no-debug-non-zts-20020429/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0


    Parse error:  syntax error, unexpected T_OBJECT_OPERATOR in /homepages/6/d274382575/htdocs/magento-prod/mycron/mag_import.php on line 52


    Any idea ?

    Thank you

    • Tiron

      (feb 27)
      Reply to this comment

      Hi
      I got this issue with Godaddy shared hosting,this happen because of php version issue. Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID); this is compatible with php 5 or higher.
      I fixed it by using
      /web/cgi-bin/php5 -f path_to_the_php_file.php 8

  • Idealclic

    (nov 11)
    Reply to this comment

    Well, I solved the first error by deleting the line
    extension=mycript.so in php.ini
    (not sure it is the best way to do it)

    But I still have this error:

    Parse error:  syntax error, unexpected T_OBJECT_OPERATOR in /homepages/6/d274382575/htdocs/magento-prod/mycron/mag_import.php on line 52

  • Magento Knight

    (nov 11)
    Reply to this comment

    Re: Paul on OCT/9, Has the login code been updated to work?

  • Eduard Fabra

    (nov 25)
    Reply to this comment

    Hi,

    I modified the login code, and now I can run it without problems. Here are my code:

    <?php

     
    /**
      * Path to the root of your magento installation.
      * include traing slash.
      */
    $root 'your/magento/installation/path';
     
     
    /**
      * Username that has the rights to import products.
      */
     
    $username 'your_user';
     
     
    /**
      * Password
      */
     
    $password "your_password";
     
     
    /**
      * DO NOT EDIT BELOW THIS LINE
      */
    //getting Magento
    require_once $root.'app/Mage.php';
    umask(0);
    Mage::app();

    $user Mage::getModel('admin/user');
    if (
    $user->authenticate($username$password)) 
    {    
        Mage
    ::getSingleton('admin/session')->setUser($user);
        
        if (
    $user->getId())
        
    {
            
    if (Mage::getSingleton('adminhtml/url')->useSecretKey()) {
                Mage
    ::getSingleton('adminhtml/url')->renewSecretUrls();
            
    }
            $session 
    Mage::getSingleton("admin/session");
            
            
    //Change the owner of the session file on root/var/session/ to the user that runs the webserver
            
    exec("chown nobody:nobody {$root}var/session/sess_".$session->getEncryptedSessionId());
            
            echo 
    json_encode(array('sessionId' => $session->getEncryptedSessionId(), 'formKey' => Mage::getSingleton('core/session')->getFormKey()));
        
    }
    }
    ?> 

    I run the process in a cron and It works well with a Magento 1.4.1.1. The problem with the original mag_login.php was that the $session->login($username, $password), the Magento login function was calling a header with a redirect to the html login page, in this version I replaced it with a call to $user->authenticate($username, $password). Finally, I added a chown command to the session file to change the owner of this file because if not, the curl session, that runs with the user of the webserver,  can’t not access to session data.

    Hope it helps.


    Eduard

  • Paul Hachmang

    (nov 25)
    Reply to this comment

    Whoa, that is great Eduard!

  • Bojan

    (dec 04)
    Reply to this comment

    Great script, but, even with the changed login I cant get it up and running.
    I am getting the same error:
    2010-12-04T18:22:57+00:00 DEBUG (7): 10,96 mb - 3150/3292 [{“ajaxExpired”:1,“ajaxRedirect”:“htt…

    Any help would we apritiated.

  • Allard Muis

    (dec 07)
    Reply to this comment

    I couldn’t get the login to work either (ajaxExpired).
    So I changed the login procedure to work through cURL as well.
    My version og mag_login.php:

    <?php

    /**
    * Path to the root of your magento installation.
    * include traing slash.
    */
    $root '/home/domainname/public_html/orsomethinglikethis';

    /** * Url to your magento installation. (include trailing /)*/
    $url 'http://www.domain.com/';

    /**
    * Username that has the rights to import products.
    */
    $username 'XXX';
     
    /**
    * Password
    */
    $password 'XXX';

    /**
    * DO NOT EDIT BELOW THIS LINE
    */

    $profileId $argv[1];


    $ch curl_init();
    curl_setopt($chCURLOPT_COOKIESESSION);
    curl_setopt$chCURLOPT_NOBODY);
    curl_setopt($chCURLOPT_URL,$url."index.php/admin/");
    curl_setopt($chCURLOPT_RETURNTRANSFER1);
    curl_setopt($chCURLOPT_HEADERtrue); // header will be at output
    curl_setopt($chCURLOPT_POST1);
    curl_setopt($chCURLOPT_POSTFIELDS'login[password]='.$password.'&login;[username]='.$username );
    $response curl_exec($ch);


    $adminhtml strpos$response'adminhtml=' )+10;
    $semic strpos$response';'$adminhtml );
    $cookie substr$response$adminhtml$semic-$adminhtml );

    $ch curl_init();
    curl_setopt($chCURLOPT_COOKIESESSION);
    curl_setopt($chCURLOPT_URL,$url."index.php/admin/system_convert_profile/run/id/".$profileId."/");
    curl_setopt($chCURLOPT_RETURNTRANSFER1);
    curl_setopt($chCURLOPT_HEADERtrue); // header will be at output
    curl_setopt($chCURLOPT_COOKIE"adminhtml=".$cookie );
    $response curl_exec($ch);

    $keypos strpos$response'var FORM_KEY = "' )+16;
    $quote strpos$response'"'$keypos+);
    $key substr$response$keypos$quote-$keypos );


    echo 
    json_encode(array('sessionId' => $cookie'formKey' => $key ));

    ?>


    Change a few lines in mag_import 
    as well:

    Change this:
    /**
    * get het login information.
    */

    exec("php -f {$root}{$login}"$result);
    $loginInformation json_decode(substr($result[0],3));

    into this:

    /**
    * get het login information.
    */

    exec("php -f {$root}{$login} {$profileId}"$result);
    $loginInformation json_decode(substr($result[0],3)); 

    For this to work, you have to disable secret keys (System->config->(advanced)->admin->security->add secret keys to url-> NO

    I’m sure a version with secret keys is possible too much work for my currect project.

  • Allard Muis

    (dec 07)
    Reply to this comment

    please note a type in above code:
    curl_setopt($ch, CURLOPT_POSTFIELDS, ‘login[password]=’.$password.’&login;[username]=’.$username );

    The ; should be removed from login;[username]
    Also note that copy-pasting from this form screws up quotes (’ and “) so you should fix these first.

  • Paul Hachmang

    (dec 07)
    Reply to this comment

    It’s great to see people get this working. We currently have it running stable for 4 months now, no real problems encountered. This month i have another project that requires the cron-import as well and will use the suggested modifications. After that i will post an update that combines it into a nice package.

  • bojan

    (dec 17)
    Reply to this comment

    Thanks for all the replies.

    Even with all the above code, I still get ajaxExpired when I try to import. I will try to play around with the server security a bit and let you know if it worked :)

  • Rich

    (jan 13)
    Reply to this comment

    Re this part of the suggested changes:

    exec(“php -f {$root}{$login} {$profileId}”, $result);
    $loginInformation = json_decode(substr($result[0],3));

    Causes the:
    [{“ajaxExpired”:1,“ajaxRedirect”:“htt…
    problem that Bojan has mentioned

    It should be:
    exec(“php -f {$root}{$login} {$profileId}”, $result);
    $loginInformation = json_decode($result[0]);

    That fixed it for me, hope it helps

  • Christian Schwan

    (jan 18)
    Reply to this comment

    Less code and without login.php. Tested with 1.4.2 and Advanced Profiles.

    <?php
    $root = ‘/home/www/html/magento/’;
    $logFileName= ‘import.log’;

    /**
    * DO NOT EDIT BELOW THIS LINE
    */

    //THIS SCRIPT JUST INITIALS THE PROFILE TO BE RUN VIA MAGENTO ADMIN "RUN PROFILE IN POPUP". Its the same thing as click just via this file that you can run via cron
    $profileId $argv[1];
    if (! isset(
    $profileId)) {
    exit ("\nPlease specify a profile id. You can find it in the admin panel->Import/Export->Profiles.\nUsage: \n\t\t php -f $argv[0] PROFILE_ID\n\t example: php -f $argv[0] 7\n");
    }

    function convert ($size{
    $unit
    =array('b','kb','mb','gb','tb','pb');
    return @
    round($size/pow(1024,($i=floor(log($size,1024)))),2).' '.$unit[$i]
    }
    set_time_limit
    (0);
    ini_set('memory_limit''128M');

    require_once 
    $root.'app/Mage.php';
    umask(0);
    Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
     
    $profile Mage::getModel('dataflow/profile');
    $userModel Mage::getModel('admin/user');
    $userModel->setUserId(0);
    Mage::getSingleton('admin/session')->setUser($userModel);
    $profile->load($profileId);



    if (!
    $profile->getId()) {
        Mage
    ::getSingleton('adminhtml/session')->addError('ERROR: Incorrect profile id');
    }
     
    Mage
    ::register('current_convert_profile'$profile);
    //starting the import
    Mage::log("\n\n"null$logFileName);
    Mage::log(convert(memory_get_usage()) . " - " "STARTING IMPORT"null$logFileName);
    Mage::log(convert(memory_get_usage()) . " - " "Preparing profile..."null$logFileName);
    $profile->run();
    Mage::log(convert(memory_get_usage()) . " - " "...Done"null$logFileName);
    $recordCount 0;
    $batchModel Mage::getSingleton('dataflow/batch');
    Mage::log(convert(memory_get_usage()) . " - " "Completed!"null$logFileName);
    echo 
    "EXPORT COMPLETE. BATCHID: " $batchModel->getId(); 

     

  • Drew

    (mar 22)
    Reply to this comment

    Getting an ajaxExpired error when you run from cron and don’t know why? Check your path variables!

    This code will fail:
    exec(“php -f {$root}{$login} {$profileId}”, $result);

    This code works on my system:
    exec(“/usr/local/bin/php -f {$root}{$login} {$profileId}”, $result);

  • Markus Fuchs

    (apr 04)
    Reply to this comment

    this is the content of my logfile:
    2011-04-04T21:43:36+00:00 DEBUG (7):


    2011-04-04T21:43:36+00:00 DEBUG (7): 8.68 mb - STARTING IMPORT
    2011-04-04T21:43:37+00:00 DEBUG (7): 10.84 mb - Table dataflow_batch_import cleaned
    2011-04-04T21:43:37+00:00 DEBUG (7): 10.84 mb - Preparing profile…
    2011-04-04T21:43:37+00:00 DEBUG (7): 13,88 mb - ...Done
    2011-04-04T21:43:37+00:00 DEBUG (7): 13,88 mb - Loaded batch id 103
    2011-04-04T21:43:37+00:00 DEBUG (7): 13,92 mb - 0/1
    2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - 1/1 [{“savedRows”:1,“errors”:[]}]
    2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load
    2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Loaded successfully: “/home/hammermu/public_html/var/import/import_produktpreise_dv_ev2.csv”.
    2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
    2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Found 1 rows.
    2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Starting catalog/convert_adapter_product :: parse
    2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Completed!

    but the product still is not updated.
    what am i doing wrong?

  • Inz

    (mei 19)
    Reply to this comment

    This is so not working at all….. Could someone clarify this thread…

  • Djé

    (jun 09)
    Reply to this comment

    Hy,

    No log files created, and no imports so… Do you have an idéa?

  • Josh

    (aug 03)
    Reply to this comment

    I couldn’t get this working at all on 1.5.1

    Doesn’t create a log file either. Very annoying.

  • Nick

    (aug 12)
    Reply to this comment

    Tried the script and got this error in the shell:

    chowncannot access `/var/www/vhosts/outlet.lenovo.com/var/session/sess_bauni8sfv36ra2k5mujm07mop1': No such file or directory                                                                                                     
    PHP Fatal error:  Uncaught exception 'Exception' with message 'Notice: Trying to get property of non-object  in /var/www/vhosts/outlet.lenovo.com/shell/mag_import.php on line 302' in /var/www/vhosts/outlet.lenovo.com/app/code/core/Mage/Core/functions.php:245 

    Here’s my log file:

    2011-08-12T15:38:59+00:00 DEBUG (7): 

    2011-08-12T15:38:59+00:00 DEBUG (7): 6.59 mb STARTING IMPORT
    2011
    -08-12T15:39:00+00:00 DEBUG (7): 7.88 mb Table dataflow_batch_import cleaned
    2011
    -08-12T15:39:00+00:00 DEBUG (7): 7.88 mb Preparing profile...
    2011-08-12T15:39:00+00:00 DEBUG (7): 10.47 mb - ...Done
    2011
    -08-12T15:39:00+00:00 DEBUG (7): 10.47 mb Loaded batch id 143
    2011
    -08-12T15:39:00+00:00 DEBUG (7): 10.49 mb 0/2
    2011
    -08-12T15:39:01+00:00 DEBUG (7): 10.56 mb 2/2 [<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    <
    html xml:lang="en" lang="en">
    <
    head>
    <
    title>Lenovo Outlet 404 Not Found 1 Discounted LaptopsCheap LaptopsRefurbished LaptopsDiscounted DesktopsCheap DesktopsRefurbished Desktops</title>
    <
    meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> 

    etc with more html from 404 page.

    What file is it trying to find that it can’t? I’m assuming thats why I’m getting this error.

    • Nick

      (aug 12)
      Reply to this comment

      For clarification, Line 302 in the mag_import file is:

      301:   Mage::log(convert(memory_get_usage()) . " - {$recordCount}/{$totalproducts} [$buffer]null$logFileName);
      302:         if (count($result->errors))
      303
  • Matt Ramage

    (sep 10)
    Reply to this comment

    Has anyone solved the ajaxExpired error. Ive tried everything here with no luck. im using magento 1.5.1

  • Cao

    (sep 15)
    Reply to this comment

    I got this error:

    Fatal errorUncaught exception 'Zend_Db_Adapter_Exception' with message 'pdo_mysql extension is not installed' in /home/developer/public_html/stoy/lib/Varien/Db/Adapter/Pdo/Mysql.php:289
    Stack trace
    :
    #0 /home/developer/public_html/stoy/lib/Zend/Db/Adapter/Abstract.php(459): Varien_Db_Adapter_Pdo_Mysql->_connect()
    #1 /home/developer/public_html/stoy/lib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('SET NAMES utf8', Array)
    #2 /home/developer/public_html/stoy/lib/Varien/Db/Adapter/Pdo/Mysql.php(389): Zend_Db_Adapter_Pdo_Abstract->query('SET NAMES utf8', Array)
    #3 /home/developer/public_html/stoy/app/code/core/Mage/Core/Model/Resource.php(169): Varien_Db_Adapter_Pdo_Mysql->query('SET NAMES utf8')
    #4 /home/developer/public_html/stoy/app/code/core/Mage/Core/Model/Resource.php(110): Mage_Core_Model_Resource->_newConnection('pdo_mysql', Object(Mage_Core_Model_Config_Element))
    #5 /home/developer/public_html/stoy/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(320): Mage_Core_Model_Resource->getCon in /home/developer/public_html/stoy/lib/Varien/Db/Adapter/Pdo/Mysql.php on line 289

    But when i check  system requirement :

        
    You have PHP 5.2.0 (or greater)
        
    Safe Mode is off
        You have MySQL 4.1.20 
    (or greater)
        
    You have the curl extension
        You have the dom extension
        You have the gd extension
        You have the hash extension
        You have the iconv extension
        You have the mcrypt extension
        You have the pcre extension
        You have the pdo extension
        You have the pdo_mysql extension
        You have the simplexml extension 

    I confused with this problem. Can someone help me . Thanks

  • Cao

    (sep 15)
    Reply to this comment

    This function make the error:

    Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID); 

    But i dont know how to fix it.

  • stephane

    (sep 19)
    Reply to this comment

    Hello paul, thanks to you to share your work.
    I install on a 1.5.1. No major error BUT…
    I got theses logs :

    2011-09-19T21:35:46+00:00 DEBUG (7): 8.35 mb STARTING IMPORT
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 9.63 mb Table dataflow_batch_import cleaned
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 9.63 mb Preparing profile...
    2011-09-19T21:35:46+00:00 DEBUG (7): 13,4 mb - ...Done
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 13,4 mb Loaded batch id 9096
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 13,44 mb 0/5
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 13,44 mb 5/Response is empty - ERROR
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 13,44 mb Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 13,44 mb Loaded successfully"/xxxx/xxxx/var/import/drop_elem.csv".
    2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 13,44 mb Found 5 rows.
    2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb Starting catalog/convert_adapter_product :: parse
    2011
    -09-19T21:35:46+00:00 DEBUG (7): 13,44 mb Completed

    so the $ch always empty. No update is done of course
    Could you help me please ?
    best regards
    stephane

  • Gareth Shaw

    (sep 25)
    Reply to this comment

    I get the same errors are Stephane above on v.1.3.2.4

    2011-09-25T14:05:07+00:00 DEBUG (7): 5.74 mb - STARTING IMPORT
    2011-09-25T14:05:07+00:00 DEBUG (7): 6.88 mb - Table dataflow_batch_import cleaned
    2011-09-25T14:05:07+00:00 DEBUG (7): 6.88 mb - Preparing profile…
    2011-09-25T14:05:40+00:00 DEBUG (7): 9.78 mb - ...Done
    2011-09-25T14:05:40+00:00 DEBUG (7): 9.78 mb - Loaded batch id 7
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 0/3533
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #1
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 50/3533 - Response is empty - ERROR
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #2
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 100/3533 - Response is empty - ERROR
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #3
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 150/3533 - Response is empty - ERROR
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #4
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 200/3533 - Response is empty - ERROR
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #5
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 250/3533 - Response is empty - ERROR
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #6
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 300/3533 - Response is empty - ERROR
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #7
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 350/3533 - Response is empty - ERROR
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #8
    2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 400/3533 - Response is empty - ERROR
    .............

    2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Start cURL request #71
    2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - 3533/3533 - Response is empty - ERROR
    2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load
    2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Loaded successfully: “/var/www/live/smilesextoys.co.uk/var/import/import_all_products.csv”
    2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
    2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Found 3533 rows
    2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Starting catalog/convert_adapter_productimport :: parse
    2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Completed!

    This will be the 5th import script I’ve tried, none of which I can get to work.

    Thanks
    Gareth

  • Dave Pearson

    (sep 27)
    Reply to this comment

    If your log file is not getting written, make sure you have logging enabled in:

    System / Configuration / (Advanced) Developer / Log Settings / Enabled = Yes

  • Dave Pearson

    (sep 27)
    Reply to this comment

    I’m getting the “ajaxExpired” error message on Magento 1.6. I have tried all suggestions in this thread but none are working.

    Any ideas would be much appreciated?

  • Stuart Drennan

    (okt 13)
    Reply to this comment

    Hi, I’m getting 550/550

    [{"ajaxExpired":1,"ajaxRedirect":"https:\/\/************.com\/index.php\/admin\/index\/login\/"}] 

    on 1.4.2 even using the modified version from Christian Schwan.  Thoughts anyone??  Pllllleeeeaassssee :(

  • Auto importeren

    (jan 12)
    Reply to this comment

    Nice article!

  • DW

    (feb 22)
    Reply to this comment

    Has anyone solved the ajaxExpired error?  It’s driving me crazy.

  • Daniel Gafitescu

    (mar 30)
    Reply to this comment

    Instead of

    $root = ‘/PATH/TO/YOUR/MAGENTOINSTALLATION/’; you can do dirname(dirname(__FILE__));

  • website design

    (apr 27)
    Reply to this comment

    This is working on Magento 1.5 too. Just you need to add a column in csv ‘has_options’ & set the value to ‘1’ for the products which are having custom options.

  • Magento UK

    (mei 16)
    Reply to this comment

    is the script updated for 1.6? anyone tried?

  • Daniel

    (jul 27)
    Reply to this comment

    This Script work perfect on Magento 1.7
    Thank you very much for this wonderfull Tutorial!

    Can i import Customers with this Script and in the shell file take Profile 6, the standart Profile from Magento?

  • Annemieke de rijp

    (sep 10)
    Reply to this comment

    I have just tried the script on Magento 1.7 and it works very well without problems. I can import a large quantity of products via the shell. Easily. Thanks for the great script. Looking forward to see more useful magento scripts.

    Annemieke

  • custom papers

    (sep 26)
    Reply to this comment

    Thanks this post really opened my eyes. It is not only eye opening rather very beneficial for the people who wants to do somethin

  • best cover letter ever

    (sep 26)
    Reply to this comment

    Wow, this actually cleared some things up. One essay i am having headache with is a footnote for an “who we are” The teacher says that we will have to stab below into ourselves and notice what we really are. But I think it would be more usefulto use

  • thesis writers

    (okt 14)
    Reply to this comment

    I really enjoyed the quality information you offer to your visitors for this blog. I will bookmark your blog and have my friends check up here often.

  • spss in research methodology

    (okt 16)
    Reply to this comment

    I am currently looking into a career change and this post helped to see another aspect of the industry. I appreciate your article

  • (nov 07)

Reply to this comment

Your article is clearly very well-researched and informative. I think your content is useful and valuable information presented in a very unique way.

  • bojan_dj

    (jan 18)
    Reply to this comment

    For ajaxEpired error try with “youturl/” . index.php/admin/admin/system_convert_profile/batchRun/?isAjax=true
    That solved the problem for me

  • Giorgio

    (jun 03)
    Reply to this comment

    Hi I am trying to use this on Magento 1.7.
    I can run export profiles without problems (if you need code I can post) - furthermore :
    - login is not needed for export
    - curl is not needed for export

    for import I have more problem:
    - Curl is returning empty buffer
    - it could be a login problem ? Also loggin in ... I get empty buffer
    - it could be a url problem ? in Magento 1.7 source code of pop up import page is
    Ajax.Request(“nydomain.comindex.php/admin/system_convert_gui/batchRun/key/fbcc588258bd52aa1e632147784661fe/”

    is “system_convert_gui” different from “current_convert_profile” ?
    why don’t you simply save data in a loop with some save() method ?

    any suggestion will be appreciated

  • Pieter

    (jul 05)
    Reply to this comment

    I understand when this idea is working, it works for every version and you are sure you have all the new Magento functionality. But reading all the forums this is still a problem. 
    I created already 2 years ago a import cron script based on Magmi and it works very speedy, besides I can handle all the specific requirements for my customers. But when the creator of Magmi stops and nobody takes it over, its the end ...

  • Andrey

    (sep 16)
    Reply to this comment

    First of all, thanks for great article. My experiment with source script wasn’t successfull. But it gave me a lot to think about. Here is mine solution. It took me whole day to figure out that status field is not parsed correctly. Batch processor should recieve only Enabled-Disabled values. No curl at all. I believe it would work in any version. Most of code comes from app/code/core/Mage/Adminhtml/controllers/System/Convert/ProfileController.php

    <?php
    $root = ‘/...www/’;

    //how many products will be parsed at each post. Usually 10-50
    $atOnce = 30;

    function convert($size)
    {
      $unit=array(‘b’,‘kb’,‘mb’,‘gb’,‘tb’,‘pb’);
      return @round($size/pow(1024,($i=floor(log($size,1024)))),2).’ ‘.$unit[$i];
    }

    set_time_limit(0);
    ini_set(‘memory_limit’, ‘128M’);

    if (!isset($argv[1])) {
      echo “Please specify a profile id. See Admin->Import/Export->Profiles.\nUsage: php -f $argv[0] PROFILE_ID\n”;
      exit;
    }
    $profileId = $argv[1];
    $recordCount = 0;

    //getting Magento
    require_once $root.‘app/Mage.php’;

    ob_implicit_flush();
    Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);
    //Mage::app()->setCurrentStore(0);

    //starting the import
    echo convert(memory_get_usage()) . ” - ” . “STARTING IMPORT\n”;

    //clean dataflow_batch_import table so it doesn’t get amazingly big.
    $db = Mage::getSingleton(‘core/resource’)->getConnection(‘core_write’);
    $db->query(“TRUNCATE TABLE `dataflow_batch_import`”);
    echo convert(memory_get_usage()) . ” - ” . “Table ‘dataflow_batch_import’ cleaned\n”;

    echo convert(memory_get_usage()) . ” - ” . “Special care for ‘status’ field..\n”;
    $str=file_get_contents($root.‘var/import/batch1.csv’);
    $str=str_replace(”Включено”, “Enabled”, $str);
    $str=str_replace(”Отключено”, “Disabled”,$str);
    file_put_contents($root.‘var/import/batch1.csv’, $str);
    unset($str);
    echo convert(memory_get_usage()) . ” - ” . “Special care for ‘status’ field..DONE\n”;

    $profile = Mage::getModel(‘dataflow/profile’);
    //load profile
    if ($profileId) {
      $profile->load($profileId);
      if (!$profile->getId()) {
          echo “ERROR: Could not load profile\n”;
          exit;
      }
    }
    Mage::register(‘current_convert_profile’, $profile);

    //run the profile
    echo convert(memory_get_usage()) . ” - ” . “Preparing profile…\n”;
    $profile->run();
    echo convert(memory_get_usage()) . ” - ” . “Preparing profile…DONE\n”;

    $batchModel = Mage::getSingleton(‘dataflow/batch’);
    if (!$batchModel->getId()) {
      echo convert(memory_get_usage()) . ” - Can’t get batchModel\n”;
      exit;
    }

    if (!$batchModel->getAdapter()) {
      echo convert(memory_get_usage()) . ” - Can’t getAdapter\n”;
      exit;
    }

    //get to work
    $batchId = $batchModel->getId();
    echo convert(memory_get_usage()) . ” - ” . “Loaded batchId $batchId\n”;
    $batchImportModel = $batchModel->getBatchImportModel();
    $importIds = $batchImportModel->getIdCollection();

    $totalproducts = count($importIds);
    echo convert(memory_get_usage()) . ” - Emulate batchRun action for the profile\n”;

    $saved = 0;
    $batchArrIds = array();
    foreach ($importIds as $importId) {
      $recordCount++;

      $batchArrIds[] = $importId;
      if ($recordCount%$atOnce == 0 || $recordCount == $totalproducts) {
          $paramsArr = array(‘batchid’ => $batchId, ‘ids’ => $batchArrIds);
          $params = json_encode($paramsArr);
          $result = null;
          exec(“php -f {$root}shell/batch_import_processor.php ‘{$params}’”, $result);
          $saved += $result[0];
          echo convert(memory_get_usage()) . ” - processed {$recordCount}/$totalproducts. Saved {$result[0]}\n”;
          $batchArrIds = array();
      }
    }
    echo convert(memory_get_usage()) . ” - Saved {$saved} of {$totalproducts} rows\n”;

    //emulate batchFinish action
    echo convert(memory_get_usage()) . ” - Emulate batchFinish action\n”;
    $batchModel = Mage::getModel(‘dataflow/batch’)->load($batchId);
    try {
      $batchModel->beforeFinish();
    } catch (Mage_Core_Exception $e) {
      echo convert(memory_get_usage()) . ” - ERROR: “. $e->getMessage() .”\n”;
    } catch (Exception $e) {
      echo convert(memory_get_usage()) . ” - ERROR: An error occurred while finishing process. Please refresh the cache”. $e->getMessage() .”\n”;
    }
    $batchModel->delete();

    //dump usefull info
    foreach ($profile->getExceptions() as $e) {
      //echo convert(memory_get_usage()) . ” - ” . $e->getMessage() .”\n”;
    }

    echo convert(memory_get_usage()) . ” - ” . “Completed!\n”;
    ============== batch_import_processor.php
    <?php
    $root = ‘../www/’;

    set_time_limit(0);
    ini_set(‘memory_limit’, ‘128M’);

    //getting Magento
    require_once $root.‘app/Mage.php’;
    ob_implicit_flush();
    Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

    //echo Mage::app()->getLocale()->getLocaleCode();
    //exit;
    //$store = Mage::app()->getStore();
    //$store->setLocaleCode(‘ru_RU’);

    $params = $argv[1];
    $paramsArr = json_decode($params, true);
    $batchId = $paramsArr[‘batchid’];
    $importIds = $paramsArr[‘ids’];

    //echo “2| {$argv[1]}\n”; exit;
    //print_r($argv);

    $saved = 0;
    $batchModel = Mage::getModel(‘dataflow/batch’)->load($batchId);
    $batchImportModel = $batchModel->getBatchImportModel();
    $adapter = Mage::getModel($batchModel->getAdapter());
    $adapter->setBatchParams($batchModel->getParams());

    foreach ($importIds as $importId) { 
      $batchImportModel->load($importId);
      if (!$batchImportModel->getId()) {
          //echo convert(memory_get_usage()) . ” - Skip undefined row {$importId}\n”;
          continue;
      }
     
      //have to recreate every time to
      try {
          $importData = $batchImportModel->getBatchData();
          $adapter->saveRow($importData);
      } catch (Exception $e) {
          continue;
      }
      $saved ++;
    }

    if (method_exists($adapter, ‘getEventPrefix’)) {
      //Event for process rules relations after products import
      Mage::dispatchEvent($adapter->getEventPrefix() . ‘_finish_before’, array(
          ‘adapter’ => $adapter
      ));

      //Clear affected ids for adapter possible reuse
      $adapter->clearAffectedEntityIds();
    }

    echo $saved;

     

  • UcheAni-Ok

    (dec 02)
    UcheAni-Ok">Reply to this comment

    Does the script still work?

  • Reageren

    Contact

    Bel mij terug

    Vul uw telefoonnummer in en we bellen u zo spoedig mogelijk terug.

    Contactgegevens

    H&O
    Veenderveld 20
    2371 TV
    Roelofarendsveen
    info@h-o.nl
    www.h-o.nl
    071 744 0084
    KvK Rijnland nr. 28119089
    BTW nr. NL818554071B01

    Nieuwe projecten

    Wilt u weten wat H&O kan betekenen voor uw Magento project? Stuur ons een mailtje:

    Paul Hachmang

    p.hachmang@h-o.nl

    Erwin Otten

    e.otten@h-o.nl

    Contactformulier