Automatic import with Magento using SSH, no browser needed

Automatic import with Magento using SSH, no browser needed

For one of our Magento webshops, one of the requirements was that the product feed was imported automatically overnight. This ment that I had to find a way to automate this process, but since Magento doesn't support a non-browser productimport this was easier said then done. After looking for a while, I found the following topic on the magento forums. This looked good, this meant I could use the regular Import/Export profiles and run them in the shell. After investigating a bit further, I discovered I could also use Advanced Import/Export profiles - what was just what I needed. So I set up my Advanced profile (one for importing products, one for importing product relations) and did my first first testrun. Everything went well, but after about 1000 - 2000 products It suddenly crashed: PHP Allowed Memory Size Exchausted Fatal Error. That is just great ...

Please note: This functionality was created for Magento 1.4.0.1 and is not tested for Magento 1.5+ and therefor may not work as expected. This script has been replaced with a new script (tested with 1.6.2 and 1.7.0), which can be found on H&O's GitHub Page.

Importing a large quantity of products via the shell.

I wondered what happened. After logging my memory usage a bit more, I noticed that after every product that was saved, the memory usage got a little bit higher every time until it eventually crashed. How could this happen, and why does the regular import doesn't have this problem? It seems that Magento doesn't free the used memory AFTER saving a product. So after a 1000-2000 product saves, the system is out of memory. One other thing I noticed, is that the import also gets much slower after imported a few hundred products.

Why does the browser import doesn't have this problem? Why doesn't it crash and why doesn't it get slower? That is because the browser import does something different:

When the importer loads (when the user has pressed run) it parses the complete source file (csv, xml) and places it in a temporary database table (dataflow_batch_import).
From that table it creates a list of all the rows that need to be imported and it sends that list to the browser.
The browser creates a Javascript Ajax request back to magento for every batch (the number you configured during import).
When Magento receives that request, magento starts, imports the batch, sends the results back to the page and closes.
When the page receives the results back, it prints them, and starts over with step three.

This process solves the problem that Magento will run out of memory and the import getting slower. If we were to call the importpage from the shell in some way, this solution wouldn't work, since the page depends on Javascript to place it's Ajax requests. What we need to do, is recreate this step within the shell. Since the shell can't execute Javascript we'll have to recreate this in PHP (in the shell, PHP is allowed to run for an infinite amount of time). In PHP we create cURL requests instead of Ajax requests. We fully simulate the browser-import-situation.

The code consists of two files. mag_login.php and mag_import.php. Both are needed. Place them in /PATH/TO/YOUR/MAGENTOINSTALLATION/shell/

mag_import.php:

mag_login.php:

Copy the above code to your local Magento installation, put them in a secure folder, that isn't accesible by a browser and login to the shell. Fill in the correct information into the files. After that is done, start up your shell, browse to the folder. Run the file with:

In the line above, 7 should be replaced by your own profile Id, which can be found the first column of the profiles.

Making the Magento product import fully automatic.

If we got the above working, we need to get the working fully automatic. We want to import our products and do a complete reindex of magento every night. First we create a file that can be executed by the shell that imports the products and reindexes it. Call it something like import.sh (sh from shell):

Try and run the file with the following:

If that all works properly, we can setup our cronjob to do it automatically every night. Enter the following into you shell:

If you already have setup your Magento cron, you should see something like:

Now, if something happens to our import, somethings goes wrong, etc, we wan't to get notified. Add the following line at the top of the crontab file.

Below the Magento line we add our import cron:

Now save the crontab (depends on the editor how) and your all set.

Note: if you want to see error regarding the import, lines that could not be processed open /PATH/TO/YOUR/MAGENTOINSTALLATION/var/log/import.log or browse to the file in the shell and type:

H&O is expert in het oplossen van complexe Magento vraagstukken, maatwerk grafisch ontwerp en het bouwen van custom Magento modules.

Als je op zoek bent naar een partner die je kan helpen met het beantwoorden van moeilijke Magento vragen, een maatwerk productimport voor je ontwikkeld of een custom Magento module voor je schrijft die jouw assortiment met een andere webshop koppelt, dan ben je aan het juiste adres. H&O heeft zich sinds 2007 verdiept in de techniek en ontwikkelingen van het e-commerce CMS, speelt een actieve rol in de community, heeft verschillende modules uitgebracht en heeft voor klanten uiteenlopende technische modules ontwikkeld. Ons kantoor is gevestigd in Roelofarendsveen, direct naast de A4 én slechts op 15 minuten afstand van Amsterdam. Neem contact met ons op en kom eens langs, dan bespreken we jouw vraagstuk!

Paul Hachmang

Paul Hachmang is a Back-end Developer from Roelofarendsveen (The Netherlands) and co-founder of H&O. His weapons of choice include PHP, MySQL, HTML, Javascript, ExpressionEngine, Magento and a good cup of coffee. Contact Paul Hachmang

Comments

Alta

(okt 05)
Reply to this comment

What version Magento?
Paul Hachmang

(okt 07)
Reply to this comment

This script is tested on Magento 1.4.0.1, but I think it will also work on 1.4.1.x and 1.3.3.0.
David O B

(okt 07)
Reply to this comment

Hi there,

I followed and tested your script in in magento 1.4.1 however the products
don’t import and the import.log file show 403 permision dined errors.

Any idea what might be causing this.

This is an excellent feature and would really like to get in working.

Log File Output 2010-10-07T14:35:48+00:00 DEBUG (7): 2010-10-07T14:35:48+00:00 DEBUG (7): 7.55 mb - STARTING IMPORT 2010-10-07T14:35:48+00:00 DEBUG (7): 9.03 mb - Table dataflow_batch_import cleaned 2010-10-07T14:35:48+00:00 DEBUG (7): 9.03 mb - Preparing profile... 2010-10-07T14:36:18+00:00 DEBUG (7): 12 mb - ...Done 2010-10-07T14:36:18+00:00 DEBUG (7): 12 mb - Loaded batch id 208 2010-10-07T14:36:18+00:00 DEBUG (7): 17.12 mb - 0/20428 2010-10-07T14:36:18+00:00 DEBUG (7): 17.13 mb - 50/20428 [<?xml version="1.0" encoding="ISO-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html > <head> <title>Access forbidden!</title> <link rev="made" href="mailto:webmaster@localhost" /> <style type="text/css"><![CDATA[/*></style> </head> <body> <h1>Access forbidden!</h1> <p> You don't have permission to access the requested directory. There is either no index document or the directory is read-protected. </p> <p> If you think this is a server error, please contact the <a href="mailto:webmaster@localhost">webmaster</a>. </p> <h2>Error 403</h2> <address> <a href="/">dev.mydomain.com</a><br /> <span>Thu Oct 7 15:36:18 2010<br />

and

2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb - Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load 2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb - Loaded successfully: "/var/www/mydomain/public/httpdocs/var/import/my_products.csv". 2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb - Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse 2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb - Found 20428 rows. 2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb - Starting catalog/convert_adapter_product :: parse 2010-10-07T14:43:18+00:00 DEBUG (7): 17.12 mb - Completed! 2010-10-07T15:04:03+00:00 DEBUG (7):
David O B

(okt 08)
Reply to this comment

When I run mag_login.php with “php mag_login.php” it just runs no error on screen, no inputs to import.log file
David O B

(okt 08)
Reply to this comment

If I run it from the browser I get the message: The page isn’t redirecting properly
Paul Hachmang

(okt 09)
Reply to this comment

Hi David,

It seems that the login isn’t functioning properly. I can reproduce it here as well.
MagentoKnight

(okt 21)
Reply to this comment

Same here. Can anyone get this to work?
Idealclic

(nov 10)
Reply to this comment

I can’t make it work !
Can you explain the difference between
Path to the root of your magento installation
and
Url to your magento installation ?

Also, it is said to put mag_login.php in shell and shell in the root, so is dataflow/shell/mag_login.php, the correct path for “relative path from the magento root to the login file.” ?

Many thanks.
Paul Hachmang

(nov 11)
Reply to this comment

Your URL is the webaddress you type in your browser, starts with http:// etc. Your Path is the location of your magento installation on the harddisk on your server. If you don’t know what it is, create a php file and put <?php echo $_SERVER[‘PHP_SELF’]; ?> inside, and you’ll see your server path.

As for the relative path to the login file, i’ve changed it, the dataflow was something i used in my own application shouldn’t be there, changed it in the post.
Idealclic

(nov 11)
Reply to this comment

Hi Paul,

Thank you for your answer !

Now I’m getting this error in ssh when I run php -f mycron/mag_import.php 11
PHP Warning: Unknown(): Unable to load dynamic library ‘/usr/lib/php/extensions/no-debug-non-zts-20020429/mcrypt.so’ - /usr/lib/php/extensions/no-debug-non-zts-20020429/mcrypt.so: cannot open shared object file: No such file or directory in Unknown on line 0

Parse error: syntax error, unexpected T_OBJECT_OPERATOR in /homepages/6/d274382575/htdocs/magento-prod/mycron/mag_import.php on line 52

Any idea ?

Thank you

Tiron

(feb 27)
Reply to this comment

Hi
I got this issue with Godaddy shared hosting,this happen because of php version issue. Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID); this is compatible with php 5 or higher.
I fixed it by using
/web/cgi-bin/php5 -f path_to_the_php_file.php 8

Idealclic

(nov 11)
Reply to this comment

Well, I solved the first error by deleting the line
extension=mycript.so in php.ini
(not sure it is the best way to do it)

But I still have this error:

Parse error: syntax error, unexpected T_OBJECT_OPERATOR in /homepages/6/d274382575/htdocs/magento-prod/mycron/mag_import.php on line 52
Magento Knight

(nov 11)
Reply to this comment

Re: Paul on OCT/9, Has the login code been updated to work?
Eduard Fabra

(nov 25)
Reply to this comment

Hi,

I modified the login code, and now I can run it without problems. Here are my code:

<?php /** * Path to the root of your magento installation. * include traing slash. */ $root = 'your/magento/installation/path'; /** * Username that has the rights to import products. */ $username = 'your_user'; /** * Password */ $password = "your_password"; /** * DO NOT EDIT BELOW THIS LINE */ //getting Magento require_once $root.'app/Mage.php'; umask(0); Mage::app(); $user = Mage::getModel('admin/user'); if ($user->authenticate($username, $password)) { Mage::getSingleton('admin/session')->setUser($user); if ($user->getId()) { if (Mage::getSingleton('adminhtml/url')->useSecretKey()) { Mage::getSingleton('adminhtml/url')->renewSecretUrls(); } $session = Mage::getSingleton("admin/session"); //Change the owner of the session file on root/var/session/ to the user that runs the webserver exec("chown nobody:nobody {$root}var/session/sess_".$session->getEncryptedSessionId()); echo json_encode(array('sessionId' => $session->getEncryptedSessionId(), 'formKey' => Mage::getSingleton('core/session')->getFormKey())); } } ?>

I run the process in a cron and It works well with a Magento 1.4.1.1. The problem with the original mag_login.php was that the $session->login($username, $password), the Magento login function was calling a header with a redirect to the html login page, in this version I replaced it with a call to $user->authenticate($username, $password). Finally, I added a chown command to the session file to change the owner of this file because if not, the curl session, that runs with the user of the webserver, can’t not access to session data.

Hope it helps.

—
Eduard
Paul Hachmang

(nov 25)
Reply to this comment

Whoa, that is great Eduard!
Bojan

(dec 04)
Reply to this comment

Great script, but, even with the changed login I cant get it up and running.
I am getting the same error:
2010-12-04T18:22:57+00:00 DEBUG (7): 10,96 mb - 3150/3292 [{“ajaxExpired”:1,“ajaxRedirect”:“htt…

Any help would we apritiated.
Allard Muis

(dec 07)
Reply to this comment

I couldn’t get the login to work either (ajaxExpired).
So I changed the login procedure to work through cURL as well.
My version og mag_login.php:

<?php /** * Path to the root of your magento installation. * include traing slash. */ $root = '/home/domainname/public_html/orsomethinglikethis'; /** * Url to your magento installation. (include trailing /)*/ $url = 'http://www.domain.com/'; /** * Username that has the rights to import products. */ $username = 'XXX'; /** * Password */ $password = 'XXX'; /** * DO NOT EDIT BELOW THIS LINE */ $profileId = $argv[1]; $ch = curl_init(); curl_setopt($ch, CURLOPT_COOKIESESSION, 1 ); curl_setopt( $ch, CURLOPT_NOBODY, 1 ); curl_setopt($ch, CURLOPT_URL,$url."index.php/admin/"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HEADER, true); // header will be at output curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, 'login[password]='.$password.'&login;[username]='.$username ); $response = curl_exec($ch); $adminhtml = strpos( $response, 'adminhtml=' )+10; $semic = strpos( $response, ';', $adminhtml ); $cookie = substr( $response, $adminhtml, $semic-$adminhtml ); $ch = curl_init(); curl_setopt($ch, CURLOPT_COOKIESESSION, 1 ); curl_setopt($ch, CURLOPT_URL,$url."index.php/admin/system_convert_profile/run/id/".$profileId."/"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_HEADER, true); // header will be at output curl_setopt($ch, CURLOPT_COOKIE, "adminhtml=".$cookie ); $response = curl_exec($ch); $keypos = strpos( $response, 'var FORM_KEY = "' )+16; $quote = strpos( $response, '"', $keypos+5 ); $key = substr( $response, $keypos, $quote-$keypos ); echo json_encode(array('sessionId' => $cookie, 'formKey' => $key )); ?> Change a few lines in mag_import as well: Change this: /** * get het login information. */ exec("php -f {$root}{$login}", $result); $loginInformation = json_decode(substr($result[0],3)); into this: /** * get het login information. */ exec("php -f {$root}{$login} {$profileId}", $result); $loginInformation = json_decode(substr($result[0],3));

For this to work, you have to disable secret keys (System->config->(advanced)->admin->security->add secret keys to url-> NO

I’m sure a version with secret keys is possible too much work for my currect project.
Allard Muis

(dec 07)
Reply to this comment

please note a type in above code:
curl_setopt($ch, CURLOPT_POSTFIELDS, ‘login[password]=’.$password.’&login;[username]=’.$username );

The ; should be removed from login;[username]
Also note that copy-pasting from this form screws up quotes (’ and “) so you should fix these first.
Paul Hachmang

(dec 07)
Reply to this comment

It’s great to see people get this working. We currently have it running stable for 4 months now, no real problems encountered. This month i have another project that requires the cron-import as well and will use the suggested modifications. After that i will post an update that combines it into a nice package.
bojan

(dec 17)
Reply to this comment

Thanks for all the replies.

Even with all the above code, I still get ajaxExpired when I try to import. I will try to play around with the server security a bit and let you know if it worked :)
Rich

(jan 13)
Reply to this comment

Re this part of the suggested changes:

exec(“php -f {$root}{$login} {$profileId}”, $result);
$loginInformation = json_decode(substr($result[0],3));

Causes the:
[{“ajaxExpired”:1,“ajaxRedirect”:“httâ€¦
problem that Bojan has mentioned

It should be:
exec(“php -f {$root}{$login} {$profileId}”, $result);
$loginInformation = json_decode($result[0]);

That fixed it for me, hope it helps
Christian Schwan

(jan 18)
Reply to this comment

Less code and without login.php. Tested with 1.4.2 and Advanced Profiles.

<?php
$root = ‘/home/www/html/magento/’;
$logFileName= ‘import.log’;

/** * DO NOT EDIT BELOW THIS LINE */ //THIS SCRIPT JUST INITIALS THE PROFILE TO BE RUN VIA MAGENTO ADMIN "RUN PROFILE IN POPUP". Its the same thing as click just via this file that you can run via cron $profileId = $argv[1]; if (! isset($profileId)) { exit ("\nPlease specify a profile id. You can find it in the admin panel->Import/Export->Profiles.\nUsage: \n\t\t php -f $argv[0] PROFILE_ID\n\t example: php -f $argv[0] 7\n"); } function convert ($size) { $unit=array('b','kb','mb','gb','tb','pb'); return @round($size/pow(1024,($i=floor(log($size,1024)))),2).' '.$unit[$i]; } set_time_limit(0); ini_set('memory_limit', '128M'); require_once $root.'app/Mage.php'; umask(0); Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID); $profile = Mage::getModel('dataflow/profile'); $userModel = Mage::getModel('admin/user'); $userModel->setUserId(0); Mage::getSingleton('admin/session')->setUser($userModel); $profile->load($profileId); if (!$profile->getId()) { Mage::getSingleton('adminhtml/session')->addError('ERROR: Incorrect profile id'); } Mage::register('current_convert_profile', $profile); //starting the import Mage::log("\n\n", null, $logFileName); Mage::log(convert(memory_get_usage()) . " - " . "STARTING IMPORT", null, $logFileName); Mage::log(convert(memory_get_usage()) . " - " . "Preparing profile...", null, $logFileName); $profile->run(); Mage::log(convert(memory_get_usage()) . " - " . "...Done", null, $logFileName); $recordCount = 0; $batchModel = Mage::getSingleton('dataflow/batch'); Mage::log(convert(memory_get_usage()) . " - " . "Completed!", null, $logFileName); echo "EXPORT COMPLETE. BATCHID: " . $batchModel->getId();
Drew

(mar 22)
Reply to this comment

Getting an ajaxExpired error when you run from cron and don’t know why? Check your path variables!

This code will fail:
exec(“php -f {$root}{$login} {$profileId}”, $result);

This code works on my system:
exec(“/usr/local/bin/php -f {$root}{$login} {$profileId}”, $result);

Erwin Otten

(mar 22)
Reply to this comment

Thanks Drew, it is indeed important to double check your paths!

Markus Fuchs

(apr 04)
Reply to this comment

this is the content of my logfile:
2011-04-04T21:43:36+00:00 DEBUG (7):

2011-04-04T21:43:36+00:00 DEBUG (7): 8.68 mb - STARTING IMPORT
2011-04-04T21:43:37+00:00 DEBUG (7): 10.84 mb - Table dataflow_batch_import cleaned
2011-04-04T21:43:37+00:00 DEBUG (7): 10.84 mb - Preparing profile…
2011-04-04T21:43:37+00:00 DEBUG (7): 13,88 mb - ...Done
2011-04-04T21:43:37+00:00 DEBUG (7): 13,88 mb - Loaded batch id 103
2011-04-04T21:43:37+00:00 DEBUG (7): 13,92 mb - 0/1
2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - 1/1 [{“savedRows”:1,“errors”:[]}]
2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load
2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Loaded successfully: “/home/hammermu/public_html/var/import/import_produktpreise_dv_ev2.csv”.
2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Found 1 rows.
2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Starting catalog/convert_adapter_product :: parse
2011-04-04T21:43:40+00:00 DEBUG (7): 13,92 mb - Completed!

but the product still is not updated.
what am i doing wrong?
Inz

(mei 19)
Reply to this comment

This is so not working at all….. Could someone clarify this thread…

Paul Hachmang

(mei 25)
Reply to this comment

Could you please clarify what isn’t working?

Djé

(jun 09)
Reply to this comment

Hy,

No log files created, and no imports so… Do you have an idéa?
Josh

(aug 03)
Reply to this comment

I couldn’t get this working at all on 1.5.1

Doesn’t create a log file either. Very annoying.
Nick

(aug 12)
Reply to this comment

Tried the script and got this error in the shell:

chown: cannot access `/var/www/vhosts/outlet.lenovo.com/var/session/sess_bauni8sfv36ra2k5mujm07mop1': No such file or directory PHP Fatal error: Uncaught exception 'Exception' with message 'Notice: Trying to get property of non-object in /var/www/vhosts/outlet.lenovo.com/shell/mag_import.php on line 302' in /var/www/vhosts/outlet.lenovo.com/app/code/core/Mage/Core/functions.php:245

Here’s my log file:
2011-08-12T15:38:59+00:00 DEBUG (7): 2011-08-12T15:38:59+00:00 DEBUG (7): 6.59 mb - STARTING IMPORT 2011-08-12T15:39:00+00:00 DEBUG (7): 7.88 mb - Table dataflow_batch_import cleaned 2011-08-12T15:39:00+00:00 DEBUG (7): 7.88 mb - Preparing profile... 2011-08-12T15:39:00+00:00 DEBUG (7): 10.47 mb - ...Done 2011-08-12T15:39:00+00:00 DEBUG (7): 10.47 mb - Loaded batch id 143 2011-08-12T15:39:00+00:00 DEBUG (7): 10.49 mb - 0/2 2011-08-12T15:39:01+00:00 DEBUG (7): 10.56 mb - 2/2 [<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xml:lang="en" lang="en"> <head> <title>Lenovo Outlet - 404 Not Found 1 - Discounted Laptops, Cheap Laptops, Refurbished Laptops, Discounted Desktops, Cheap Desktops, Refurbished Desktops</title> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

etc with more html from 404 page.

What file is it trying to find that it can’t? I’m assuming thats why I’m getting this error.

Nick

(aug 12)
Reply to this comment

For clarification, Line 302 in the mag_import file is:

301: Mage::log(convert(memory_get_usage()) . " - {$recordCount}/{$totalproducts} [$buffer]" , null, $logFileName); 302: if (count($result->errors)) 303: {

Matt Ramage

(sep 10)
Reply to this comment

Has anyone solved the ajaxExpired error. Ive tried everything here with no luck. im using magento 1.5.1
Cao

(sep 15)
Reply to this comment

I got this error:

Fatal error: Uncaught exception 'Zend_Db_Adapter_Exception' with message 'pdo_mysql extension is not installed' in /home/developer/public_html/stoy/lib/Varien/Db/Adapter/Pdo/Mysql.php:289 Stack trace: #0 /home/developer/public_html/stoy/lib/Zend/Db/Adapter/Abstract.php(459): Varien_Db_Adapter_Pdo_Mysql->_connect() #1 /home/developer/public_html/stoy/lib/Zend/Db/Adapter/Pdo/Abstract.php(238): Zend_Db_Adapter_Abstract->query('SET NAMES utf8', Array) #2 /home/developer/public_html/stoy/lib/Varien/Db/Adapter/Pdo/Mysql.php(389): Zend_Db_Adapter_Pdo_Abstract->query('SET NAMES utf8', Array) #3 /home/developer/public_html/stoy/app/code/core/Mage/Core/Model/Resource.php(169): Varien_Db_Adapter_Pdo_Mysql->query('SET NAMES utf8') #4 /home/developer/public_html/stoy/app/code/core/Mage/Core/Model/Resource.php(110): Mage_Core_Model_Resource->_newConnection('pdo_mysql', Object(Mage_Core_Model_Config_Element)) #5 /home/developer/public_html/stoy/app/code/core/Mage/Core/Model/Resource/Db/Abstract.php(320): Mage_Core_Model_Resource->getCon in /home/developer/public_html/stoy/lib/Varien/Db/Adapter/Pdo/Mysql.php on line 289 But when i check system requirement : You have PHP 5.2.0 (or greater) Safe Mode is off You have MySQL 4.1.20 (or greater) You have the curl extension You have the dom extension You have the gd extension You have the hash extension You have the iconv extension You have the mcrypt extension You have the pcre extension You have the pdo extension You have the pdo_mysql extension You have the simplexml extension

I confused with this problem. Can someone help me . Thanks
Cao

(sep 15)
Reply to this comment

This function make the error:

Mage::app()->setCurrentStore(Mage_Core_Model_App::ADMIN_STORE_ID);

But i dont know how to fix it.
stephane

(sep 19)
Reply to this comment

Hello paul, thanks to you to share your work.
I install on a 1.5.1. No major error BUT…
I got theses logs :

2011-09-19T21:35:46+00:00 DEBUG (7): 8.35 mb - STARTING IMPORT 2011-09-19T21:35:46+00:00 DEBUG (7): 9.63 mb - Table dataflow_batch_import cleaned 2011-09-19T21:35:46+00:00 DEBUG (7): 9.63 mb - Preparing profile... 2011-09-19T21:35:46+00:00 DEBUG (7): 13,4 mb - ...Done 2011-09-19T21:35:46+00:00 DEBUG (7): 13,4 mb - Loaded batch id 9096 2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb - 0/5 2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb - 5/5 - Response is empty - ERROR 2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb - Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load 2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb - Loaded successfully: "/xxxx/xxxx/var/import/drop_elem.csv". 2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb - Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse 2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb - Found 5 rows. 2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb - Starting catalog/convert_adapter_product :: parse 2011-09-19T21:35:46+00:00 DEBUG (7): 13,44 mb - Completed!

so the $ch always empty. No update is done of course
Could you help me please ?
best regards
stephane
Gareth Shaw

(sep 25)
Reply to this comment

I get the same errors are Stephane above on v.1.3.2.4

2011-09-25T14:05:07+00:00 DEBUG (7): 5.74 mb - STARTING IMPORT
2011-09-25T14:05:07+00:00 DEBUG (7): 6.88 mb - Table dataflow_batch_import cleaned
2011-09-25T14:05:07+00:00 DEBUG (7): 6.88 mb - Preparing profile…
2011-09-25T14:05:40+00:00 DEBUG (7): 9.78 mb - ...Done
2011-09-25T14:05:40+00:00 DEBUG (7): 9.78 mb - Loaded batch id 7
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 0/3533
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #1
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 50/3533 - Response is empty - ERROR
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #2
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 100/3533 - Response is empty - ERROR
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #3
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 150/3533 - Response is empty - ERROR
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #4
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 200/3533 - Response is empty - ERROR
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #5
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 250/3533 - Response is empty - ERROR
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #6
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 300/3533 - Response is empty - ERROR
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #7
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 350/3533 - Response is empty - ERROR
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - Start cURL request #8
2011-09-25T14:05:40+00:00 DEBUG (7): 10.38 mb - 400/3533 - Response is empty - ERROR
.............

2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Start cURL request #71
2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - 3533/3533 - Response is empty - ERROR
2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Starting Mage_Dataflow_Model_Convert_Adapter_Io :: load
2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Loaded successfully: “/var/www/live/smilesextoys.co.uk/var/import/import_all_products.csv”
2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Starting Mage_Dataflow_Model_Convert_Parser_Csv :: parse
2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Found 3533 rows
2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Starting catalog/convert_adapter_productimport :: parse
2011-09-25T14:05:41+00:00 DEBUG (7): 10.38 mb - Completed!

This will be the 5th import script I’ve tried, none of which I can get to work.

Thanks
Gareth
Dave Pearson

(sep 27)
Reply to this comment

If your log file is not getting written, make sure you have logging enabled in:

System / Configuration / (Advanced) Developer / Log Settings / Enabled = Yes
Dave Pearson

(sep 27)
Reply to this comment

I’m getting the “ajaxExpired” error message on Magento 1.6. I have tried all suggestions in this thread but none are working.

Any ideas would be much appreciated?
Stuart Drennan

(okt 13)
Reply to this comment

Hi, I’m getting 550/550
[{"ajaxExpired":1,"ajaxRedirect":"https:\/\/************.com\/index.php\/admin\/index\/login\/"}]
on 1.4.2 even using the modified version from Christian Schwan. Thoughts anyone?? Pllllleeeeaassssee :(
Auto importeren

(jan 12)
Reply to this comment

Nice article!
DW

(feb 22)
Reply to this comment

Has anyone solved the ajaxExpired error? It’s driving me crazy.
Daniel Gafitescu

(mar 30)
Reply to this comment

Instead of

$root = ‘/PATH/TO/YOUR/MAGENTOINSTALLATION/’; you can do dirname(dirname(__FILE__));
website design

(apr 27)
Reply to this comment

This is working on Magento 1.5 too. Just you need to add a column in csv ‘has_options’ & set the value to ‘1’ for the products which are having custom options.
Magento UK

(mei 16)
Reply to this comment

is the script updated for 1.6? anyone tried?
Daniel

(jul 27)
Reply to this comment

This Script work perfect on Magento 1.7
Thank you very much for this wonderfull Tutorial!

Can i import Customers with this Script and in the shell file take Profile 6, the standart Profile from Magento?
Annemieke de rijp

(sep 10)
Reply to this comment

I have just tried the script on Magento 1.7 and it works very well without problems. I can import a large quantity of products via the shell. Easily. Thanks for the great script. Looking forward to see more useful magento scripts.

Annemieke
custom papers

(sep 26)
Reply to this comment

Thanks this post really opened my eyes. It is not only eye opening rather very beneficial for the people who wants to do somethin
best cover letter ever

(sep 26)
Reply to this comment

Wow, this actually cleared some things up. One essay i am having headache with is a footnote for an “who we are” The teacher says that we will have to stab below into ourselves and notice what we really are. But I think it would be more usefulto use
thesis writers

(okt 14)
Reply to this comment

I really enjoyed the quality information you offer to your visitors for this blog. I will bookmark your blog and have my friends check up here often.
spss in research methodology

(okt 16)
Reply to this comment

I am currently looking into a career change and this post helped to see another aspect of the industry. I appreciate your article
(nov 07)