Money-Code

Coding For Online Success

Parsing XML CJ Product Feeds

| 22 Comments

I mentioned on May 29th that CJ allows you to create and manage product catalogs now. This is great. You can create a product feed and have it available in your FTP directory (or whatever delivery method you wish). For my personal set up, I have it stored on their server, and I need to FTP to it and grab it. I do everything via PHP on a scheduled intervals.

I wanted to show you how you can grab the XML data and parse it. You could use it directly or push it into a database, etc. In my situation, I want to update a local database with product items. Items will be updated every week. I won’t go into the database piece (let me know if you need me to go into INSERT/DELETE SQL examples).

For this example, you will need PHP5 since we’ll be using the simplexml_load_file() function. This is assuming that you’ve already FTP’d the XML to your working directory. Again, let me know if you need this process as well, and I can provide some examples.

Once we have the XML file, we pass it to the simplexml_load_file function. This function quickly parses the XML into a object. Since it’s a object, it’s snap to reference the values. My example loops through the XML object and sets local variables with the values from the XML file. You could then INSERT this values into a database or display them in real time.

<?php
$xml                    = ’1111111_11111_20090528.xml’;
$resp                   = simplexml_load_file($xml);
foreach($resp->product as $item){
        $name                   = $item->name;
        $catalogname    = $item->catalogname;
        $lastupdated    = $item->lastupdated;
        $programurl             = $item->programurl;
        $keywords               = $item->keywords;
        $description    = $item->description;
        $sku                    = $item->sku;
        $manufacturer   = $item->manufacturer;
        $manufacturerid = $item->manufacturerid;
        $upc                    = $item->upc;
        $currency               = $item->currency;
        $saleprice              = $item->saleprice;
        $price                  = $item->price;
        $retailprice    = $item->retailprice;
        $buyurl                 = $item->buyurl;
        $impressionurl  = $item->impressionurl;
        $imageurl               = $item->imageurl;
        $advertisercategory     = $item->advertisercategory;
        $instock                = $item->instock;
        $condition              = $item->condition;
}
?>
 

My version of MySQL is 5, if you’re running 5.1 or 6, you can use a sweet MySQL command called “LOAD XML LOCAL INFILE”. For example, you could run this and it would populate your database table very quickly.

<?php
$sql = "LOAD XML LOCAL INFILE ’1111111_11111_20090528.xml’
                INTO TABLE tblProductsTest
                ROWS IDENTIFIED BY ‘ ‘"
;
if(!mysql_query($sql,$connect)){
        echo mysql_error()."
        "
.$sql;
}
?>
 

I think the ability to quickly make some product feeds is awesome. Now, I just need to find time to create some new store fronts and make some cash. If you find this post useful, please follow me on Twitter and Digg this post!

Incoming search terms:

  • cj xml feed
  • cj rss feed
  • cj product feed
  • parser products ebay
  • use cj data feeds to populate your wordpress blog

22 Comments

  1. Interesting… but I still confuse about 1111111_11111_20090528.xml what does it stands for?

  2. That is the XML file that you would have created at CJ. In my example, I set up a product feed from one of my merchants. They created the XML file on a weekly basis and put it in a FTP directory at CJ. I ‘fetch’ the XML file (named similar to what I have above) and then I parse it.

    HTH
    hanji

  3. Hi Hanji, thanks for sharing with us your code. Would it be possible to perhaps share with us the full php program, bascially, the unzipping of the file from CJ, the schema of the DB, and the SQL inserts / deletions that you are doing? Thanks a lot for your help.

  4. IcEDFiRE

    Thanks for the comment. I’ll do this, I won’t be able to do it today but will create a new blog post in the near future with the full application.

    Again, thanks for the comment.. hope to see you around!
    hanji

  5. Thanks for this hanji! I would also love to see the full application, but still this is a very nice tut! I would also like to discuss further on skype/phone if I have some time.

  6. Why do all of you xml experts try to make it as hard as you possibly can? Is it like an ego boost that you know a little tid bit that other people don’t. You would starve to death as a teacher. Crap is crap no matter how you package it. Don’t waste out time with geek crap.

  7. @POed

    Wow.. let me address this.

    A. Why do all of you xml experts try to make it as hard as you possibly can?
    This is 1 (one) foreach loop using simplexml_load_file which automagically parses the XML. Explain.. how I should make this easier? I even provide yet an easier option if your MySQL version 5.1 or 6 which doesn’t even use PHP.

    B. Is it like an ego boost that you know a little tid bit that other people don’t.
    No, I try to provide tools that I personally don’t use. Is it a ego boost for you to criticize on a blog about difficulty, yet don’t provide a easier solution?

    C. Don’t waste out time with geek crap.
    I will continue to post ‘geek crap’. I’m a geek. If you don’t like it.. go elsewhere.

  8. Hi
    Great Work, do you think you could parse this xml uncompressed product feed to work in Dreamweaver:

    http://feeds.perfb.com/index.php/download?OEMAIL=adityakishor@hotmail.com&PX=1c53a6624ce5b96bcd7bf7c7971ffcfa&DISPLAYFORMAT=XML&REVERSEMAPXML=yes&PRODUCTDB_ID=253

    If you can please help me out because i cant seem to figure out howto do it.

    just reply to me at adityakishor@hotmail.com

    Thanks

  9. Hello Aditya Gandhi

    I downloaded your XML file, and this should be easily parseable using PHP. You’re saying you’re using Dreamweaver which is a HTML editor. You can create and edit PHP with Dreamweaver, but you’ll need a server (either where your at or at a hosting facility) that can run and execute PHP code.

    If you have access to this, you can rewrite the parser code I have above and grab the individual XML nodes with the file (ie: product_name, product_code, etc). You can see those in the XML files. Once you know the node names you can parse through them.

    Optionally, if you didn’t want to mess with PHP, there may be some some XML parsing ‘services’ on the net that parse and send it back as javascript or html. I recently found one that does this with RSS feeds.

    Let me know if you need any additional help.

    Thanks!
    hanji

  10. Holey moley, you are a wealth of information!. I’m a fellow code/affiliate, but you’re definitely a couple years ahead of me. Your solutions make stuff that I’ve come up with look like amateur-city.

    I have a question for you. I use FeedWordpress to “drip” products from various RSS feeds into various blogs. It works quite well, except that good RSS product feeds are hard to come by. So, we turn to CJ and their XML product feeds. Great stuff, but FeedWordpress doesn’t parse XML feeds.

    Here’s what I’m thinking (let me know if you see holes in this or if there’s a better solution). I would create a script. When the script is called, it would connect to CJ via HTTP and pull the current XML feed for a merchant. The script would then parse the XML feed, and output the RSS that FeedWordPress likes.

    Is this a good plan? Can you point me toward some resources that would help me code this (including variations of any posts you’ve done)?

    Greatly appreciated.
    .-= Walt´s last blog ..FURminator Small Dog Brush =-.

  11. Hello Walt

    Thanks for commenting. I like your idea. I’ve been thinking about drip feeds for WordPress. If you’re interested, maybe we can collaborate? On another idea, I’m wondering if PopShops might be a easier solution. I can do some testing on my end.

    Thanks!
    hanji

  12. So is it better to display the results directly, or populate your database then show them like that?

  13. I personally like to store results in a database. I do this for a few reasons. In case there is a something wrong with a particular API call (ie: CJ is done, etc) it doesn’t break my site. Also, it’s much faster accessing a local DB vs making a API call every time. What I do is populate the local database on a set time.. ie: every hour, or once a day, etc.

    Hope this helps.
    hanji

    BTW.. thanks for commenting!

  14. Cool stuff, thanks for this. When we chatted on DP you mentioned that you didn’t think even large files would be a problem with this method – so you think doing a simplexml_load_file on an XML of, say 200Mb, with maybe 150,000 distinct items, wouldn’t place that much of a load on the server, or on memory?

    Mind you, the second method, with LOAD XML LOCAL INFILE, seems a lot neater, and I guess that would be very fast. I am not clear exactly how it works though – I guess it is creating a unique table entry for each ITEM, but what have you specified as ROWS IDENTIFIED BY? It just looks like some whitespace. I would have expected ROWS IDENTIFIED BY “”?

    Also, I am not clear what happens when you have nested XML – there is none in CJ XML, but what about Amazon, say? How does LOAD XML handle a hierarchy of data? It would be nice if it created related tables on its own initiative, obviously :)
    .-= markowe´s last blog ..The Zune goes HD – preorder now! =-.

  15. P.S. Oops, the tag I put after “I would have expected ROWS IDENTIFIED BY …” got stripped out – I meant to write a “product” closing tag.
    .-= markowe´s last blog ..The Zune goes HD – preorder now! =-.

  16. Would it be better to use xmlreader for a task like this? I’ve steered clear of using simplexml with cj feeds because these feeds can be very big and simplexml loads everything to memory — but maybe my understanding isn’t clear?

  17. When dealing with large files it’s better to use XMLreader.

    SimpleXML is not practical for big file since it loads files to memory before processing them (It’s a tree-based parser).

    XMLreader uses another approach (It’s a stream-based parser), so you should get the job done easily with XMLreader

    Read http://www.ibm.com/developerworks/xml/library/x-xmlphp2.html

  18. Currently I am running localhost server on my computer experimenting with my worpress blog and other site. I have downloaded all feeds to my c drive folder. When you say transfer it to a working directory? what does that mean? Are you saying creating a database on the server?

    This is the part of your comment I am trying to understand.
    “This is assuming that you’ve already FTP’d the XML to your working directory. Again, let me know if you need this process as well, and I can provide some examples.”

    By the way incredible post and thanks for the help.

  19. HI

    still confuse about 1111111_11111_20090528.xml . on the cj ftp I have product catalogs with 5 digit numbers and within each catalog there are dozens of advertisers of course for the life of me I cannot locate the above numbers anywhere. Whee do you get the above numbers relating to an advertisers xml feed on cj’s ftp?

  20. We have implemented a cj XML feed parser in ASP.NET. An example of which is at: http://eastgrandlake.com/fishing-shop/
    Depending on the merchant, the XML files tend to be large > 30 megs. We are about to release a MYSQL version to better work with the data and increase the performance.
    NexWeb´s last blog post ..Ecommerce Website ASPNET Shopping Cart System

  21. Acually, oops, we have just implemented the MySQL version at the same URL , which is much faster.
    NexWeb´s last blog post ..Ecommerce Website ASPNET Shopping Cart System

  22. can you suggest store script that can handle the cj datafeed?
    thanks

Leave a Reply

Required fields are marked *.


CommentLuv badge