tag:blogger.com,1999:blog-2523819181563716059.post4947263372591097903..comments2023-11-02T06:14:07.280-07:00Comments on Vancouver Data Blog by Neil McGuigan: Web Crawling with RapidMinerNeil McGuiganhttp://www.blogger.com/profile/14122981831780837323noreply@blogger.comBlogger22125tag:blogger.com,1999:blog-2523819181563716059.post-55846692357804076622013-12-18T13:00:50.983-08:002013-12-18T13:00:50.983-08:00Excellent video,
I tried with my RapidMiner V5.3....Excellent video,<br /><br />I tried with my RapidMiner V5.3.015 and web Mining 5.3.1 which is checked.<br />But I can not find web mining extension in operators view as well as crawl web operator to use it.<br />Any ideas?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-10028479762475974782013-06-20T06:34:39.681-07:002013-06-20T06:34:39.681-07:00No filename given for result file, using stdout fo...No filename given for result file, using stdout for logging results!<br />Process //NewLocalRepository/crawl_web_pages starts<br />PM INFO: Loading initial data.<br />Saving results.<br />Process //NewLocalRepository/crawl_web_pages finished successfully after 11 s<br /><br />after reunnig the Crawl Web operator am getting above specified lines in Log and example set is null(i.e only column Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-26955946313936571682013-06-19T22:56:46.168-07:002013-06-19T22:56:46.168-07:00Hello,
@ Neil sir,
I have followed all the steps ...Hello, <br />@ Neil sir,<br />I have followed all the steps of Your video as it is, but still am not able get the list of Webpages as output.In the result mode i only headings :"Row No. Link Document_source ".<br /><br />please help me out.Sumithranoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-15261632938840338732013-03-24T23:30:39.018-07:002013-03-24T23:30:39.018-07:00Hi, i completed my UG in IT last and interested in...Hi, i completed my UG in IT last and interested in web designing and seo facts. I am just a newbie to this term web crawling and rapid miner web. I read and gathered many basic info bout this from <a href="http://webspiders.biz/" rel="nofollow">WebSpiders.biz</a>. They just presented a good facts about web designing and crawling. Check out.....JackWilliamhttp://webspiders.biz/noreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-91786228212704517192012-03-17T13:21:26.507-07:002012-03-17T13:21:26.507-07:00I have this below XML wherein i wanted to pick up ...I have this below XML wherein i wanted to pick up the anchor tag "VANCOUVER Airport, Canada" instead of the URL. Tried using the following in RapidMiner<br /><br />//h:*[contains(.,'Departure airport:')]/../h:td[2]/a and also <br />//h:*[contains(.,'Departure airport:')]/../h:td[2]/a/text() <br />but no luck...can you suggest something. <br />----------------------------Manojhttps://www.blogger.com/profile/00069664948746333405noreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-77388485092429753822012-03-17T13:10:38.654-07:002012-03-17T13:10:38.654-07:00This comment has been removed by the author.Manojhttps://www.blogger.com/profile/00069664948746333405noreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-73175279884452908362012-02-21T09:58:37.102-08:002012-02-21T09:58:37.102-08:00@Thomas, I am not sure exactly, though that page i...@Thomas, I am not sure exactly, though that page is quite large, and slow to open. Can you try upping the max size and timeout?<br /><br />You could also try my ajax scraper (latest post as of today), even though it doesn't use ajax.Neil McGuiganhttps://www.blogger.com/profile/14122981831780837323noreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-88012774586390730422012-02-19T16:30:08.938-08:002012-02-19T16:30:08.938-08:00i also can't seem to get certain sites to work...i also can't seem to get certain sites to work. for example:<br />http://www.immobilienscout24.de/Suche/S-T/P-2/Wohnung-Miete/Hamburg/Hamburg/-/-/-/EURO--1000,00<br /><br />doesn't use cookies, doesn't seem to use ajax (maybe i am wrong about it?), i set the user agent properly, i uncheck "obey robot exclusion" and check "really ignore exclusion".<br />anyone knowsThomasnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-51601607131619290752012-02-07T14:37:31.260-08:002012-02-07T14:37:31.260-08:00Hi Neil,
Any update on crawling AJAX websites? B...Hi Neil, <br /><br />Any update on crawling AJAX websites? Best solution I have come up with is just saving the search pages I am interested manually as "completed webpages" using Mozilla/Chrome and going from there. I can display up to 500 search results at once for the particular website I am crawling but I can see this being a hastle if its capped at 50 results a page. I think the Yaroslavhttps://www.blogger.com/profile/10729919232900949911noreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-22152686129732495222012-02-06T17:09:54.545-08:002012-02-06T17:09:54.545-08:00Hi Neil,
Any update on issue #1(crawling ajax bas...Hi Neil,<br /><br />Any update on issue #1(crawling ajax based websites?) It seems the website I am interseted in crawling (www.uship.com/find) is ajax based. <br /><br />I've noticed I can save the "post-ajax" webpage using chrome/mozilla "web complete". Luckily I can save up to 500 search results at a time. This way I think I can get the hyperlinks for each shipment Yaronoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-9881755993948149512011-09-28T02:55:14.519-07:002011-09-28T02:55:14.519-07:00Neil, this is really useful. Thanks!
Is there anyw...Neil, this is really useful. Thanks!<br />Is there anyway to save the page using it's 'Title' tag rather than 0,1,2,3,4....Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-62979027037437547822011-09-25T07:11:54.097-07:002011-09-25T07:11:54.097-07:00To the last commenter: Why would they tell you how...To the last commenter: Why would they tell you how to scrape their own website in their own documentation?<br /><br />https://docs.google.com/support/bin/answer.py?answer=155184Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-85757351033016425512011-09-07T15:40:40.587-07:002011-09-07T15:40:40.587-07:00I think Google disallow crawling their website. yo...I think Google disallow crawling their website. you can see that here http://www.google.com/robots.txtAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-51747139810290531342011-06-27T23:02:35.490-07:002011-06-27T23:02:35.490-07:00how to use this to crawl google??? with other webs...how to use this to crawl google??? with other websites rapidminer so powerfull, but it can't handle google. Could you show me how ???Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-31793857858938962011-06-17T21:28:38.987-07:002011-06-17T21:28:38.987-07:00This is incredibly informative. Thank you so much!...This is incredibly informative. Thank you so much!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-74025084741299242382011-04-27T17:09:13.603-07:002011-04-27T17:09:13.603-07:00The burkeware link is awesome! Makes it so much ea...The burkeware link is awesome! Makes it so much easier to get regular expressions going. Thanks so much for sharing!Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-74477090402451534682011-04-26T09:46:19.034-07:002011-04-26T09:46:19.034-07:00@anonymous: regular expressions are powerful, but ...@anonymous: regular expressions are powerful, but can definitely be tricky. I would recommend checking out this to learn by example:<br /><br />http://www.regular-expressions.info/examples.html<br /><br />and using this to learn by doing:<br /><br />http://burkeware.com/software/regex_playground.html<br /><br />good luckNeil McGuiganhttps://www.blogger.com/profile/14122981831780837323noreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-55911811129636449712011-04-26T04:52:42.865-07:002011-04-26T04:52:42.865-07:00Can you give us some suggestions for common expres...Can you give us some suggestions for common expressions?<br /><br />I noticed that with .+key.+ I only get sites wich are http:/www.meh.com/mehh/key.html or http:/www.meh.com/mehh/mehh/key.html but not http:/www.meh.com/mehh/key/site/mehhh/index.html<br /><br />Any ideas for how to use common expressions on urls? It seems quite tricky for someone new to this.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-77562386493257377092011-04-11T17:21:08.569-07:002011-04-11T17:21:08.569-07:001. if a website uses AJAX, you have to crawl it di...1. if a website uses AJAX, you have to crawl it differently. <br /><br />2. these were my crawling rules:<br /><br />store_with_matching_url .+suiteid.+<br />follow_link_with_matching_url .+pagenum.+|.+suiteid.+<br /><br />3. regarding the european sites, they may check the user agent, so you could try changing that to your browser's, or they may check a cookie.<br /><br />A later video will Neil McGuiganhttps://www.blogger.com/profile/14122981831780837323noreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-62668491784413566412011-04-11T14:16:27.578-07:002011-04-11T14:16:27.578-07:00Can you tell me what you wrote in the crawling rul...Can you tell me what you wrote in the crawling rules.I can´t see clearly.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-19761230169506590412011-04-05T23:28:05.456-07:002011-04-05T23:28:05.456-07:00Please can you tell me why I can't crawl other...Please can you tell me why I can't crawl other websites e.g (yahoo, Google..etc.)? I followed your video and it's just worked perfect on the the site that you used but if I did the same thing with another sites it dose not work. can you please help me? <br />Thank you so much for the valuable information.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-2523819181563716059.post-66211058810607214342011-04-05T08:49:21.019-07:002011-04-05T08:49:21.019-07:00Nice!Nice!Anonymoushttps://www.blogger.com/profile/06090266698386776628noreply@blogger.com