Free Web Site Data Extractor / Miner / Harvester Software - Need to Find?

Question

Reason: I want to find a good automatic web-data-mining software. Simply put, it’s a piece of software that can explore a specific web site and extract specific data (that I specify) from a site and save in some useful format on my computer. Specifically speaking, I want to use this software to automatically harvest data from Flat (Apartment) Letting web sites like ‘RightMove’. The ultimate result I’d like from this software is that I configure it to work with whichever Letting Agencies' web site, press 1 button, and it automatically harvests all data I’ve specified, and outputs it into a spreadsheet format.

Requirements:

1) It needs to be free. Computers have been around way too long with the mostly-sole purpose of repeating instructions at phenomenal speeds and sifting / sorting data… I don’t really believe that it should still cost so very much to purchase software of this type when performing calculations and sorts is practically what computers were invented to do.

2) No ‘Trial Versions’ unless they last indefinitely and have no real set-backs in capability. In other words, I don’t need to know about a really good software but which is only free on its trial version (in which it only lasts 14 days and caps the amount of data you can harvest).


3) The software must be ‘automatic’. What I mean is simply after I specify the data I want to extract from a site, I can set it to ‘automatically harvest data’ in the way I’ve told it to and export that data to whatever file I want. I don’t want to have to manually tell it, each time, how to extract the data. I realize if the web page in question changes it’s layout or format that I would have to re-create the extraction process. I don’t mind this.

4) It needs to be somewhat easy-to-use (hopefully visual in nature). I can handle small tinkering / ‘mod’ing’ (modifying) of simple languages (Javascript, etc)… however my programming knowledge is, sadly, not vast enough to write large bits of code myself or do anything more than light to medium ‘modifying’ of code to fit my purposes.


Software I’m already aware of:

Mozenda – This, to my knowledge, is free only for a 14 day Trial, otherwise it’s pay to purchase. For these reasons I am not interested in it (unless there is, in fact, a “Lite” version which has no expiration date).

‘Data Handling Software’ I currently have:

I have Excel, Office, and Access 2003. Other than these do not have any software capable of parsing through data gathered by a web-harvester / extractor / miner. I am completely willing to download other programs which can deal well with the data extracted by any web mining software you guys recommend to me. Again, however, they need to be free and relatively easy-to-use.

Side-notes:

I am aware Excel has methods by which you can Import Web Queries directly into the sheet. This, however, is not automatic (from what I know), and requires a lot of time and manual handling.

Thank you all, in advance, for any assistance. I’ve spent a lot of time and effort trying to locate software of this nature, and so far come up short.

Answer 1

You were on the right track. Excel does indeed have the ability to do web queries, and you can automate Excel using the free built-in Visual Basic for Applications (VBA) programming language. With VBA, if you can think it, you can probably do it because it really is a complete programming language.



Having a web query run at the touch of a button or on a predetermined schedule is quite do-able with VBA. So is formatting, saving files, and all the other operations that you mentioned. The catch is that because you are seeking a unique software solution, you're going to have to learn how to program in VBA to make it happen. That will take some effort, but there is plenty of free VBA sample code to get you started.



Given that your copy of Office is old, you should be able to pick up books on Excel VBA 2003 for free or just pennies.



Here's a link that explains how to programatically make a web query in just about all versions of Excel:

http://support.microsoft.com/default.aspx?scid=kb%3Ben-us%3B187364

Answer 2

Web Data Extraction Software

Answer 3

Web Content Extractor is the most powerful and easy-to-use data extraction software for web scraping and data extraction from the websites.Web data extraction is completely automatic. I just give a link for best Web Content Extractor. See below

Answer 4

Really there isnt, if you want a good host that gives you some awesome software when you sign up, use Lunarpages, they give you about $300 dollars worth of the web design software called Coffee cup, I use this myself along with another program called Antenna Wed Studio, its $60 but well worth it since its pretty much the simplest way to build webpages using WYSIWYG to build them. But if you are setting up your own site then look into Lunarpages, it rocks.

Answer 5

Here is a good one http://webminer.avantprime.com