What you want is a scraper, but that is filled with all kinds potholes with respect to how well the localized version of the site is going to run. Far too many to get into. I'm sure there are many types of freeware scrapers (there used to be).
I looked at realoem's robots.txt and they have a line which effectively says to crawlers don't crawl any URL containing a question mark, which is just about every URL in realoem, which means they are attempting to discourage scrapers.
I looked at realoem's robots.txt and they have a line which effectively says to crawlers don't crawl any URL containing a question mark, which is just about every URL in realoem, which means they are attempting to discourage scrapers.
Comment