Hi there,
It can be done and we can make it an automated process where through a cron , on a particular time gap,we can update the data base of your big commerce site and it will get updated accordingly.
we have done this earlier for several betting sites and it worked well for 1-2 months ,then it started crashing day by day only for one limitations on this process:--
Such Scraping are being done on two major parameters:-
1. url of the site
2. HTML structure of the pages.
if any of them started getting changed ,then the cron will not effect as the scrapper will not work properly. Are you aware of this fact?
It basically happens when the source website owner get to knows their site being scrapped,they started changing above changes very frequently to get rid of the scrapping.
We can actually get rid of this issues as well in our solution if we can keep updating and changing whenever required depending on the source.
I wanted to be honest with you from the beginning for that reason shared a true lacking in scrapping automation solution.
Please let me know the target websites and the data types and also share your website reference too.
--
Regards,
Sumantra