Add proxy to Python crawler
$30-50 USD
Pagado a la entrega
The requirement is to add a proxy capability to an existing crawler based on beautiful soup, urllib2 and sqlite.
The proxies can rotate based on a per-thread basis or whatever is the most efficient and effective way in terms of crawling success and not needing a lot of proxies to make the crawling work.
The added code needs to be:
* Handling download failures and retrying specified amount of times with different proxies.
* Removing failing proxies after a specified number of consecutive failures.
* Adding log messages for the above.
**IMPORTANT:**
Please quote the full name of the inventor of Python for your bid to be considered.
Testing of the code by the worker to be 100% up-to-spec is a requirement of this project.
The 3 days deadline can be increased on the request of the worker IF there is sufficient progress observed during the first 3 days (and each successive extension) of the project.
Nº del proyecto: #2779403