Add proxy to Python crawler

Cancelado Publicado Sep 24, 2012 Pagado a la entrega
Cancelado Pagado a la entrega

The requirement is to add a proxy capability to an existing crawler based on beautiful soup, urllib2 and sqlite.

The proxies can rotate based on a per-thread basis or whatever is the most efficient and effective way in terms of crawling success and not needing a lot of proxies to make the crawling work.

The added code needs to be:

* Handling download failures and retrying specified amount of times with different proxies.

* Removing failing proxies after a specified number of consecutive failures.

* Adding log messages for the above.

**IMPORTANT:**

Please quote the full name of the inventor of Python for your bid to be considered.

Testing of the code by the worker to be 100% up-to-spec is a requirement of this project.

The 3 days deadline can be increased on the request of the worker IF there is sufficient progress observed during the first 3 days (and each successive extension) of the project.

PHP

Nº del proyecto: #2779403

Sobre el proyecto

2 propuestas Proyecto remoto Activo Oct 17, 2012

2 freelancers están ofertando un promedio de $32 por este trabajo

xhcvw

See private message.

$30.6 USD en 3 días
(0 comentarios)
0.0
AlexAltea

See private message.

$33.15 USD en 3 días
(0 comentarios)
0.0