ahmia / ahmia-crawler Goto Github PK

After fixing #9 this came up, seems like a PySocks bug: #todo report it.

Exception happened during processing of request from ('127.0.0.1', 48068)
Traceback (most recent call last):
  File "<path2venv>/ahmia-crawler/lib/python3.6/site-packages/socks.py", line 851, in connect
    negotiate(self, dest_addr, dest_port)
  File "<path2venv>/ahmia-crawler/lib/python3.6/site-packages/socks.py", line 497, in _negotiate_SOCKS5
    self, CONNECT, dest_addr)
  File "<path2venv>/ahmia-crawler/lib/python3.6/site-packages/socks.py", line 578, in _SOCKS5_request
    raise SOCKS5Error("{0:#04x}: {1}".format(status, error))
socks.SOCKS5Error: 0x06: TTL expired

Currently polipo remains the stable proxy solution for the crawler.

should be：
hash_target = hashlib.sha1(request.url.encode('utf-8')).hexdigest()

Anyone seen this before?

Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/internet/defer.py", line 653, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/local/lib/python2.7/dist-packages/scrapyelasticsearch/scrapyelasticsearch.py", line 116, in process_item
self.index_item(item)
File "/home/akent/Desktop/ahmia-crawler/ahmia/ahmia/pipelines.py", line 87, in index_item
self.send_items()
File "/usr/local/lib/python2.7/dist-packages/scrapyelasticsearch/scrapyelasticsearch.py", line 109, in send_items
helpers.bulk(self.es, self.items_buffer)
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/init.py", line 190, in bulk
for ok, item in streaming_bulk(client, actions, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/init.py", line 162, in streaming_bulk
for result in _process_bulk_chunk(client, bulk_actions, raise_on_exception, raise_on_error, **kwargs):
File "/usr/local/lib/python2.7/dist-packages/elasticsearch/helpers/init.py", line 134, in _process_bulk_chunk
raise BulkIndexError('%i document(s) failed to index.' % len(errors), errors)
BulkIndexError: (u'316 document(s) failed to index.', [{u'update': {u'status': 400, u'_type': u'tor', u'_id': u'd3ba376d34cbea850dabc25b1d4a884da852c995', u'error': {u'caused_by': {u'reason': u'scripts of type [inline], operation [update] and lang [groovy] are disabled', u'type': u'script_exception'}, u'reason': u'failed to execute script', u'type': u'illegal_argument_exception'}, u'_index': u'crawl'}}, {u'update': {u'status': 400, u'_type': u'tor', u'_id': u'9bbd33851fb71cc9980b262797a00c1a06932094', u'error': {u'caused_by': {u'reason': u'scripts of type [inline], operation [update] and lang [groovy] are disabled', u'type': u'script_exception'}, u'reason': u'failed to execute script', u'type': u'illegal_argument_exception'}, u'_index': u'crawl'}}, {u'update': {u'status': 400, u'_type': u'tor', u'_id': u'd730f4a540adad5892aa58a5870baed904b6c4c1', u'error': {u'caused_by': {u'reason': u'sc

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

ERROR:

ERROR: twisted 19.2.1 has requirement attrs>=17.4.0, but you'll have attrs 17.2.0 which is incompatible.

https://www.irif.fr/~jch/software/polipo/

Polipo will no longer be maintained. Are you going to move into some other alternative?
https://github.com/jech/polipo

Does this require tor to be installed? It's failing the initial run daily trying to retrieve and .onion site.

Would it be harrd to build a docker image so this all just works?

Please I need money very urgent to make money help me pls