Code Monkey home page Code Monkey logo

twint-search's Introduction

How to use

First use

For fisrt usage, you need to build image docker.

docker-compose build

Quick start

docker network create nw_twint
docker-compose up -d elasticsearch
sleep 10
docker-compose up -d searchapp
docker-compose run twint -u noneprivacy -es elasticsearch:9200 --json -o /opt/twint/noneprivacy.json
open http://localhost:3000

Twint, Elasticsearch & Searchapp

Start to up elaticsearch and searchapp

docker network create nw_twint
docker-compose up -d elasticsearch searchapp twint

Execute Twint command

docker network create nw_twint
docker-compose run -v $PWD/output:/srv/twint twint {{CMD TWINT}}

Examples of command

A few simple examples to help you understand the basics:

docker-compose run twint -u username -es elasticsearch:9200
docker-compose run twint -s "#osint" -es elasticsearch:9200
docker-compose run twint -u username -es elasticsearch:9200 --json -o /opt/twint/username.json
USERNAME=username docker-compose run twint -u ${USERNAME} -es elasticsearch:9200 --json -o /opt/twint/${USERNAME}.json

if local install of twint

twint -u username -es localhost:9200
twint -u username -es localhost:9200 --json -o /opt/twint/username.json

Search engine

  • Allows to do faceted search
  • Current build is developpment so you can do change and re-compile in real-time.
open http://localhost:3000

Debug Elasticsearch

open http://localhost:9000
open http://localhost:9200/twinttweets/_search?pretty=true&q=*:*
open http://localhost:9200/twinttweets/_count?pretty

Screenshots

alt text

alt text

Known Issues โš ๏ธ

I have noticed when running the new 5.0+ version on a linux host you need to increase the memory map areas with the following command

sudo sysctl -w vm.max_map_count=262144

More at https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

Issues

Find a bug? Want more features? Find something missing in the documentation? Let me know! Please don't hesitate to file an issue

To do

  • Embed video in tweets
  • Tweet with images
  • Tweet with location
  • Filter re-tweets
  • All the most crazy things possible

twint-search's People

Contributors

lucmski avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

twint-search's Issues

WARNING:fake_useragent:Error occurred during loading data.

Hi,

Hope you are all well !

I have the following error message from twint while fetching a username.

Command:

 lucasmd$ export USERNAME=lucmichalski; docker-compose run twint -u ${USERNAME} -es elasticsearch:9200 --json -o /opt/twint/${USERNAME}.json

Output:

[+] Indexing to Elasticsearch @ elasticsearch:9200
WARNING:fake_useragent:Error occurred during loading data. Trying to use cache server https://fake-useragent.herokuapp.com/browsers/0.1.11
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/fake_useragent/utils.py", line 67, in get
    context=context,
  File "/usr/local/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/urllib/request.py", line 1346, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/usr/local/lib/python3.6/urllib/request.py", line 1321, in do_open
    r = h.getresponse()
  File "/usr/local/lib/python3.6/http/client.py", line 1346, in getresponse
    response.begin()
  File "/usr/local/lib/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.6/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/fake_useragent/utils.py", line 166, in load
    verify_ssl=verify_ssl,
  File "/usr/local/lib/python3.6/site-packages/fake_useragent/utils.py", line 122, in get_browser_versions
    verify_ssl=verify_ssl,
  File "/usr/local/lib/python3.6/site-packages/fake_useragent/utils.py", line 84, in get
    raise FakeUserAgentError('Maximum amount of retries reached')
fake_useragent.errors.FakeUserAgentError: Maximum amount of retries reached

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.