Code Monkey home page Code Monkey logo

whoogle-search's Introduction

Whoogle Search

Latest Release License: MIT tests buildx codebeat badge Docker Pulls

SourceHut GitHub

Get Google search results, but without any ads, JavaScript, AMP links, cookies, or IP address tracking. Easily deployable in one click as a Docker app, and customizable with a single config file. Quick and simple to implement as a primary search engine replacement on both desktop and mobile.

Contents

  1. Features
  2. Install/Deploy Options
    1. Heroku Quick Deploy
    2. Render.com
    3. Repl.it
    4. Fly.io
    5. Koyeb
    6. pipx
    7. pip
    8. Manual
    9. Docker
    10. Arch/AUR
    11. Helm/Kubernetes
  3. Environment Variables and Configuration
  4. Usage
  5. Extra Steps
    1. Set Primary Search Engine
    2. Custom Redirecting
    3. Custom Bangs
    4. Prevent Downtime (Heroku Only)
    5. Manual HTTPS Enforcement
    6. Using with Firefox Containers
    7. Reverse Proxying
      1. Nginx
  6. Contributing
  7. FAQ
  8. Public Instances
  9. Screenshots

Features

  • No ads or sponsored content
  • No JavaScript*
  • No cookies**
  • No tracking/linking of your personal IP address***
  • No AMP links
  • No URL tracking tags (i.e. utm=%s)
  • No referrer header
  • Tor and HTTP/SOCKS proxy support
  • Autocomplete/search suggestions
  • POST request search and suggestion queries (when possible)
  • View images at full res without site redirect (currently mobile only)
  • Light/Dark/System theme modes (with support for custom CSS theming)
  • Randomly generated User Agent
  • Easy to install/deploy
  • DDG-style bang (i.e. !<tag> <query>) searches
  • User-defined custom bangs
  • Optional location-based searching (i.e. results near <city>)
  • Optional NoJS mode to view search results in a separate window with JavaScript blocked

*No third party JavaScript. Whoogle can be used with JavaScript disabled, but if enabled, uses JavaScript for things like presenting search suggestions.

**No third party cookies. Whoogle uses server side cookies (sessions) to store non-sensitive configuration settings such as theme, language, etc. Just like with JavaScript, cookies can be disabled and not affect Whoogle's search functionality.

***If deployed to a remote server, or configured to send requests through a VPN, Tor, proxy, etc.

Install

There are a few different ways to begin using the app, depending on your preferences:


Deploy

Provides:

  • Easy Deployment of App
  • A HTTPS url (https://<your app name>.herokuapp.com)

Notes:

  • Requires a PAID Heroku Account.
  • Sometimes has issues with auto-redirecting to https. Make sure to navigate to the https version of your app before adding as a default search engine.

Create an account on render.com and import the Whoogle repo with the following settings:

  • Runtime: Python 3
  • Build Command: pip install -r requirements.txt
  • Run Command: ./run

Run on Repl.it

Note: Requires a (free) Replit account

Provides:

  • Free deployment of app
  • Free HTTPS url (https://<app name>.<username>.repl.co)
    • Supports custom domains
  • Downtime after periods of inactivity (solution)

You will need a Fly.io account to deploy Whoogle. The free allowances are enough for personal use.

Deploy the app

flyctl auth login
flyctl launch --image benbusby/whoogle-search:latest

The first deploy won't succeed because the default internal_port is wrong. To fix this, open the generated fly.toml file, set services.internal_port to 5000 and run flyctl launch again.

Your app is now available at https://<app-name>.fly.dev.


Use one of the following guides to install Whoogle on Koyeb:

  1. Using GitHub: https://www.koyeb.com/docs/quickstart/deploy-with-git
  2. Using Docker: https://www.koyeb.com/docs/quickstart/deploy-a-docker-application

Persistent install:

pipx install git+https://github.com/benbusby/whoogle-search.git

Sandboxed temporary instance:

pipx run --spec git+https://github.com/benbusby/whoogle-search.git whoogle-search


pip

pip install whoogle-search

$ whoogle-search --help
usage: whoogle-search [-h] [--port <port number>] [--host <ip address>] [--debug] [--https-only] [--userpass <username:password>]
                      [--proxyauth <username:password>] [--proxytype <socks4|socks5|http>] [--proxyloc <location:port>]

Whoogle Search console runner

optional arguments:
  -h, --help            Show this help message and exit
  --port <port number>  Specifies a port to run on (default 5000)
  --host <ip address>   Specifies the host address to use (default 127.0.0.1)
  --debug               Activates debug mode for the server (default False)
  --https-only          Enforces HTTPS redirects for all requests
  --userpass <username:password>
                        Sets a username/password basic auth combo (default None)
  --proxyauth <username:password>
                        Sets a username/password for a HTTP/SOCKS proxy (default None)
  --proxytype <socks4|socks5|http>
                        Sets a proxy type for all connections (default None)
  --proxyloc <location:port>
                        Sets a proxy location for all connections (default None)

See the available environment variables for additional configuration.


Manual

Note: Content-Security-Policy headers can be sent by Whoogle if you set WHOOGLE_CSP.

Dependencies

  • Python3
  • libcurl4-openssl-dev and libssl-dev
    • macOS: brew install openssl curl-openssl
    • Ubuntu: sudo apt-get install -y libcurl4-openssl-dev libssl-dev
    • Arch: pacman -S curl openssl

Install

Clone the repo and run the following commands to start the app in a local-only environment:

git clone https://github.com/benbusby/whoogle-search.git
cd whoogle-search
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
./run

See the available environment variables for additional configuration.

systemd Configuration

After building the virtual environment, you can add something like the following to /lib/systemd/system/whoogle.service to set up a Whoogle Search systemd service:

[Unit]
Description=Whoogle

[Service]
# Basic auth configuration, uncomment to enable
#Environment=WHOOGLE_USER=<username>
#Environment=WHOOGLE_PASS=<password>
# Proxy configuration, uncomment to enable
#Environment=WHOOGLE_PROXY_USER=<proxy username>
#Environment=WHOOGLE_PROXY_PASS=<proxy password>
#Environment=WHOOGLE_PROXY_TYPE=<proxy type (http|https|proxy4|proxy5)
#Environment=WHOOGLE_PROXY_LOC=<proxy host/ip>
# Site alternative configurations, uncomment to enable
# Note: If not set, the feature will still be available
# with default values.
#Environment=WHOOGLE_ALT_TW=farside.link/nitter
#Environment=WHOOGLE_ALT_YT=farside.link/invidious
#Environment=WHOOGLE_ALT_RD=farside.link/libreddit
#Environment=WHOOGLE_ALT_MD=farside.link/scribe
#Environment=WHOOGLE_ALT_TL=farside.link/lingva
#Environment=WHOOGLE_ALT_IMG=farside.link/rimgo
#Environment=WHOOGLE_ALT_WIKI=farside.link/wikiless
#Environment=WHOOGLE_ALT_IMDB=farside.link/libremdb
#Environment=WHOOGLE_ALT_QUORA=farside.link/quetre
# Load values from dotenv only
#Environment=WHOOGLE_DOTENV=1
Type=simple
User=<username>
# If installed as a package, add:
ExecStart=<python_install_dir>/python3 <whoogle_install_dir>/whoogle-search --host 127.0.0.1 --port 5000
# For example:
# ExecStart=/usr/bin/python3 /home/my_username/.local/bin/whoogle-search --host 127.0.0.1 --port 5000
# Otherwise if running the app from source, add:
ExecStart=<whoogle_repo_dir>/run
# For example:
# ExecStart=/var/www/whoogle-search/run
WorkingDirectory=<whoogle_repo_dir>
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=3
SyslogIdentifier=whoogle

[Install]
WantedBy=multi-user.target

Then,

sudo systemctl daemon-reload
sudo systemctl enable whoogle
sudo systemctl start whoogle

Tor Configuration optional

If routing your request through Tor you will need to make the following adjustments. Due to the nature of interacting with Google through Tor we will need to be able to send signals to Tor and therefore authenticate with it.

There are two authentication methods, password and cookie. You will need to make changes to your torrc:

  • Cookie

    1. Uncomment or add the following lines in your torrc:

      • ControlPort 9051
      • CookieAuthentication 1
      • DataDirectoryGroupReadable 1
      • CookieAuthFileGroupReadable 1
    2. Make the tor auth cookie readable:

      • This is assuming that you are using a dedicated user to run whoogle. If you are using a different user replace whoogle with that user.
      1. chmod tor:whoogle /var/lib/tor
      2. chmod tor:whoogle /var/lib/tor/control_auth_cookie
    3. Restart the tor service:

      • systemctl restart tor
    4. Set the Tor environment variable to 1, WHOOGLE_CONFIG_TOR. Refer to the Environment Variables section for more details.

      • This may be added in the systemd unit file or env file WHOOGLE_CONFIG_TOR=1
  • Password

    1. Run this command:

      • tor --hash-password {Your Password Here}; put your password in place of {Your Password Here}.
      • Keep the output of this command, you will be placing it in your torrc.
      • Keep the password input of this command, you will be using it later.
    2. Uncomment or add the following lines in your torrc:

      • ControlPort 9051
      • HashedControlPassword {Place output here}; put the output of the previous command in place of {Place output here}.
    3. Now take the password from the first step and place it in the control.conf file within the whoogle working directory, ie. misc/tor/control.conf

      • If you want to place your password file in a different location set this location with the WHOOGLE_TOR_CONF environment variable. Refer to the Environment Variables section for more details.
    4. Heavily restrict access to control.conf to only be readable by the user running whoogle:

      • chmod 400 control.conf
    5. Finally set the Tor environment variable and use password variable to 1, WHOOGLE_CONFIG_TOR and WHOOGLE_TOR_USE_PASS. Refer to the Environment Variables section for more details.

      • These may be added to the systemd unit file or env file:
        • WHOOGLE_CONFIG_TOR=1
        • WHOOGLE_TOR_USE_PASS=1

Manual (Docker)

  1. Ensure the Docker daemon is running, and is accessible by your user account
  • To add user permissions, you can execute sudo usermod -aG docker yourusername
  • Running docker ps should return something besides an error. If you encounter an error saying the daemon isn't running, try sudo systemctl start docker (Linux) or ensure the docker tool is running (Windows/macOS).
  1. Clone and deploy the docker app using a method below:

Docker CLI

Through Docker Hub:

docker pull benbusby/whoogle-search
docker run --publish 5000:5000 --detach --name whoogle-search benbusby/whoogle-search:latest

or with docker-compose:

git clone https://github.com/benbusby/whoogle-search.git
cd whoogle-search
docker-compose up

or by building yourself:

git clone https://github.com/benbusby/whoogle-search.git
cd whoogle-search
docker build --tag whoogle-search:1.0 .
docker run --publish 5000:5000 --detach --name whoogle-search whoogle-search:1.0

Optionally, you can also enable some of the following environment variables to further customize your instance:

docker run --publish 5000:5000 --detach --name whoogle-search \
  -e WHOOGLE_USER=username \
  -e WHOOGLE_PASS=password \
  -e WHOOGLE_PROXY_USER=username \
  -e WHOOGLE_PROXY_PASS=password \
  -e WHOOGLE_PROXY_TYPE=socks5 \
  -e WHOOGLE_PROXY_LOC=ip \
  whoogle-search:1.0

And kill with: docker rm --force whoogle-search

heroku login
heroku container:login
git clone https://github.com/benbusby/whoogle-search.git
cd whoogle-search
heroku create
heroku container:push web
heroku container:release web
heroku open

This series of commands can take a while, but once you run it once, you shouldn't have to run it again. The final command, heroku open will launch a tab in your web browser, where you can test out Whoogle and even set it as your primary search engine. You may also edit environment variables from your app’s Settings tab in the Heroku Dashboard.


Arch Linux & Arch-based Distributions

There is an AUR package available, as well as a pre-built and daily updated package available at Chaotic-AUR.


Helm chart for Kubernetes

To use the Kubernetes Helm Chart:

  1. Ensure you have Helm >=3.0.0 installed
  2. Clone this repository
  3. Update charts/whoogle/values.yaml as desired
  4. Run helm install whoogle ./charts/whoogle

Using your own server, or alternative container deployment

There are other methods for deploying docker containers that are well outlined in this article, but there are too many to describe set up for each here. Generally it should be about the same amount of effort as the Heroku deployment.

Depending on your preferences, you can also deploy the app yourself on your own infrastructure. This route would require a few extra steps:

  • A server (I personally recommend Digital Ocean or Linode, their cheapest tiers will work fine)
  • Your own URL (I suppose this is optional, but recommended)
  • SSL certificates (free through Let's Encrypt)
  • A bit more experience or willingness to work through issues

Environment Variables

There are a few optional environment variables available for customizing a Whoogle instance. These can be set manually, or copied into whoogle.env and enabled for your preferred deployment method:

  • Local runs: Set WHOOGLE_DOTENV=1 before running
  • With docker-compose: Uncomment the env_file option
  • With docker build/run: Add --env-file ./whoogle.env to your command
Variable Description
WHOOGLE_URL_PREFIX The URL prefix to use for the whoogle instance (i.e. "/whoogle")
WHOOGLE_DOTENV Load environment variables in whoogle.env
WHOOGLE_USER The username for basic auth. WHOOGLE_PASS must also be set if used.
WHOOGLE_PASS The password for basic auth. WHOOGLE_USER must also be set if used.
WHOOGLE_PROXY_USER The username of the proxy server.
WHOOGLE_PROXY_PASS The password of the proxy server.
WHOOGLE_PROXY_TYPE The type of the proxy server. Can be "socks5", "socks4", or "http".
WHOOGLE_PROXY_LOC The location of the proxy server (host or ip).
WHOOGLE_USER_AGENT The desktop user agent to use. Defaults to a randomly generated one.
WHOOGLE_USER_AGENT_MOBILE The mobile user agent to use. Defaults to a randomly generated one.
WHOOGLE_USE_CLIENT_USER_AGENT Enable to use your own user agent for all requests. Defaults to false.
WHOOGLE_REDIRECTS Specify sites that should be redirected elsewhere. See custom redirecting.
EXPOSE_PORT The port where Whoogle will be exposed.
HTTPS_ONLY Enforce HTTPS. (See here)
WHOOGLE_ALT_TW The twitter.com alternative to use when site alternatives are enabled in the config. Set to "" to disable.
WHOOGLE_ALT_YT The youtube.com alternative to use when site alternatives are enabled in the config. Set to "" to disable.
WHOOGLE_ALT_RD The reddit.com alternative to use when site alternatives are enabled in the config. Set to "" to disable.
WHOOGLE_ALT_TL The Google Translate alternative to use. This is used for all "translate ____" searches. Set to "" to disable.
WHOOGLE_ALT_MD The medium.com alternative to use when site alternatives are enabled in the config. Set to "" to disable.
WHOOGLE_ALT_IMG The imgur.com alternative to use when site alternatives are enabled in the config. Set to "" to disable.
WHOOGLE_ALT_WIKI The wikipedia.org alternative to use when site alternatives are enabled in the config. Set to "" to disable.
WHOOGLE_ALT_IMDB The imdb.com alternative to use when site alternatives are enabled in the config. Set to "" to disable.
WHOOGLE_ALT_QUORA The quora.com alternative to use when site alternatives are enabled in the config. Set to "" to disable.
WHOOGLE_AUTOCOMPLETE Controls visibility of autocomplete/search suggestions. Default on -- use '0' to disable.
WHOOGLE_MINIMAL Remove everything except basic result cards from all search queries.
WHOOGLE_CSP Sets a default set of 'Content-Security-Policy' headers
WHOOGLE_RESULTS_PER_PAGE Set the number of results per page
WHOOGLE_TOR_SERVICE Enable/disable the Tor service on startup. Default on -- use '0' to disable.
WHOOGLE_TOR_USE_PASS Use password authentication for tor control port.
WHOOGLE_TOR_CONF The absolute path to the config file containing the password for the tor control port. Default: ./misc/tor/control.conf WHOOGLE_TOR_PASS must be 1 for this to work.
WHOOGLE_SHOW_FAVICONS Show/hide favicons next to search result URLs. Default on.
WHOOGLE_UPDATE_CHECK Enable/disable the automatic daily check for new versions of Whoogle. Default on.

Config Environment Variables

These environment variables allow setting default config values, but can be overwritten manually by using the home page config menu. These allow a shortcut for destroying/rebuilding an instance to the same config state every time.

Variable Description
WHOOGLE_CONFIG_DISABLE Hide config from UI and disallow changes to config by client
WHOOGLE_CONFIG_COUNTRY Filter results by hosting country
WHOOGLE_CONFIG_LANGUAGE Set interface language
WHOOGLE_CONFIG_SEARCH_LANGUAGE Set search result language
WHOOGLE_CONFIG_BLOCK Block websites from search results (use comma-separated list)
WHOOGLE_CONFIG_BLOCK_TITLE Block search result with a REGEX filter on title
WHOOGLE_CONFIG_BLOCK_URL Block search result with a REGEX filter on URL
WHOOGLE_CONFIG_THEME Set theme mode (light, dark, or system)
WHOOGLE_CONFIG_SAFE Enable safe searches
WHOOGLE_CONFIG_ALTS Use social media site alternatives (nitter, invidious, etc)
WHOOGLE_CONFIG_NEAR Restrict results to only those near a particular city
WHOOGLE_CONFIG_TOR Use Tor routing (if available)
WHOOGLE_CONFIG_NEW_TAB Always open results in new tab
WHOOGLE_CONFIG_VIEW_IMAGE Enable View Image option
WHOOGLE_CONFIG_GET_ONLY Search using GET requests only
WHOOGLE_CONFIG_URL The root url of the instance (https://<your url>/)
WHOOGLE_CONFIG_STYLE The custom CSS to use for styling (should be single line)
WHOOGLE_CONFIG_PREFERENCES_ENCRYPTED Encrypt preferences token, requires preferences key
WHOOGLE_CONFIG_PREFERENCES_KEY Key to encrypt preferences in URL (REQUIRED to show url)
WHOOGLE_CONFIG_ANON_VIEW Include the "anonymous view" option for each search result

Usage

Same as most search engines, with the exception of filtering by time range.

To filter by a range of time, append ":past " to the end of your search, where can be hour, day, month, or year. Example: coronavirus updates :past hour

Extra Steps

Set Whoogle as your primary search engine

Note: If you're using a reverse proxy to run Whoogle Search, make sure the "Root URL" config option on the home page is set to your URL before going through these steps.

Browser settings:

  • Firefox (Desktop)
    • Version 89+
      • Navigate to your app's url, right click the address bar, and select "Add Search Engine".
    • Previous versions
      • Navigate to your app's url, and click the 3 dot menu in the address bar. At the bottom, there should be an option to "Add Search Engine".
    • Once you've added the new search engine, open your Firefox Preferences menu, click "Search" in the left menu, and use the available dropdown to select "Whoogle" from the list.
    • Note: If your Whoogle instance uses Firefox Containers, you'll need to go through the steps here to get it working properly.
  • Firefox (iOS)
    • In the mobile app Settings page, tap "Search" within the "General" section. There should be an option titled "Add Search Engine" to select. It should prompt you to enter a title and search query url - use the following elements to fill out the form:
      • Title: "Whoogle"
      • URL: http[s]://\<your whoogle url\>/search?q=%s
  • Firefox (Android)
    • Version <79.0.0
      • Navigate to your app's url
      • Long-press on the search text field
      • Click the "Add Search Engine" menu item
        • Select a name and click ok
      • Click the 3 dot menu in the top right
      • Navigate to the settings menu and select the "Search" sub-menu
      • Select Whoogle and press "Set as default"
    • Version >=79.0.0
      • Click the 3 dot menu in the top right
      • Navigate to the settings menu and select the "Search" sub-menu
      • Click "Add search engine"
      • Select the 'Other' radio button
        • Name: "Whoogle"
        • Search string to use: https://\<your whoogle url\>/search?q=%s
  • Alfred (Mac OS X)
    1. Go to Alfred Preferences > Features > Web Search and click Add Custom Search. Then configure these settings

      • Search URL: `https://<your whoogle url>/search?q={query}
      • Title: Whoogle for '{query}' (or whatever you want)
      • Keyword: whoogle
    2. Go to Default Results and click the Setup fallback results button. Click + and add Whoogle, then drag it to the top.

  • Chrome/Chromium-based Browsers
    • Automatic
      • Visit the home page of your Whoogle Search instance -- this will automatically add the search engine if the requirements are met (GET request, no OnSubmit script, no path). If not, you can add it manually.
    • Manual
      • Under search engines > manage search engines > add, manually enter your Whoogle instance details with a <whoogle url>/search?q=%s formatted search URL.

Custom Redirecting

You can set custom site redirects using the WHOOGLE_REDIRECTS environment variable. A lot of sites, such as Twitter, Reddit, etc, have built-in redirects to Farside links, but you may want to define your own.

To do this, you can use the following syntax:

WHOOGLE_REDIRECTS="<parent_domain>:<new_domain>"

For example, if you want to redirect from "badsite.com" to "goodsite.com":

WHOOGLE_REDIRECTS="badsite.com:goodsite.com"

This can be used for multiple sites as well, with comma separation:

WHOOGLE_REDIRECTS="badA.com:goodA.com,badB.com:goodB.com"

NOTE: Do not include "http(s)://" when defining your redirect.

Custom Bangs

You can create your own custom bangs. By default, bangs are stored in app/static/bangs. See 00-whoogle.json for an example. These are parsed in alphabetical order with later files overriding bangs set in earlier files, with the exception that DDG bangs (downloaded to app/static/bangs/bangs.json) are always parsed first. Thus, any custom bangs will always override the DDG ones.

Prevent Downtime (Heroku only)

Part of the deal with Heroku's free tier is that you're allocated 550 hours/month (meaning it can't stay active 24/7), and the app is temporarily shut down after 30 minutes of inactivity. Once it becomes inactive, any Whoogle searches will still work, but it'll take an extra 10-15 seconds for the app to come back online before displaying the result, which can be frustrating if you're in a hurry.

A good solution for this is to set up a simple cronjob on any device at your home that is consistently powered on and connected to the internet (in my case, a PiHole worked perfectly). All the device needs to do is fetch app content on a consistent basis to keep the app alive in whatever ~17 hour window you want it on (17 hrs * 31 days = 527, meaning you'd still have 23 leftover hours each month if you searched outside of your target window).

For instance, adding */20 7-23 * * * curl https://<your heroku app name>.herokuapp.com > /home/<username>/whoogle-refresh will fetch the home page of the app every 20 minutes between 7am and midnight, allowing for downtime from midnight to 7am. And again, this wouldn't be a hard limit - you'd still have plenty of remaining hours of uptime each month in case you were searching after this window has closed.

Since the instance is destroyed and rebuilt after inactivity, config settings will be reset once the app enters downtime. If you have configuration settings active that you'd like to keep between periods of downtime (like dark mode for example), you could instead add */20 7-23 * * * curl -d "dark=1" -X POST https://<your heroku app name>.herokuapp.com/config > /home/<username>/whoogle-refresh to keep these settings more or less permanent, and still keep the app from entering downtime when you're using it.

HTTPS Enforcement

Only needed if your setup requires Flask to redirect to HTTPS on its own -- generally this is something that doesn't need to be handled by Whoogle Search.

Note: You should have your own domain name and an https certificate in order for this to work properly.

  • Heroku: Ensure that the Root URL configuration on the home page begins with https:// and not http://
  • Docker build: Add --build-arg use_https=1 to your run command
  • Docker image: Set the environment variable HTTPS_ONLY=1
  • Pip/Pipx: Add the --https-only flag to the end of the whoogle-search command
  • Default run script: Modify the script locally to include the --https-only flag at the end of the python run command

Using with Firefox Containers

Unfortunately, Firefox Containers do not currently pass through POST requests (the default) to the engine, and Firefox caches the opensearch template on initial page load. To get around this, you can take the following steps to get it working as expected:

  1. Remove any existing Whoogle search engines from Firefox settings
  2. Enable GET Requests Only in Whoogle config
  3. Clear Firefox cache
  4. Restart Firefox
  5. Navigate to Whoogle instance and re-add the engine

Reverse Proxying

Nginx

Here is a sample Nginx config for Whoogle:

server {
	server_name your_domain_name.com;
	access_log /dev/null;
	error_log /dev/null;

	location / {
	    proxy_set_header X-Real-IP $remote_addr;
	    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	    proxy_set_header X-Forwarded-Proto $scheme;
	    proxy_set_header Host $host;
	    proxy_set_header X-NginX-Proxy true;
	    proxy_pass http://localhost:5000;
	}
}

You can then add SSL support using LetsEncrypt by following a guide such as this one.

Contributing

Under the hood, Whoogle is a basic Flask app with the following structure:

  • app/
    • routes.py: Primary app entrypoint, contains all API routes
    • request.py: Handles all outbound requests, including proxied/Tor connectivity
    • filter.py: Functions and utilities used for filtering out content from upstream Google search results
    • utils/
      • bangs.py: All logic related to handling DDG-style "bang" queries
      • results.py: Utility functions for interpreting/modifying individual search results
      • search.py: Creates and handles new search queries
      • session.py: Miscellaneous methods related to user sessions
    • templates/
      • index.html: The home page template
      • display.html: The search results template
      • header.html: A general "top of the page" query header for desktop and mobile
      • search.html: An iframe-able search page
      • logo.html: A template consisting mostly of the Whoogle logo as an SVG (separated to help keep index.html a bit cleaner)
      • opensearch.xml: A template used for supporting OpenSearch.
      • imageresults.html: An "experimental" template used for supporting the "Full Size" image feature on desktop.
    • static/<css|js>
      • CSS/JavaScript files, should be self-explanatory
    • static/settings
      • Key-value JSON files for establishing valid configuration values

If you're new to the project, the easiest way to get started would be to try fixing an open bug report. If there aren't any open, or if the open ones are too stale, try taking on a feature request. Generally speaking, if you can write something that has any potential of breaking down in the future, you should write a test for it.

The project follows the PEP 8 Style Guide, but is liable to change. Static typing should always be used when possible. Function documentation is greatly appreciated, and typically follows the below format:

def contains(x: list, y: int) -> bool:
    """Check a list (x) for the presence of an element (y)

    Args:
        x: The list to inspect
        y: The int to look for

    Returns:
        bool: True if the list contains the item, otherwise False
    """

    return y in x

Translating

Whoogle currently supports translations using translations.json. Language values in this file need to match the "value" of the according language in languages.json (i.e. "lang_en" for English, "lang_es" for Spanish, etc). After you add a new set of translations to translations.json, open a PR with your changes and they will be merged in as soon as possible.

FAQ

What's the difference between this and Searx?

Whoogle is intended to only ever be deployed to private instances by individuals of any background, with as little effort as possible. Prior knowledge of/experience with the command line or deploying applications is not necessary to deploy Whoogle, which isn't the case with Searx. As a result, Whoogle is missing some features of Searx in order to be as easy to deploy as possible.

Whoogle also only uses Google search results, not Bing/Quant/etc, and uses the existing Google search UI to make the transition away from Google search as unnoticeable as possible.

I'm a huge fan of Searx though and encourage anyone to use that instead if they want access to other search engines/a different UI/more configuration.

Why does the image results page look different?

A lot of the app currently piggybacks on Google's existing support for fetching results pages with JavaScript disabled. To their credit, they've done an excellent job with styling pages, but it seems that the image results page - particularly on mobile - is a little rough. Moving forward, with enough interest, I'd like to transition to fetching the results and parsing them into a unique Whoogle-fied interface that I can style myself.

Public Instances

Note: Use public instances at your own discretion. The maintainers of Whoogle do not personally validate the integrity of any other instances. Popular public instances are more likely to be rate-limited or blocked.

Website Country Language Cloudflare
https://search.albony.xyz 🇮🇳 IN Multi-choice
https://search.garudalinux.org 🇫🇮 FI Multi-choice
https://search.dr460nf1r3.org 🇩🇪 DE Multi-choice
https://s.tokhmi.xyz 🇺🇸 US Multi-choice
https://search.sethforprivacy.com 🇩🇪 DE English
https://whoogle.dcs0.hu 🇭🇺 HU Multi-choice
https://gowogle.voring.me 🇺🇸 US Multi-choice
https://whoogle.privacydev.net 🇫🇷 FR English
https://wg.vern.cc 🇺🇸 US English
https://whoogle.hxvy0.gq 🇨🇦 CA Turkish Only
https://whoogle.hostux.net  🇫🇷 FR Multi-choice
https://whoogle.lunar.icu 🇩🇪 DE Multi-choice
https://wgl.frail.duckdns.org 🇧🇷 BR Multi-choice
https://whoogle.no-logs.com 🇸🇪 SE Multi-choice
https://whoogle.ftw.lol 🇩🇪 DE Multi-choice
https://whoogle-search--replitcomreside.repl.co 🇺🇸 US English
https://search.notrustverify.ch 🇨🇭 CH Multi-choice
https://whoogle.datura.network 🇩🇪 DE Multi-choice
https://whoogle.yepserver.xyz 🇺🇦 UA Multi-choice
https://search.nezumi.party 🇮🇹 IT Multi-choice
https://search.snine.nl 🇳🇱 NL Mult-choice
  • A checkmark in the "Cloudflare" category here refers to the use of the reverse proxy, Cloudflare. The checkmark will not be listed for a site which uses Cloudflare DNS but rather the proxying service which grants Cloudflare the ability to monitor traffic to the website.

Onion Instances

Website Country Language
http://whoglqjdkgt2an4tdepberwqz3hk7tjo4kqgdnuj77rt7nshw2xqhqad.onion 🇺🇸 US Multi-choice
http://nuifgsnbb2mcyza74o7illtqmuaqbwu4flam3cdmsrnudwcmkqur37qd.onion 🇩🇪 DE English
http://whoogle.vernccvbvyi5qhfzyqengccj7lkove6bjot2xhh5kajhwvidqafczrad.onion 🇺🇸 US English
http://whoogle.g4c3eya4clenolymqbpgwz3q3tawoxw56yhzk4vugqrl6dtu3ejvhjid.onion 🇫🇷 FR English
http://whoogle.daturab6drmkhyeia4ch5gvfc2f3wgo6bhjrv3pz6n7kxmvoznlkq4yd.onion 🇩🇪 DE Multi-choice

I2P Instances

Website Country Language
http://verneks7rfjptpz5fpii7n7nrxilsidi2qxepeuuf66c3tsf4nhq.b32.i2p 🇺🇸 US English

Screenshots

Desktop

Whoogle Desktop

Mobile

Whoogle Mobile

whoogle-search's People

Contributors

ahmad-alkadri avatar albonycal avatar alefvanoon avatar benbusby avatar cyaxxx avatar cybertailor avatar dependabot[bot] avatar dr460nf1r3 avatar duolabs333 avatar fiestasiesta avatar fredster33 avatar gdm85 avatar glitsj16 avatar gripped avatar invis-z avatar jacr13 avatar madcowog avatar marvinborner avatar pantherman594 avatar privacydevel avatar ras07 avatar realorangeone avatar sayuri-gi avatar sesseor avatar shimuldn avatar suzaku avatar vacom13 avatar watchakorn-18k avatar xanax-c-137 avatar xatier avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

whoogle-search's Issues

[BUG] opensearch.xml always using localhost

When using Docker behind an Nginx proxy and ran into an issue where "Adding Search Engine" with Firefox isn't working. The base URL in the opensearch.xml file ends up being "localhost:8888" instead of my host URL.

To Reproduce

  1. In Firefox, navigate to a non-localhost instance of Whoogle
  2. Click on the three dots in the URL -> Add Search Engine
  3. Attempt to search in Firefox with the new Whoogle engine
  4. See "unable to connect to localhost:8888"

Expected behavior
The browser search engine would use the actual URL of the instance (passed with the --host param).

Workaround
I worked around this issue by changing the main_url param in routes.py to use my custom domain:
template = render_template('opensearch.xml', main_url='<MY_URL_HERE>')

[BUG] Whoogle-Search doesn't work behind Apache2 reverse proxy

Describe the bug
Whoogle Search is unable to work behing an Apache2 reverse proxy, and it only works when started up by explicitly telling it to run on the external IP (else, it tries to run locally on 127.0.0.1, denying any external request).

To Reproduce
Steps to reproduce the behavior:

  1. Install whoogle-search via pip
  2. Make an apache2 reverse proxy, using a config like this:
<VirtualHost *:443>
    ServerName search.domain.com

    ProxyRequests Off
    ProxyPreserveHost On
    ProxyVia Full
    <Proxy *>
       Require all granted
    </Proxy>

    ProxyPass "/" "http://127.0.0.1:8888"
    ProxyPassReverse "/" "http://127.0.0.1:8888"

    Include <removed>
    SSLCertificateFile <removed>
    SSLCertificateKeyFile <removed>
</VirtualHost>
  1. Enable website with a2ensite
  2. Restart apache2
  3. Start whoogle with whoogle-search

Expected behavior
Whoogle homepage should load correctly. However, all I get is a blank page.
The server logs only show this:

 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
127.0.0.1 - - [12/May/2020 00:32:34] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [12/May/2020 00:32:37] "GET / HTTP/1.1" 200 -

Showing that requests are indeed coming, however from IP 127.0.0.1, which is the localhost of the server.

If I run whoogle-search by using the same default command, "whoogle-search", and I try accessing it directly from IP:port, I am unable to even reach it and the console doesn't log anything.

The only way I have to access the homepage is by using whoogle-search --host 149.XX.YY.Z and then directly accessing it in the browser via IP:port. Then, the console shows correct info:

 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://149.XX.YY.Z:8888/ (Press CTRL+C to quit)
84.17.YY.XX - - [12/May/2020 00:39:24] "GET / HTTP/1.1" 200 -
84.17.YY.XX - - [12/May/2020 00:39:24] "GET /static/js/controller.js HTTP/1.1" 200 -
84.17.YY.XX - - [12/May/2020 00:39:24] "GET /static/css/main.css HTTP/1.1" 200 -
84.17.YY.XX - - [12/May/2020 00:39:24] "GET /static/img/logo.png HTTP/1.1" 200 -
84.17.YY.XX - - [12/May/2020 00:39:24] "GET /config HTTP/1.1" 200 -
84.17.YY.XX - - [12/May/2020 00:39:24] "GET /static/img/favicon/favicon-32x32.png HTTP/1.1" 200 -
84.17.YY.XX - - [12/May/2020 00:39:24] "GET /static/img/favicon/android-icon-192x192.png HTTP/1.1" 200 -

Desktop (please complete the following information):

  • OS: Ubuntu 20.04
  • Browser: Chrome and Firefox

[BUG] Image page does not adhere to Dark Mode

When you search for something and you click on Images, Dark Mode does not adhere to that page. Also, it looks like the search bar text is white which would be true if Dark Mode was on for this page.

I'm running this in Docker on CentOS8. Everything else works great.

Screen Shot 2020-05-15 at 11 51 24 AM

[BUG]

Describe the bug
Unable to install, getting error messages.

To Reproduce
Using pip:
pip install whoogle-search Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Collecting whoogle-search Could not find a version that satisfies the requirement whoogle-search (from versions: ) No matching distribution found for whoogle-search

Using manual:
sudo pip install -r requirements.txt Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Collecting beautifulsoup4==4.8.2 (from -r requirements.txt (line 1)) Using cached https://files.pythonhosted.org/packages/c5/48/c88b0b390ae1f785942fc83413feb1268a1eb696f343d4d55db735b9bb39/beautifulsoup4-4.8.2-py2-none-any.whl Collecting bs4==0.0.1 (from -r requirements.txt (line 2)) Using cached https://files.pythonhosted.org/packages/10/ed/7e8b97591f6f456174139ec089c769f89a94a1a4025fe967691de971f314/bs4-0.0.1.tar.gz Collecting cffi==1.13.2 (from -r requirements.txt (line 3)) Using cached https://files.pythonhosted.org/packages/2d/bf/960e5a422db3ac1a5e612cb35ca436c3fc985ed4b7ed13a1b4879006f450/cffi-1.13.2.tar.gz Requirement already satisfied: Click==7.0 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 4)) (7.0) Collecting cryptography==2.8 (from -r requirements.txt (line 5)) Using cached https://files.pythonhosted.org/packages/be/60/da377e1bed002716fb2d5d1d1cab720f298cb33ecff7bf7adea72788e4e4/cryptography-2.8.tar.gz Installing build dependencies ... error Complete output from command /usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-i8SqDp --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- setuptools>=40.6.0 wheel "cffi>=1.8,!=1.11.3; platform_python_implementation != 'PyPy'": Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple Collecting setuptools>=40.6.0 Using cached https://files.pythonhosted.org/packages/ab/b5/3679d7c98be5b65fa5522671ef437b792d909cf3908ba54fe9eca5d2a766/setuptools-44.1.0-py2.py3-none-any.whl Collecting wheel Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl Collecting cffi!=1.11.3,>=1.8 Using cached https://files.pythonhosted.org/packages/05/54/3324b0c46340c31b909fcec598696aaec7ddc8c18a63f2db352562d3354c/cffi-1.14.0.tar.gz Collecting pycparser (from cffi!=1.11.3,>=1.8) Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl Building wheels for collected packages: cffi Running setup.py bdist_wheel for cffi: started Running setup.py bdist_wheel for cffi: finished with status 'error' Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-brYaqp/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-cSmGIq --python-tag cp27: Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc' to the PKG_CONFIG_PATH environment variable No package 'libffi' found Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc' to the PKG_CONFIG_PATH environment variable No package 'libffi' found Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-armv7l-2.7
creating build/lib.linux-armv7l-2.7/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/model.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/recompiler.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/pkgconfig.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/lock.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/verifier.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/error.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/api.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/cparser.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/vengine_gen.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/ffiplatform.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/init.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/commontypes.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_cffi_include.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/parse_c_type.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_embedding.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-armv7l-2.7/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-armv7l-2.7
creating build/temp.linux-armv7l-2.7/c
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-InigCj/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv7l-2.7/c/_cffi_backend.o
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
#include <ffi.h>
^~~~~~~
compilation terminated.
error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

----------------------------------------
Failed building wheel for cffi
Running setup.py clean for cffi

Failed to build cffi
Installing collected packages: setuptools, wheel, pycparser, cffi
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-brYaqp/cffi/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-YSiiMB/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-i8SqDp --compile:
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc' to the PKG_CONFIG_PATH environment variable No package 'libffi' found Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc' to the PKG_CONFIG_PATH environment variable No package 'libffi' found Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
running install
running build
running build_py
creating build
creating build/lib.linux-armv7l-2.7
creating build/lib.linux-armv7l-2.7/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/model.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/recompiler.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/pkgconfig.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/lock.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/verifier.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/error.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/api.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/cparser.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/vengine_gen.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/ffiplatform.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/init.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/commontypes.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_cffi_include.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/parse_c_type.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_embedding.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-armv7l-2.7/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-armv7l-2.7
creating build/temp.linux-armv7l-2.7/c
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-InigCj/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv7l-2.7/c/_cffi_backend.o
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
#include <ffi.h>
^~~~~~~
compilation terminated.
error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

  ----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-brYaqp/cffi/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-YSiiMB/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-i8SqDp --compile" failed with error code 1 in /tmp/pip-install-brYaqp/cffi/


Command "/usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-i8SqDp --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- setuptools>=40.6.0 wheel "cffi>=1.8,!=1.11.3; platform_python_implementation != 'PyPy'"" failed with error code 1 in None
(venv) pi@raspberrypi:/var/www/whoogle-search $ sudo apt-get install -y libcurl4-openssl-dev libssl-dev
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
cgroupfs-mount docker.io golang-docker-credential-helpers libfam0 libintl-perl libintl-xs-perl
libllvm8 libmicrodns0 libmodule-find-perl libmodule-scandeps-perl libproc-processtable-perl
libsort-naturally-perl libva-wayland2 needrestart python3-cached-property python3-docker
python3-dockerpty python3-dockerpycreds python3-docopt python3-jsonschema python3-texttable
python3-websocket python3-yaml runc tini
Use 'sudo apt autoremove' to remove them.
Suggested packages:
libcurl4-doc libidn11-dev libkrb5-dev libldap2-dev librtmp-dev libssh2-1-dev libssl-doc
The following NEW packages will be installed:
libcurl4-openssl-dev libssl-dev
0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,950 kB of archives.
After this operation, 7,042 kB of additional disk space will be used.
Get:1 http://archive.raspberrypi.org/debian buster/main armhf libssl-dev armhf 1.1.1d-0+deb10u3+rpt1 [1,583 kB]
Get:2 http://raspbian.mirror.constant.com/raspbian buster/main armhf libcurl4-openssl-dev armhf 7.64.0-4+deb10u1 [366 kB]
Fetched 1,950 kB in 2s (1,060 kB/s)
Selecting previously unselected package libcurl4-openssl-dev:armhf.
(Reading database ... 157925 files and directories currently installed.)
Preparing to unpack .../libcurl4-openssl-dev_7.64.0-4+deb10u1_armhf.deb ...
Unpacking libcurl4-openssl-dev:armhf (7.64.0-4+deb10u1) ...
Selecting previously unselected package libssl-dev:armhf.
Preparing to unpack .../libssl-dev_1.1.1d-0+deb10u3+rpt1_armhf.deb ...
Unpacking libssl-dev:armhf (1.1.1d-0+deb10u3+rpt1) ...
Setting up libcurl4-openssl-dev:armhf (7.64.0-4+deb10u1) ...
Setting up libssl-dev:armhf (1.1.1d-0+deb10u3+rpt1) ...
Processing triggers for man-db (2.8.5-2) ...
Scanning processes...
Scanning linux images...

Failed to check for processor microcode upgrades.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.
(venv) pi@raspberrypi:/var/www/whoogle-search $ sudo python3 -m venv venv
(venv) pi@raspberrypi:/var/www/whoogle-search $ source venv/bin/activate
(venv) pi@raspberrypi:/var/www/whoogle-search $ sudo pip install -r requirements.txt
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple
Collecting beautifulsoup4==4.8.2 (from -r requirements.txt (line 1))
Using cached https://files.pythonhosted.org/packages/c5/48/c88b0b390ae1f785942fc83413feb1268a1eb696f343d4d55db735b9bb39/beautifulsoup4-4.8.2-py2-none-any.whl
Collecting bs4==0.0.1 (from -r requirements.txt (line 2))
Using cached https://files.pythonhosted.org/packages/10/ed/7e8b97591f6f456174139ec089c769f89a94a1a4025fe967691de971f314/bs4-0.0.1.tar.gz
Collecting cffi==1.13.2 (from -r requirements.txt (line 3))
Using cached https://files.pythonhosted.org/packages/2d/bf/960e5a422db3ac1a5e612cb35ca436c3fc985ed4b7ed13a1b4879006f450/cffi-1.13.2.tar.gz
Requirement already satisfied: Click==7.0 in /usr/lib/python2.7/dist-packages (from -r requirements.txt (line 4)) (7.0)
Collecting cryptography==2.8 (from -r requirements.txt (line 5))
Using cached https://files.pythonhosted.org/packages/be/60/da377e1bed002716fb2d5d1d1cab720f298cb33ecff7bf7adea72788e4e4/cryptography-2.8.tar.gz
Installing build dependencies ... error
Complete output from command /usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-lN17zy --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- setuptools>=40.6.0 wheel "cffi>=1.8,!=1.11.3; platform_python_implementation != 'PyPy'":
Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple, https://www.piwheels.org/simple
Collecting setuptools>=40.6.0
Using cached https://files.pythonhosted.org/packages/ab/b5/3679d7c98be5b65fa5522671ef437b792d909cf3908ba54fe9eca5d2a766/setuptools-44.1.0-py2.py3-none-any.whl
Collecting wheel
Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8
Using cached https://files.pythonhosted.org/packages/05/54/3324b0c46340c31b909fcec598696aaec7ddc8c18a63f2db352562d3354c/cffi-1.14.0.tar.gz
Collecting pycparser (from cffi!=1.11.3,>=1.8)
Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
Building wheels for collected packages: cffi
Running setup.py bdist_wheel for cffi: started
Running setup.py bdist_wheel for cffi: finished with status 'error'
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-A6lrAK/cffi/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/pip-wheel-KkNaxR --python-tag cp27:
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc' to the PKG_CONFIG_PATH environment variable No package 'libffi' found Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc' to the PKG_CONFIG_PATH environment variable No package 'libffi' found Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-armv7l-2.7
creating build/lib.linux-armv7l-2.7/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/model.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/recompiler.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/pkgconfig.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/lock.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/verifier.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/error.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/api.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/cparser.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/vengine_gen.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/ffiplatform.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/init.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/commontypes.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_cffi_include.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/parse_c_type.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_embedding.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-armv7l-2.7/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-armv7l-2.7
creating build/temp.linux-armv7l-2.7/c
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-InigCj/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv7l-2.7/c/_cffi_backend.o
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
#include <ffi.h>
^~~~~~~
compilation terminated.
error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

----------------------------------------
Failed building wheel for cffi
Running setup.py clean for cffi

Failed to build cffi
Installing collected packages: setuptools, wheel, pycparser, cffi
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-A6lrAK/cffi/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-049HJN/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-lN17zy --compile:
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc' to the PKG_CONFIG_PATH environment variable No package 'libffi' found Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing libffi.pc' to the PKG_CONFIG_PATH environment variable No package 'libffi' found Package libffi was not found in the pkg-config search path. Perhaps you should add the directory containing libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
Package libffi was not found in the pkg-config search path.
Perhaps you should add the directory containing `libffi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libffi' found
running install
running build
running build_py
creating build
creating build/lib.linux-armv7l-2.7
creating build/lib.linux-armv7l-2.7/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/model.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/recompiler.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/pkgconfig.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/lock.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/verifier.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/error.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/api.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/cparser.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/vengine_gen.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/ffiplatform.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/init.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/commontypes.py -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_cffi_include.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/parse_c_type.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_embedding.h -> build/lib.linux-armv7l-2.7/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-armv7l-2.7/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-armv7l-2.7
creating build/temp.linux-armv7l-2.7/c
arm-linux-gnueabihf-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fdebug-prefix-map=/build/python2.7-InigCj/python2.7-2.7.16=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/include/python2.7 -c c/_cffi_backend.c -o build/temp.linux-armv7l-2.7/c/_cffi_backend.o
c/_cffi_backend.c:15:10: fatal error: ffi.h: No such file or directory
#include <ffi.h>
^~~~~~~
compilation terminated.
error: command 'arm-linux-gnueabihf-gcc' failed with exit status 1

  ----------------------------------------

Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-install-A6lrAK/cffi/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-record-049HJN/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-lN17zy --compile" failed with error code 1 in /tmp/pip-install-A6lrAK/cffi/


Command "/usr/bin/python -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-lN17zy --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --extra-index-url https://www.piwheels.org/simple -- setuptools>=40.6.0 wheel "cffi>=1.8,!=1.11.3; platform_python_implementation != 'PyPy'"" failed with error code 1 in None`

Desktop (please complete the following information):

  • OS: Raspbian 10
  • Python 3.7.3

[BUG] Firefox search bar just redirects me to the Whoogle homepage

Describe the bug
When I search on my self-hosted Whoogle using Firefox bar the results is the homepage of my Whoogle instance rather than actual result page.

To Reproduce
Steps to reproduce the behavior:

  1. Add Whoogle to your search engine by clicking on the three dots in Firefox address bar
  2. Search something with the new added engine
  3. See error

Expected behavior
I expect to see the actual results of my query, not the Whoogle homepage.

Desktop (please complete the following information):

  • OS: Windows 10
  • Browser Firefox (both stable, nightly and beta)

Additional context
On my mobile phone everything works as expected: it seems the destkop version forgets the %s.

[QUESTION] Can Whoogle be deployed with ______?

Probably. Here are some currently requested deployment options. Support for these is TBD.

  • Vercel/Zeit
  • YunoHost App
  • Netlify

If you have another request, or want to contribute a quick guide on how to deploy to one of these, please comment here.

Add Safesearch options

Describe the feature you'd like to see added
Add the ability to set Safe Search options. An added bonus would be to force the option at the server level to hide it from the user

Describe which parts of the project this would modify (front end/back end/configuration/etc)
The UI and the required back end / config elements.
Additional context

image

[BUG] Cannot set search to 'any language'

Describe the bug
After setting the language in the config to German the results are always in German, I cannot go back to 'any language', it just flips back to German.

To Reproduce
Steps to reproduce the behavior:

  1. Search
  2. Set langauge to any language
  3. See that it's still set to German

Deployment Method

  • Heroku (one-click deploy)
  • Docker
  • run executable
  • pip/pipx
  • Other: [describe setup]

Version of Whoogle Search

  • Latest build from [source] (i.e. GitHub, Docker Hub, pip, etc)
  • Version [version number]
  • Not sure

Desktop (please complete the following information):

  • OS: MacOS
  • Browser: Chrome
  • Version: 83

[BUG] Installation fails when using "D) Manual" on Linux Mint 19.3 Cinnamon

My system is running on Linux Mint 19.3 Cinnamon. I would like to install Whoogle using the method "D) Manual" as described in the README.md.

These steps have worked:

git clone https://github.com/benbusby/whoogle-search.git
cd whoogle-search
python3 -m venv venv
source venv/bin/activate

But when I run the next step pip install -r requirements.txt I get this error:

$ pip install -r requirements.txt 
Collecting beautifulsoup4==4.8.2 (from -r requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/cb/a1/c698cf319e9cfed6b17376281bd0efc6bfc8465698f54170ef60a485ab5d/beautifulsoup4-4.8.2-py3-none-any.whl (106kB)
    100% |████████████████████████████████| 112kB 1.9MB/s 
Collecting bs4==0.0.1 (from -r requirements.txt (line 2))
  Downloading https://files.pythonhosted.org/packages/10/ed/7e8b97591f6f456174139ec089c769f89a94a1a4025fe967691de971f314/bs4-0.0.1.tar.gz
Collecting cffi==1.13.2 (from -r requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/49/72/0d42f94fe94afa8030350c26e9d787219f3f008ec9bf6b86c66532b29236/cffi-1.13.2-cp36-cp36m-manylinux1_x86_64.whl (397kB)
    100% |████████████████████████████████| 399kB 2.4MB/s 
Collecting Click==7.0 (from -r requirements.txt (line 4))
  Downloading https://files.pythonhosted.org/packages/fa/37/45185cb5abbc30d7257104c434fe0b07e5a195a6847506c074527aa599ec/Click-7.0-py2.py3-none-any.whl (81kB)
    100% |████████████████████████████████| 81kB 4.5MB/s 
Collecting cryptography==2.8 (from -r requirements.txt (line 5))
  Downloading https://files.pythonhosted.org/packages/45/73/d18a8884de8bffdcda475728008b5b13be7fbef40a2acc81a0d5d524175d/cryptography-2.8-cp34-abi3-manylinux1_x86_64.whl (2.3MB)
    100% |████████████████████████████████| 2.3MB 763kB/s 
Collecting Flask==1.1.1 (from -r requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/9b/93/628509b8d5dc749656a9641f4caf13540e2cdec85276964ff8f43bbb1d3b/Flask-1.1.1-py2.py3-none-any.whl (94kB)
    100% |████████████████████████████████| 102kB 7.4MB/s 
Collecting itsdangerous==1.1.0 (from -r requirements.txt (line 7))
  Downloading https://files.pythonhosted.org/packages/76/ae/44b03b253d6fade317f32c24d100b3b35c2239807046a4c953c7b89fa49e/itsdangerous-1.1.0-py2.py3-none-any.whl
Collecting Jinja2==2.10.3 (from -r requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/65/e0/eb35e762802015cab1ccee04e8a277b03f1d8e53da3ec3106882ec42558b/Jinja2-2.10.3-py2.py3-none-any.whl (125kB)
    100% |████████████████████████████████| 133kB 5.4MB/s 
Collecting MarkupSafe==1.1.1 (from -r requirements.txt (line 9))
  Downloading https://files.pythonhosted.org/packages/b2/5f/23e0023be6bb885d00ffbefad2942bc51a620328ee910f64abe5a8d18dd1/MarkupSafe-1.1.1-cp36-cp36m-manylinux1_x86_64.whl
Collecting pycparser==2.19 (from -r requirements.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/68/9e/49196946aee219aead1290e00d1e7fdeab8567783e83e1b9ab5585e6206a/pycparser-2.19.tar.gz (158kB)
    100% |████████████████████████████████| 163kB 6.4MB/s 
Collecting pycurl==7.43.0.4 (from -r requirements.txt (line 11))
  Downloading https://files.pythonhosted.org/packages/0f/52/2c6951c4cd8a4e9288f2561eb9da1dc15b5d0f4e610a7e2acf39bc703281/pycurl-7.43.0.4.tar.gz (215kB)
    100% |████████████████████████████████| 225kB 4.7MB/s 
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "/tmp/pip-build-s8wym11l/pycurl/setup.py", line 234, in configure_unix
        stdout=subprocess.PIPE, stderr=subprocess.PIPE)
      File "/usr/lib/python3.6/subprocess.py", line 729, in __init__
        restore_signals, start_new_session)
      File "/usr/lib/python3.6/subprocess.py", line 1364, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'curl-config': 'curl-config'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-s8wym11l/pycurl/setup.py", line 957, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-build-s8wym11l/pycurl/setup.py", line 618, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-build-s8wym11l/pycurl/setup.py", line 101, in __init__
        self.configure()
      File "/tmp/pip-build-s8wym11l/pycurl/setup.py", line 238, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory: 'curl-config': 'curl-config'

Can you help me?

please make app.json & easy to deploy button with buildpack requirement for heroku

Hi Ben ,
1 - please make app.json & easy to deploy button on your repo with buildpack requirement for heroku
2 - on another cloud container , must be set chmod on run & dockerfile file's to can exutable , but my os is win , please chmod & run 2 file & re-push on your repository :
3 - port 500 is point to 80 for run ?
log of my failed deploy :
Imgur
crash problem :
A - Previous status: ContainerCannotRun
B - OCI runtime create failed: container_linux.go:348: starting container process caused "exec: "./run": permission denied": unknown
WAITING FOR YOUR RESPONSE
fine regards
goodbye

[BUG] Save config http only

Describe the bug
I've installed this both in an AWS free instance manually and also using your heroku quick deploy
At first, in the case of AWS, saving the config settings worked because I'd opened port 80 to get ssl certs with certbot. Once I closed port 80 again the config changes won't save . No biggie I can open port 80 ! My nginx settings redirect back to https anyway.
However with heroku I start with a https page eg https://mywhoogle.herokuapp.com. Saving the config settings take me to the http page http://mywhoogle.herokuapp.com where it will stay for any subsequent searches

To Reproduce
Steps to reproduce the behavior: AWS

  1. Only allow traffic on port 443 https
  2. Go to https://mywhoogle.domain.com
  3. Edit settings and click save
  4. Page hangs

Steps to reproduce the behavior: Heroku

  1. https://mywhoogle.herokuapp.com
  2. Edit settings and click save
  3. Page is now http://mywhoogle.herokuapp.com

Expected behavior
AWS: allow save via https
heroku: remain on https after save if that's what we we on before saving.

Nice work :)

Fix docker repo at ReadMe.md line 136

docker run --publish 5000:5000 --detach --name whoogle-search whoogle-search:latest doesn't work.

You should add the full image name docker run --publish 5000:5000 --detach --name whoogle-search benbusby/whoogle-search:latest

I couldn't do a PR.

[BUG] High RAM usage

First off, I'm really excited about this project! I've been looking for something more approachable than searx for quite awhile, and was really happy to see you publish this. It seems like a great project, and I'm looking forward to using it. That said...

Describe the bug
Whoogle seems to use a lot of RAM, even before a single request has reached the application.

To Reproduce
Steps to reproduce the behavior:

  1. docker run --name=whoogle-search --init --rm benbusby/whoogle-search
  2. docker stats --no-stream whoogle-search
  3. Observe MEM USAGE / LIMIT column

Expected behavior
Essentially, less RAM usage.

Desktop (please complete the following information):
I'm testing this in a VM on my laptop, using Vagrant with a VirtualBox provider.

  • OS: macOS 10.15.4
  • Vagrant: 2.2.9
  • VirtualBox: 6.0.14r133895

Additional context
I'm seeing 264MB used by whoogle in my local VM cluster. I tried to deploy this to a t3a.nano in EC2 (456MB RAM), but triggered OOM killer (the docker daemon uses a lot of RAM on its own, which leaves that machine with about 250MB of RAM free). Unfortunately, this is the RAM usage at-rest. I haven't even directed a request at this application yet, and it's already allocated over a quarter of a gigabyte.

For context, most PHP applications run with a default memory limit of 64MB, and many will operate just fine with a limit of 32MB (for example, bookstack, which I'm operating right now with 23MB of RAM usage). RAM is sparse in VMs (both locally and in the cloud), so anything you can do to lower the RAM requirements of the application will make this project more viable for a wider variety of users.

Publish to dockerhub

Please publish and tag a build of the docker image to dockerhub so applications like unraid can take advantage of whoogle-search.

Configurable config.json location

Describe the feature you'd like to see added
static/config.json is used to set the main configuration settings for whoogle. However when the docker container is rebuilt these settings are lost.

Please modify the location of config.json so that it is easier for docker users to provide a volume-mounted copy to the container so we can persist our settings between rebuilds.

Describe which parts of the project this would modify (front end/back end/configuration/etc)

Minimal changes to achieve this

Move config.json into its own folder outside of static, preferably to a location that does not need other files to be written to it. Additionally make sure whoogle does not overwrite this directory/file if it already exists at setup.

With a hardcoded, but separate, directory a docker user can easily volume-mount a folder to that location and provide their own config.json.

Optionally

Expose the configuration directory string as an environmental variable in the Dockerfile so users can set their own directory.

[BUG] Searching results in page stating "Our systems have detected unusual traffic from your computer network."

Describe the bug
Searching results in the following message:

About this page

Our systems have detected unusual traffic from your computer network. This page checks to see if it's really you sending the requests, and not a robot. Why did this happen?

IP address: xxx.xxx.xxx.xxx
Time: 2020-05-13T18:07:51Z
URL: https://www.google.com/search?gbv=1&q=namemesh

To Reproduce
Steps to reproduce the behavior:

  1. Search (I searched about 5 times this morning)

Expected behavior
Google results.

Desktop (please complete the following information):

  • OS: Windows 10
  • Browser Firefox
  • Version latest

Additional info:
Going to Google.com proper and searching "namemesh" provides expected results page.

Whoogle is deployed via Docker on my home network (on a QNAP); the Whoogle search and Google search originate from the same WAN IP, though different LAN IP. So it does not appear to be a simple flag against my WAN IP as my desktop PC can make valid google.com searches.

To be clear, this error occurs with any search query now, not just "namemesh".

[BUG] UAE Search Results (Docker)

When searching using the docker image, the results appear to be targeted towards the UAE.

The weather results are correct and says the same location to that set in the configuration. The news along with other results appear to be the UAE versions.

Example 1:

  1. Search "apple"
  2. The first result is "https://www.apple.com/ae/"

Example 2:

  1. Search "amazon"
  2. The first result is "https://www.amazon.ae/"

This is running on a Hetzner VPS, located in Germany. With the location set within the config being in the UK.

This is using the latest docker image (although it also happened with previous ones).
Digest: 688ab5d1a265

Select country and language

Describe the feature you'd like to see added
an option to select the country and language

Describe which parts of the project this would modify (front end/back end/configuration/etc)
Not completely sure what will be required, but assuming both front and back end changes will be needed.

Additional context
quite basic.

[Question] Two questions about Whoogle: Privacy and a public instance

Hi @benbusby,

Thank you for creating Whoogle! It's really cool and I like to use it.

I have two questions about Whoogle regarding privacy and the availability of a public instance.

1. Privacy

I was wondering how you get the data from the Google search results page to my local machine. As far as I understand this line in request.py sends a request to the Google servers.

So when I search for something, Google can still see the metadata of my search, e.g. my search terms, my IP address, my request time, etc., right?

2. Public instance

As far as I know there is currently no public instance of Whoogle. In the README.md it also says this:
"Whoogle is intended to only ever be deployed to private instances [...]".

What's the reason for this?

Maybe you can operate a public instance so people can get a first glimpse of Whoogle.

Thanks

Add official image to Docker Hub

Describe the feature you'd like to see added
It would be wonderful to be able to have this as a prebuilt image on dockerhub. Using their free build tools it should just be a matter of connecting the repo and being sure that tags are appropriately pushed.

Is this something y'all could add?

[FEATURE] Theme/color scheme/UI customization

Describe the feature you'd like to see added
It would be nice to have the chance to edit the Logo/colors in homepage and hide the configuration option.
Basically if I run it on my server I'd like an easy way to give my search engine a nicer looking look and hide the "Configuration" line beneath it so if I share that URL people can search and not edit the options.

Describe which parts of the project this would modify (front end/back end/configuration/etc)
Mainly frontend, but maybe this can be integrated with #43

[QUESTION] How to update when deployed via Heroku Quick Deploy?

It's awesome how easy (and cheap) it is to deploy it to heroku. ❤️
And it works really great since a week.
But now i saw nice changes like https-only and so on and i tried to find out how i update a heroku app.
Found https://blog.heroku.com/six-strategies-deploy-to-heroku where he says:

Cons: Apps deployed via button do not auto-update when new commits are added to the GitHub repo from which it was deployed

Could someone help me here please?
Is heroku cli the way to go? Or can i link my github account to the app for automated updates?

[BUG] Does not work with Firefox Container Tabs (Always open in custom tab)

Describe the bug
In Firefox container tabs, if we enable woogle to always open in a custom container tab. When we set woogle as default search and search through the address bar, the search target is not propogated and just the url localhost:8888/search which redirects to localhost:8888 and no search results are shown.

To Reproduce
Steps to reproduce the behavior:

  1. Open Firefox and install firefox container tabs.
  2. Open a new container in any of the default container or create you own container.
  3. Click on the Container tab extension and select always open in selected container.
  4. Make woogle your default search.
  5. Create a new tab (normal or any container other that the one we set as default for woogle) and search for something.
  6. No Search results are shown.

Expected behavior
A new container tab was opened with the search results.

Additional context
This extension does the containerised isolation of google urls. Can we have something similar for whoogle.

Autocomplete - would be nice [ 2nd edition ]

As mentioned by user @gripped
The following API works really well in terms of autocomplete and I haven't found any limits to it YET.

https://github.com/asciimoo/searx/blob/master/searx/autocomplete.py

Might be worth a look ;) Doesn't look that complex but then I can't program.

Here would be an example of implementing the autocomplete.

    import requests, json
    URL="http://suggestqueries.google.com/complete/search?client=firefox&q=Hel Wor"
    headers = {'User-agent':'Mozilla/5.0'}
    response = requests.get(URL, headers=headers)
    result = json.loads(response.content.decode('utf-8'))
    print(result)

This is from Quora by: Michael Staniek Thank you!

There is no auth key or token, should work pretty well I will try to spam it for a little bit to see if there are limits.

Autocomplete - would be nice.

Autocomplete - It would be nice. I'm not sure that it is possible to pass queries like that as well as completes but would be a great feature to have.

Login page for Whoogle

Describe the feature you'd like to see added
Login to Whoogle.

Describe which parts of the project this would modify (front end/back end/configuration/etc)
Probably both?

Additional context
Would it be possible to add a login page to Whoogle?
This way, when my Whoogle server is exposed to the internet it won't be hammered.
You will need to first login before you can use Whoogle.

[BUG] Whoogle crashes upon startup in Docker

Describe the bug
Whoogle crashes immediately in Docker after running the setup commands.

To Reproduce
Steps to reproduce the behavior:

  1. Enter in your terminal:
git clone https://github.com/benbusby/whoogle-search.git
cd whoogle-search
docker build --tag whooglesearch:1.0 .
docker run --publish 8888:5000 --detach --name whooglesearch whooglesearch:1.0
  1. Type docker ps -a and see:
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS                       PORTS                                      NAMES
d05696f259ef        whooglesearch:1.0                        "./whoogle-search"       21 seconds ago      Exited (137) 2 seconds ago                                              whooglesearch
  1. View logs at docker logs whooglesearch and see error:
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
./whoogle-search: line 27:    10 Killed                  flask run --host="0.0.0.0" --port=$PORT

Expected behavior
Whoogle should be able to start in Docker without crashing.

Screenshots
N/A

Environment:

Additional context
N/A

Add config option to search via GET request only

Describe the bug
In Firefox container tabs, if we enable whoogle to always open in a custom container tab. When we set whoogle as default search and search through the address bar, the search target is not propagated and just the url localhost:8888/search which redirects to localhost:8888 and no search results are shown.

To Reproduce
Steps to reproduce the behavior:

  1. Open Firefox and install firefox container tabs.
  2. Open a new tab in any of the default container or create your own container and open a tab in that.
  3. Open whoogle url, Click on the Container tab extension and select always open in selected container.
  4. Make whoogle your default search.
  5. Create a new tab (normal or any container other that the one we set as default for whoogle) and search for something.
  6. No Search results are shown.

Expected behavior
A new container tab was opened with the search results.

Additional context
This extension does the containerised isolation of google urls. Can we have something similar for whoogle.

Add config setting for language

Hi, thank's for this awsome project.

Currently, the language or the results will be depending in the country the server is.

We should be able to change the language with the tag &hl=

[FEATURE] Change between languages from results page

Describe the feature you'd like to see added
Brought up in #73, where a user brought up needing the ability to switch between results languages from the results page instead of the home page.
Additional context
The implementation should be similar to the built in Google UI for switching between a specified language and “all languages”.

Unable to deploy on render website

Deploy failed on Render with exit status 1.
please check the issue.

Here are the some last lines of deploy logs:

INFO[0098] ARG config_dir=/config
INFO[0098] RUN mkdir $config_dir
mkdir: cannot create directory ‘/config’: File exists
error building image: error building stage: failed to execute command: waiting for process to exit: exit status 1
error: exit status 1

the earlier 5-6 days ago build code got deployed without any issues.

Tor support

Describe the feature you'd like to see added
Whoogle Search should allow conditional support for Tor routing, to add an extra option for further anonymity.

Describe which parts of the project this would modify (front end/back end/configuration/etc)
Configuration would need to be updated with a key/value pairing for "tor": to enable/disable routing per request.

Back end modification of the request class would be needed to use proxy if the config value returns True. Example code from prior conversation:

crl.setopt(crl.PROXY, "123.123.123.123")
crl.setopt(crl.PROXYPORT, 9050)
crl.setopt(crl.PROXYTYPE, pycurl.PROXYTYPE_SOCKS5)

Front end needs update to expose Tor routing config option

Docker files will need to be updated to install Tor package and run any necessary setup commands as well.

Additional context
Would also need to update the javascript filter to allow captchas through on the "unusual traffic" page, since requests through Tor generally encounter this result from Google.

Target new tab

Describe the feature you'd like to see added
Is it possible to add a toggle to target a new tab for the links ie. target="_blank"?

Describe which parts of the project this would modify (front end/back end/configuration/etc)
This would be config and front-end.

Additional context
I keep losing my search tab when clicking links and then having to redo the search. Rinse and repeat.

[BUG] Whoogle cannot be installed with pip

Describe the bug
By typing:

pip install whoogle-search

I get this stack trace:

    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4zn4xd73/whoogle-search/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4zn4xd73/whoogle-search/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-8iry6ov9
         cwd: /tmp/pip-install-4zn4xd73/whoogle-search/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-4zn4xd73/whoogle-search/setup.py", line 5, in <module>
        requirements = list(open('requirements.txt', 'r'))
    FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I have tried many options, such as:

pip3 install whoogle-search -r requirements.txt
pip3 install whoogle-search -r "requirements.txt"
pip3 install whoogle-search -r /root/requirements.txt --no-cache-dir

None of which worked.
Running as root, or as any other user, does not make any difference.

(the requirements.txt file is set to 777 permissions. Pip shouldn't even need a requirements.txt file!)

System Info
Ubuntu 20.04 with Python3.8.2

Expose flask port in Dockerfile

The port of flask should be exposed in the Dockerfile, so reverse proxies like Traefik are able to find the correct port automatically.

[BUG] Firefox autofocus not working properly

Describe the bug
Autofocus of the search field on the main page was added in #16, but didn't seem to be working properly on Firefox (version 76, macOS).

To Reproduce

  1. Go to the main page
  2. Check to see if typing in the search bar is instantly accessible

Expected behavior
Search field should be focused on page load

Desktop (please complete the following information):

  • OS: macOS
  • Browser: Firefox
  • Version 76

[QUESTION] How do I run Whoogle behind Traefik?

I am trying to get Whoogle behind the Traefik Reverse Proxy, but unable to do so. The certificate has already been issued successfully.

Here are the labels I've specified for Whoogle:

  - "traefik.enable=true"
  - "traefik.http.routers.search.tls=true"
  - "traefik.http.routers.search.rule=Host(`search.example.com`)"
  - "traefik.http.services.search.loadbalancer.server.port=5000"
  - "traefik.http.routers.search.service=search"
  - "traefik.http.routers.search.entrypoints=websecure"
  - "traefik.http.routers.search.tls.certresolver=myresolver"
  - "traefik.http.services.search.loadbalancer.server.scheme=https"
  - "traefik.http.middlewares.my-redirect.redirectscheme.scheme=https"
  - "traefik.http.middlewares.my-redirect.redirectscheme.permanent=true"

[FEATURE] Time-based results filter

This is such a great application and I'm loving every search with it. DuckDuckGo just wasn't doing it for me, and this does. The only feature I miss from Google is the ability to filter results based on time (example image). Would love this addition!

HTTP, SOCKS proxy, Tor support

Hi,

first of all - great project!

Any chance to add a configurable HTTP, SOCKS proxy support for the requests made towards Google?

Something like this:

  if proxy:
            if proxy["type"] == "socks4":
                self.c.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_SOCKS4)
            elif proxy["type"] == "socks5":
                self.c.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_SOCKS5)
            else:
                self.c.setopt(pycurl.PROXYTYPE, pycurl.PROXYTYPE_HTTP)

            self.c.setopt(pycurl.PROXY, str(proxy["address"]))
            self.c.setopt(pycurl.PROXYPORT, proxy["port"])

            if proxy["username"]:
                self.c.setopt(
                    pycurl.PROXYUSERPWD,
                    "{0}:{1}".format(proxy["username"], proxy["password"])
                )

This way the search could be even more private.

Thanks!

"I'm Feeling Lucky" with shortcut

Describe the feature you'd like to see added
If you put an exclamation point at the beginning of your search, Whoogle would get the first result and redirect you to that page securely and privately.

Describe which parts of the project this would modify (front end/back end/configuration/etc)
When receiving the search query, check the beginning of the string for ! . If it's there, remove it from the search query and search Google. When processing the results, do a redirect to the first one instead of showing the search results.

Additional context
DuckDuckGo has the concept of Bangs which are shortcuts to specific search results. The most useful is the I'm Feeling Lucky, which is just ! . In the future Whoogle could have other ones, or the ability to configure your own.

Use a two step Docker build

The requirements.txt should be copied in a seperate step, so the dependencies do not have to be dowloaded every time the docker image is being build.

[BUG] docker container won't do anything, no logs after branch 27 merge

Describe the bug
A clear and concise description of what the bug is.
Just installed it and hour ago, was working fine except results in wrong language. I saw pull request 27, then saw it was merged, so I deleted the container and git clone folder and started over. The new version built but when it starts, nothing happens. Docker logs says "No logs."
To Reproduce
Steps to reproduce the behavior:
Follow docker steps in readme

Expected behavior
Whoogle server listening on 8888 and allowing search

Desktop (please complete the following information):

  • OS: docker on Ubuntu 18.04
  • Browser chrome android

[FEATURE] Ability to disable IPv6 or enable JS when Bot Check

Describe the feature you'd like to see added

I am using a hosted VPS, which has both IPv4 and IPv6 enabled.
However, when I run whoogle-search, and try searching anything with it, I get a google spam-bot popup, saying that there's unusual traffic coming from my IP (and I see my VPS' IPv6 IP).
I am however unable to go past this screen, since Whoogle automatically disables JavaScript.

However, if I disable IPv6 system-wide, and restart whoogle, I can succesfully browse the web.

This is probably related to my own VPS hosting provider, doing nasty stuff with shared IPv6s, thus a feature to disable IPv6 (either via command line arg or via the "options" button in the homepage) would add a really nice touch, since not everyone has a working or usable (although enabled) IPv6 address.

Another solution, related to this case scenario, would be to detect if the current page is the spam bot check, and eventually enable the JS captcha script.

Firebase Deploy

Thanks sir for you great project !
i really love it !

can you provide a way to deploy to firebase sir ?

and also when i searching gmail and click it it nothing happen

thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.