jarun / googler Goto Github PK

Bug reports

I tried to use googler -d to watch what happened but nothing show on screen

• Output of googler -d

[proxychains] config file found: /etc/proxychains.conf
[proxychains] preloading /usr/lib/libproxychains4.so
[proxychains] DLL init: proxychains-ng 4.11
[proxychains] DLL init: proxychains-ng 4.11
[DEBUG] Base URL [/search?ie=UTF-8&oe=UTF-8&start=0&]
[DEBUG] Search URL [www.google.com : /search?ie=UTF-8&oe=UTF-8&start=0&q=mutt+tutorial]
[proxychains] Dynamic chain  ...  127.0.0.1:1080  ...  www.google.com:443  ...  OK
[DEBUG] Response body written to '/tmp/googler-response-lmyvnq9j'.

• Response body, something like /tmp/googler-response-xxxxxx
  There is a lot of things in googler-response-xxxxxx
• Details of operating system, Python version used, terminal emulator and shell
  My Python version: 3.5.1
  OS: Arch linux x86_64 4.4.5-1-ARCH
  Terminal: Urxvt v9.22
  Shell: zsh
  local: zh_CN.urf8.

Also I tested with tsocks with googler, that's ok, I can use number of search result to open page at my chromium.

Please,Thinks!

1. Google News doesn't work
2. File mime type doesn't show
3. A regular hello search results:

$ ./googler hello

1 HELLO! Online: celebrity & royal news, magazine, babies, weddings ...
/url?q=http://www.hellomagazine.com/&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFghoMAw&usg=AFQjCNHe-OhTvjOvC5smGVTvd1mxsfl2FA
HELLO! Online brings you the latest celebrity & royal news from the UK & around  the world, magazine exclusives,
celeb babies, weddings, pregnancies and ...

2 hello network
/url?q=https://hello.com/&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFghuMA0&usg=AFQjCNHu3ySM5MQ_ySXDktfr8Ax4fFxy4w
hello connects you with people and content around your passions. Show the  world who you are, express what you
love, and create meaningful connections.

3 Hello (Adele song) - Wikipedia, the free encyclopedia
/url?q=https://en.wikipedia.org/wiki/Hello_(Adele_song)&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgh0MA4&usg=AFQjCNHJRKExeuRwwdFmAKkJ6dH7qv_TeQ
"Hello" is a song by English singer Adele. It was released on 23 October 2015 by  XL Recordings as the lead
single from her third studio album, 25 (2015). Adele ...

4 Hello - Wikipedia, the free encyclopedia
/url?q=https://en.wikipedia.org/wiki/Hello&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgh6MA8&usg=AFQjCNEPQPRz_g4ms5UWaM9OQrZ8-qWzew
Hello is a salutation or greeting in the English language. It is first attested in  writing from 1833. Contents.
[hide]. 1 First use; 2 Etymology. 2.1 Telephone; 2.2 ...

5 Images for hello
/search?q=hello&ie=UTF-8&oe=UTF-8&prmd=ivnsfdm&tbm=isch&tbo=u&source=univ&sa=X&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQsAQIgAE

6 HELLO! - Facebook
/url?q=https://www.facebook.com/hello/&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgiMATAV&usg=AFQjCNFX6GnLlw8l4pE7PM-eVFUijQimIg
HELLO! 1794375 likes · 54963 talking about this. The official Facebook page for  HELLO! magazine &
http://www.hellomagazine.com/ FOLLOW US @...

7 ADELE LYRICS - Hello - A-Z Lyrics
/url?q=http://www.azlyrics.com/lyrics/adele/hello.html&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgiSATAW&usg=AFQjCNHGIyMSO2JIfxsiYutLKdH21BVmvw
Hello, it's me. I was wondering if after all these years you'd like to meet. To go  over everything. They say
that time's supposed to heal ya. But I ain't done much ...

8 Hello
/url?q=https://hello.is/&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgiXATAX&usg=AFQjCNGwdUWTtCdsDuH8FLfNW3oDxaTyqQ
Sense by Hello wakes you up feeling great, improves your sleep, and helps you  learn how the environment of your
bedroom affects your rest.

9 Hello — Caller ID & Blocking - Android Apps on Google Play
/url?q=https://play.google.com/store/apps/details%3Fid%3Dcom.facebook.phone%26hl%3Den&sa=U&ved=0ahUKEwjQw7PF3Z_NAhUQSI8KHfVNDTMQFgidATAY&usg=AFQjCNHcLO9dQ-fNU0fT-KM6O5TW50jggA
Make your phone smarter with Hello. Built by Messenger just for Android, Hello  combines info from Facebook with
the contact info on your phone so it's easy to ...

a. URLs are malformed
b. multiple spaces in results: Show the world, around the world, To go over
c. Images for hello show in results

Can you please change the code to detect the proxy settings

Issue 1: googler exits if p is pressed at first page
Expected: It shouldn't exit and we shouldn't point it as an ERROR. Just Already at the first page. should do.

And why are we raising an exception for this? Just printing the message and maintaining start at 0 should do.

$ googler -d bohoomil infinality deb package 16.04 ubuntu -t m3
[DEBUG] Connecting to new host www.google.com
[DEBUG] Fetching URL /search?ie=UTF-8&num=10&oe=UTF-8&q=bohoomil+infinality+deb+package+16.04+ubuntu&start=0&tbs=qdr:m3
[DEBUG] Redirecting to URL https://www.google.co.in/search?ie=UTF-8&num=10&oe=UTF-8&q=bohoomil+infinality+deb+package+16.04+ubuntu&start=0&tbs=qdr:m3&gws_rd=cr&ei=htlYV7T0PIP_vgTampT4AQ
[DEBUG] Connecting to new host www.google.co.in
[DEBUG] Fetching URL /search?ie=UTF-8&num=10&oe=UTF-8&q=bohoomil+infinality+deb+package+16.04+ubuntu&start=0&tbs=qdr:m3&gws_rd=cr&ei=htlYV7T0PIP_vgTampT4AQ
[DEBUG] Response body written to '/tmp/googler-response-tpi1kded'.

1 Ubuntu 16.04 support · Issue #25 · achaphiv/ppa-fonts · GitHub
https://github.com/achaphiv/ppa-fonts/issues/25
Apr 21, 2016 - Please build both freetype and fontconfig-infinality for ubuntu 16.04. thanks. ... It looks like
bohoomil's patch for freetype 2.6.1 is no longer applying. Might be simpler ... Thank you for your work despite
hating debian packaging :). @greatquux.

2 Install fonts and improve font rendering quality in Arch Linux
http://www.ostechnix.com/install-fonts-improve-font-rendering-quality-arch-linux/
May 6, 2016 - infinality. For the past two years, I have been using Ubuntu as my primary OS, but now it's Arch
Linux with Deepin Desktop ... [infinality-bundle] Server = http://bohoomil.com/repo/$arch ... The package name
will be given in the left side for each language set. ... Install Nginx, MariaDB, PHP (LEMP Stack) in Ubuntu
16.04 LTS.

3 Shell Github Star Ranking at 2016/05/06 - Main
https://38elements.github.io/2016/05/06/shell-repository-github-star-ranking.html
May 6, 2016 - Nyr/openvpn-install 1677 OpenVPN road warrior installer for Debian, Ubuntu and ..... for the
terminal. bohoomil/fontconfig-ultimate 330 freetype2-infinality run-time ... IPsec/L2TP VPN server on Ubuntu
16.04/14.04/12.04, Debian 8 and CentOS ...

4 Después de instalar openSUSE Leap 42.1 KDE | La sombra del ...
http://lasombradelhelicoptero.com/despues-de-instalar-opensuse-leap-42-1-kde/
May 11, 2016 - ... que asegura incorporar los parches del famoso usuario Bohoomil de Arch Linux. ... Infinality
openSUSE ... Si no están instaladas, como las fuentes de Ubuntu, por ejemplo, las encontraréis en la web de
software de openSUSE. ..... Synchronizing package databases… .... Ubuntu 16.04 Xenial Xerus: un futuro
prometedor.

googler (? for help) p
Traceback (most recent call last):
  File "/usr/local/bin/googler", line 1859, in <module>
    main()
  File "/usr/local/bin/googler", line 1848, in main
    repl.cmdloop()
  File "/usr/local/bin/googler", line 1643, in cmdloop
    self.do_previous('')
  File "/usr/local/bin/googler", line 1378, in enforced_method
    method(self, *args, **kwargs)
  File "/usr/local/bin/googler", line 1392, in enforced_method
    method(self)
  File "/usr/local/bin/googler", line 1608, in do_previous
    self._google_url.prev_page()
  File "/usr/local/bin/googler", line 347, in prev_page
    raise ValueError('Already at the first page.')
ValueError: Already at the first page.

Issue 2: Redundant newline and WARNING message when no more results

Expected: No warning in an informative message please. Just No results. should do. And we need to remove the extra newline above the message.

$ googler -d bohoomil infinality deb package 16.04 ubuntu -t m3
[DEBUG] Connecting to new host www.google.com
[DEBUG] Fetching URL /search?ie=UTF-8&num=10&oe=UTF-8&q=bohoomil+infinality+deb+package+16.04+ubuntu&start=0&tbs=qdr:m3
[DEBUG] Redirecting to URL https://www.google.co.in/search?ie=UTF-8&num=10&oe=UTF-8&q=bohoomil+infinality+deb+package+16.04+ubuntu&start=0&tbs=qdr:m3&gws_rd=cr&ei=jdpYV6yYO8nqvAT4y4-YDg
[DEBUG] Connecting to new host www.google.co.in
[DEBUG] Fetching URL /search?ie=UTF-8&num=10&oe=UTF-8&q=bohoomil+infinality+deb+package+16.04+ubuntu&start=0&tbs=qdr:m3&gws_rd=cr&ei=jdpYV6yYO8nqvAT4y4-YDg
[DEBUG] Response body written to '/tmp/googler-response-l5dsr7is'.

1 Ubuntu 16.04 support · Issue #25 · achaphiv/ppa-fonts · GitHub
https://github.com/achaphiv/ppa-fonts/issues/25
Apr 21, 2016 - Please build both freetype and fontconfig-infinality for ubuntu 16.04. thanks. ... It looks like
bohoomil's patch for freetype 2.6.1 is no longer applying. Might be simpler ... Thank you for your work despite
hating debian packaging :). @greatquux.

2 Install fonts and improve font rendering quality in Arch Linux
http://www.ostechnix.com/install-fonts-improve-font-rendering-quality-arch-linux/
May 6, 2016 - infinality. For the past two years, I have been using Ubuntu as my primary OS, but now it's Arch
Linux with Deepin Desktop ... [infinality-bundle] Server = http://bohoomil.com/repo/$arch ... The package name
will be given in the left side for each language set. ... Install Nginx, MariaDB, PHP (LEMP Stack) in Ubuntu
16.04 LTS.

3 Shell Github Star Ranking at 2016/05/06 - Main
https://38elements.github.io/2016/05/06/shell-repository-github-star-ranking.html
May 6, 2016 - Nyr/openvpn-install 1677 OpenVPN road warrior installer for Debian, Ubuntu and ..... for the
terminal. bohoomil/fontconfig-ultimate 330 freetype2-infinality run-time ... IPsec/L2TP VPN server on Ubuntu
16.04/14.04/12.04, Debian 8 and CentOS ...

4 Después de instalar openSUSE Leap 42.1 KDE | La sombra del ...
http://lasombradelhelicoptero.com/despues-de-instalar-opensuse-leap-42-1-kde/
May 11, 2016 - ... que asegura incorporar los parches del famoso usuario Bohoomil de Arch Linux. ... Infinality
openSUSE ... Si no están instaladas, como las fuentes de Ubuntu, por ejemplo, las encontraréis en la web de
software de openSUSE. ..... Synchronizing package databases… .... Ubuntu 16.04 Xenial Xerus: un futuro
prometedor.

googler (? for help) n
[DEBUG] Fetching URL /search?ie=UTF-8&num=10&oe=UTF-8&q=bohoomil+infinality+deb+package+16.04+ubuntu&start=10&tbs=qdr:m3
[DEBUG] Response body written to '/tmp/googler-response-7j__n2jy'.

[WARNING] No results.
googler (? for help)

OS X 10.11.5, Python 3.5.1, googler 2.5:

> /usr/local/Cellar/googler/2.5/bin/googler --debug --noprompt Homebrew
[DEBUG] Connecting to new host www.google.com
Traceback (most recent call last):
  File "/usr/local/Cellar/googler/2.5/bin/googler", line 579, in new_connection
    self._conn.connect()
  File "/usr/local/Cellar/googler/2.5/bin/googler", line 122, in connect
    sock.setsockopt(socket.SOL_TCP, socket.TCP_DEFER_ACCEPT, 1)
AttributeError: module 'socket' has no attribute 'TCP_DEFER_ACCEPT'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/googler/2.5/bin/googler", line 1908, in <module>
    main()
  File "/usr/local/Cellar/googler/2.5/bin/googler", line 1882, in main
    repl = GooglerCmd(opts)
  File "/usr/local/Cellar/googler/2.5/bin/googler", line 1471, in __init__
    self._conn = GoogleConnection(self._google_url.hostname, proxy=proxy)
  File "/usr/local/Cellar/googler/2.5/bin/googler", line 530, in __init__
    self.new_connection(host, port=port, timeout=timeout)
  File "/usr/local/Cellar/googler/2.5/bin/googler", line 582, in new_connection
    raise GoogleConnectionError(msg)
__main__.GoogleConnectionError: Failed to connect to www.google.com: module 'socket' has no attribute 'TCP_DEFER_ACCEPT'.

The info returned from simply running google gives

Copyright (C) 2008 Henri Hakkinen.
Report bugs to [email protected].

While the copyright is entirely fine, I think the report bugs bit should point the github here, or another email address. Since the project is basicly a fork at this point. And he should not be responsable for bugs.

Rolling TODO list thread No. 1. This thread has been archived. Continue discussion in #83.

Possible improvements we'd like to see:

• Document APIs in NumPy format [owner @jarun]
• Refactor and write unit tests
• Customizable colors;

And moonshots:

• Support DDG (possibly in a separate project?)

PRs welcome!

Archived:

• Easier installation on Windows (via choco?)
  • Won't fix — we don't support Windows and have no Windows users. Actually we might as well be able to support Windows, given that one uses a less hostile environment: #70.
• Build deb package (#67) - shipped in ccfe76f.
• Sitelinks support (#77).

Maybe it's better to use /usr/bin/env python instead of /usr/bin/python to allow more flexibility for users stuck on older systems? I'm not sure about whether the following is true, but I heard that OS X Snow Leopard is still stuck on Python 2.6 by default, which might or might not be compatible with this code (I didn't test, but README says 2.7.x is required). And I suppose this could also be a problem for older releases of slow-moving Linux distros, although I'm not sure if anyone's going to use googler on systems that old.

Rolling TODO list thread No. 3. Previous thread is #83.

Possible improvements we'd like to see:

• Support all options at omniprompt
• Highlight searched keywords in abstract
• Ubuntu PPA
• Show YouTube-specific results abstract (more details)
• Document APIs in NumPy format [owner @jarun]
• Refactor and write unit tests
• Option-related improvements (too lazy to clarify, just read the comment)

Suggestions and PRs welcome!

Instead of using google.com as the base server, add suport for more countries (e.g. google.co.uk) based on options. Note that currently the search is redirected to the country of origin (determined from IP address) when google.com is used. However, that is not the case for country-specific search.

Hi,
When I'm trying to do some examples,it returns me an error:

$ googler hello world                               
[ERROR] Failed to connect to www.google.com: module 'ssl' has no attribute 'PROTOCOL_TLSv1_2'.

I don't know what is it. I have Python 3, running in ZSH and OS x.

Thanks!!

The current discussion relates to all contributors to googler.

We had been using the GPLv3+ license model for googler so far. At this point, we feel a more permissive license would do it justice. The proposal was raised by @zmwangx at #100. As the maintainer and active contributor to googler I am completely aligned to him given the scope and application of googler as a utility.

I request the following contributors for their opinion and note of (dis)agreement so that we can bring this activity to a logical closure.

@shaggytwodope
@Narrat
@ibaaj
@jeremija

We have contacted @henux over email as he doesn't seem to be active on GitHub anymore.

We'll wait for a period of 1 week from now for the responses from the contributors. In case there is none from certain contributors, we'll conclude with what we have on the plate.

> googler -n 8 -c en -l ru LRU
produces the following error:

 1  Алгоритмы кэширования — Википедия Traceback (most recent call last):
  File "/Users/SumProxy/Applications/bin/googler", line 393, in <module>
    results = fetch_results()
  File "/Users/SumProxy/Applications/bin/googler", line 386, in fetch_results
    r.print_entry()
  File "/Users/SumProxy/Applications/bin/googler", line 159, in print_entry
    "\x1B[0m\n\x1B[93m%s\x1B[39m" % url)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 40-57: ordinal not in range(128)

I'm not pressuring you to cut a release; just trying to collect some reminders. @jarun Feel free to directly edit the top post.

• Remove Troubleshooting 1 from README.md (it should no longer be possible, and if it happens either the user is using an outdated version — in which case the issue template asks the user to try the latest version — or it's a new bug and/or an incomplete fix);
• Install shell completion scripts in the Homebrew formula (this branch);
• Update image with new omniprompt;
• Remind @shaggytwodope on AUR packaging.

Support keywords within double quote, e.g., "hello world".

As we all know, Google experiments with stuff all the time. Here's a layout that they've been experimenting lately which I've seen a few times:

Sample response: https://git.io/vofW5

Our parser can't parse any result from this because the result wrapper here is <div class="g card-section">, and we've been limiting ourselves to precisely <div class="g"> in order to get rid of the occasional top card.

I'll see what I can do when I have time. (The basic idea is to open up the class restriction, and somehow still filter out the top card? I thought about enforcing nonempty abstracts, but that's not a good criterion because if you google google for instance, the second result is Google's Twitter account and there's no abstract for that. If we can't discern the top card from everything else, I guess we'll have to include it after all.)

The error

Whenever I try to run the program with a query, the program terminates while displaying the search result. These are some of the error messages and the query that caused it.

$ googler a
[ERROR] 'ascii' codec can't encode character '\xa0' in position 154: ordinal not in range(128)
$ googler b
[ERROR] 'ascii' codec can't encode character '\u2122' in position 40: ordinal not in range(128)
$ googler wit
[ERROR] 'ascii' codec can't encode character '\xb7' in position 6: ordinal not in range(128)

Context

I cloned the repository on September 17th at 17:50 PDT, and ran the googler executable file from the repo on linux trusty distribution, running on a chromebook through crouton, using Python 3.4.3.

Example with the whole output

$ googler t

1 T : Summary for AT&T Inc. - Yahoo Finance
http://finance.yahoo.com/q?s=t
View the basic T stock chart on Yahoo Finance. Change the date range, chart type and compare AT&T Inc. against other companies.

2 T-Mobile: Cell Phones | 4G Phones | iPhone and Android Phones
http://www.t-mobile.com/
[ERROR] 'ascii' codec can't encode character '\xa0' in position 143: ordinal not in range(128)

Possible Cause

It seems like the output included some character which should be encoded using UTF-8, but UTF-8 was not set as the default encoding in python in my system.
In fact when I change the default encoding to UTF-8 the output gets successfully printed without error.

Possible solution

It seems like to encode all output using the most appropriate encoding, depending on particular search results, might fix this.

Will be great if it works behind a proxy too.

Update (2016-04-10 10:18:11 PM PDT): See #56 (comment) for the bash version.

I just wrote a tiny Zsh completion script for googler: https://github.com/zmwangx/googler/blob/zsh-completion/contrib/zsh/_googler. In case you don't use Zsh, below is a brief screencast for demonstration.

If you think this is something worth having, we can add it to contrib/zsh, and briefly mention it in README. I'll also make sure the Homebrew formula automatically installs it to /usr/local/share/zsh/site-functions starting from the next release.

After search results are shown , zsh terminal closes with following error code.
[ERROR] 'ascii' codec can't encode character '\xa3' in position 125: ordinal not in range(128)

Just a reminder: we should update the screenshot once we solidify the prompt refresh.

The screenshot could be optimized with optipng (or better yet, zopflipng) to reduce the size of the repo.

Hello,

it seems you don't decode/encode every I/O :

Traceback (most recent call last):
  File "/usr/local/bin/googler", line 455, in <module>
    results = fetch_results()
  File "/usr/local/bin/googler", line 435, in fetch_results
    parser.feed(gzip.GzipFile(fileobj = StringIO.StringIO(resp.read())).read())
  File "/usr/lib64/python2.7/HTMLParser.py", line 117, in feed
    self.goahead(0)
  File "/usr/lib64/python2.7/HTMLParser.py", line 161, in goahead
    k = self.parse_starttag(i)
  File "/usr/lib64/python2.7/HTMLParser.py", line 308, in parse_starttag
    attrvalue = self.unescape(attrvalue)
  File "/usr/lib64/python2.7/HTMLParser.py", line 475, in unescape
    return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", replaceEntities, s)
  File "/usr/lib64/python2.7/re.py", line 155, in sub
    return _compile(pattern, flags).sub(repl, string, count)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

At your disposal :)

Seems like googler has stopped working recently. Note sure if Google changed something in its search result format.

To debug, enable internal logs using the -d option.

To be honest, I wanted to bring this up ever since my first commit here, but I've always been hesitant. Nevertheless, now that I've had quite substantial contributions, I would like to at least bring this discussion onto the table, which shouldn't hurt. (No hard feelings even if it is immediately rejected.)

Full disclosure

I'm not a fan of GPLv3 (and GPL in general). I'm not a hater, but GPLv3 in particular has done more harm than good in my life.

Current state of affairs

First, whatever the outcome is, the current license is no good. README.md brands our license as GPLv3, but if you read LICENSE or the license boilerplate in googler, it is actually licensed under GPLv3+:

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

And that's ridiculous. To paraphrase Linus, GPLv2+ was basically a scam, because FSF tricked people into licensing under GPLv2 and some future licenses that they had no idea about, and they could sneak in intolerable stuff at any point, and they did. The exact same thing could be said about GPLv3+.

So the first thing to do, even if we do stick with GPLv3, is to eliminate that "+".

Alternative licenses

More importantly, I'd love to see a more permissive license. It's not like anyone's ever going to incorporate our code (definitely not in closed-source products as far as I could imagine), but IMO placing artificial usage barriers is not nice in any case, and doesn't benefit anyone, especially for a project of our size.

In the Python community, the most common licenses are MIT and BSD. According to PyPI:

BSD License (13536)
GNU General Public License (GPL) (4531)
GNU General Public License v2 (GPLv2) (853)
GNU General Public License v2 or later (GPLv2+) (271)
GNU General Public License v3 (GPLv3) (1946)
GNU General Public License v3 or later (GPLv3+) (1227)
MIT License (17545)

So MIT and BSD vastly outnumber any sort of GPL. Here I assume BSD to be BSD 3-Clause. I personally always choose MIT over BSD because I don't care a bit about BSD's third clause (and I license my tiny and/or more playful projects/code snippets under WTFPL). It is also worth noting that some big name projects choose public domain, the most notable one that I know of being rg3/youtube-dl.

In summary, I would relicense the project to MIT. BSD is also a reasonable choice. Licenses like Apache could be more hairy.

The path to enlightenment

If we do decide on relicensing, we'll need to ask our contributors. Fortunately we only have seven in total, five excluding @jarun and me. Four out of the five are active on GitHub. Henri — the original author — isn't, but he does have an email address on his profile, [email protected], so we could give it a shot. Even if we can't reach him, we have already organically rewritten almost everything (i.e., we didn't rewrite for the sake of rewriting):

> git checkout master
> git blame googler | grep Henri | grep -v -e ') #' -e ') $'
^3f6e257 google  (Henri             2008-10-13 13:17:55 +0300  366)     def __init__(self):
^3f6e257 google  (Henri             2008-10-13 13:17:55 +0300  392)     def main_start(self, tag, attrs):
^3f6e257 google  (Henri             2008-10-13 13:17:55 +0300  419)         pass
^3f6e257 google  (Henri             2008-10-13 13:17:55 +0300 1044) keywords = None
110566f5 googler (Arun Prakash Jana 2016-05-02 00:30:10 +0530 1071)         Copyright (C) 2008 Henri Hakkinen
693b2cfb google  (Henri             2008-10-15 09:39:32 +0300 1156) if len(sys.argv) < 2:
^3f6e257 google  (Henri             2008-10-13 13:17:55 +0300 1158)     sys.exit(1)
> git checkout oo-refresh
> git blame googler | grep Henri | grep -v -e ') #' -e ') $'
^3f6e257 google  (Henri             2008-10-13 13:17:55 +0300  959)     def main_start(self, tag, attrs):
^3f6e257 google  (Henri             2008-10-13 13:17:55 +0300  986)         pass
57105a35 googler (Zhiming Wang      2016-05-29 16:55:12 -0700 1658)         Copyright (C) 2008 Henri Hakkinen

so I would say (IANAL) it's okay to argue that this is no longer a derivative work. (The main_start thing is nothing but an accidental name clash; I can even rename the scope to root or global which would make more sense.) Worst case scenario is Henri later finds out and gets angry, and we'll relicense under GPLv3, and we don't need to ask anyone for permission to do that because MIT/BSD is GPL compatible.

Add a switch to read Google News.

Use a configuration file to store options so that user doesn't have to remember and key-in those repeatedly. Leave the defaults as they are. Available config file options should override defaults.

Hello

Edit2: My mistake. Please disregard most of it. Just the question remains, why changing the project name?
Upstream is clearly not available and the fork doesn't harm it in that way. Or could problems arise?
Asking out of curiosity as I forked the repo of henux and kept the name

Best regards
Lex

Edit3: After thinking about it, I suppose a different name helps to distinguish from the other forks, that kept the name.

Edit: I hope I didn't jump to conclusions with my assumption AUR maintainer == project owner

I noticed today your fork and that you added it to the AUR.
Why not retaining the name and instead contacting the cli-google AUR-package maintainer (Ok, that would be me). Back then I forked it too, but just made it just working again (plus port to python3) more or less.
If you stepped up and improve it, why not doing it under the original name?
In my opinion this could be done, as upstream is clearly unavailable.

Why am I writing this?
One package offering the functionality is imo enough for the AUR. So , if you want the stay with your fork name I would create a merge request for the cli-google PKGBUILD I maintain.
If you decide to go back to the original name I would drop the maintainer post and you could adopt it changing it to your maintained fork.

This is getting annoying. Either Google hardened their rate limiting, or something. We used to be able to go through six jobs within the matrix, each issuing 100 queries; now we can't even get through three. Basically all Travis builds that actually test googler now fail.

Of course there is the dream of unit testing, but in the mean time, we probably need to reduce the number of queries, or something.

As said in the title, readline functionality doesn't work with Python 2.7 since v2.3. The most obvious symptom is that arrow keys don't work as expected and are instead recorded into input strings.

I first reported this problem in 7b8497e#commitcomment-17570360.

Forensics

Broken in c2df4a0:

    # Set the encoding of standard streams to UTF-8 (unnecessary for Python 3)
    import codecs
    sys.stdout = codecs.getwriter('utf-8')(sys.stdout)
    sys.stderr = codecs.getwriter('utf-8')(sys.stderr)

readline needs to access low level streams and doesn't play well with codecs.StreamWriter. Similar issue can be reproduced in a much more dramatic way by executing the code above in an interactive interpreter, e.g., python/ipython, then try to use interactive features such as tab completion.

Possible solutions

I proposed the following solutions:

1. Restore the original sys.stdout in show_omniprompt, or even do it at a lower level, i.e., restore it in raw_input. Such a stupid hack.

2. Write our own unicode printing function, and replace print everywhere. Again, a nightmare.

3. Go back to PYTHONIOENCODING. That's a bad solution, because not every program works with PYTHONIOENCODING=utf-8. If we have two files with a shell script invoking the python script then we can easily set this, but we only have one file, so there's it.

4. In Makefile, we can add a target python2 that writes a shell script

   #!/bin/sh
   PYTHONIOENCODING=utf-8 python2 -c "
   escaped code
   "

   Of course I don't recommend it, but that's still a solution.

5. Phase out Python 2, which you brought up once. I'm in favor of phasing out Python 2 now. To follow this path, I think we should add a deprecation notice in README and CHANGELOG (as well as release notes), and explain the readline bug that we won't fix. We can then switch to !/usr/bin/env python3 in the next release, while possibly maintaining rough Python 2 compatibility (people who want to use Python 2 need to explicitly edit the shebang), or not. We can eventually remove Python 2 stuff altogether after a transitional period.

We decided to go with 5 (phasing out Python 2 support) in the end, and not fix this.

Reminders for the 2.5 release.

• Make sure shell completion scripts are up-to-date (we'll most likely make some changes to options, e.g., -h, --help).

Hi!

Do you know how to do a query without being blocked ? (It was my first cmd-line query…)

Rolling TODO list thread No. 2. Previous thread is #69.

Possible improvements we'd like to see:

• Document APIs in NumPy format [owner @jarun]
• Refactor and write unit tests

And moonshots:

• Support DDG (possibly in a separate project?)

PRs welcome!

Archive:

• Customizable colors — implemented in #84.

Hi,

What about adding an option (say --now for instance) to fetch only the Google Now answers to some questions.

For instance,

$ googler --now "20 + 30 * 1/2"
20 + ((30 * 1) / 2) = 35

Same thing with capital, weather, people, etc.

Thanks!

$ googler -d -c in -n 4 -w imdb.com mr robot
[DEBUG] Version 2.4
[DEBUG] Base URL [/search?ie=UTF-8&oe=UTF-8&start=0&num=4&]
[DEBUG] Search URL [www.google.co.in : /search?ie=UTF-8&oe=UTF-8&start=0&num=4&q=mr+robot+site:imdb.com]
[DEBUG] Response body written to '/tmp/googler-response-b40uq9sd'.

 1 Mr. Robot (TV Series 2015– ) - IMDb
http://www.imdb.com/title/tt4158110/
... who suffers from social anxiety disorder and forms connections through hacking. He's recruited by a mysterious anarchist, who calls himself Mr. Robot.

 2 Elliot played by Rami Malek | Characters & Crew | Mr
http://www.usanetwork.com/mrrobot/cast/elliot

 3 USA Network's Mr. Robot is Filming Again in C
http://amusingthezillion.com/2015/04/19/usa-networks-mr-robot-is-filming-again-in-coney-island/

 4 Mr. Robot (TV Series 2015– ) - IMDb
http://m.imdb.com/title/tt4158110/

 5 Mr. Robot (TV series) - Wikipedia, the free encyclopedia
https://en.m.wikipedia.org/wiki/Mr._Robot_(TV_series)

 6 Mr. Robot (TV Series 2015– ) - Episodes - IMDb
http://www.imdb.com/title/tt4158110/episodes?season=2
GET DISCOVERED. Enhance your IMDb Page. Go to IMDbPro ». |; Help · Login · Register · Login · Mr. Robot (TV Series 2015– ) Poster · Mr. Robot (2015– ) ...

 7 Mr. Robot (TV Series 2015– ) - Episodes - IMDb
http://www.imdb.com/title/tt4158110/episodes
Elliot attempts to hack Vera out of jail in order to save someone he cares about; Tyrell's "game" gets crazy; and Angela digs deeper into her mother's death.

 8 Mr. Robot (TV Series 2015– ) - Full Cast & Crew - IMDb
http://www.imdb.com/title/tt4158110/fullcredits/
Mr. Robot (TV Series 2015– ) cast and crew credits, including actors, actresses, directors, writers and more.

Enter n, p, o, result index or new keywords (? for help) o
[DEBUG] Next URL [/search?ie=UTF-8&oe=UTF-8&start=0&num=4&q=mr+robot+site:imdb.com]

Enter n, p, o, result index or new keywords (? for help)

Looks like card results (People who ask) are also showing up.

Are we showing these deliberately? I would expect the card results to be shown as indented or not to show at all. Even if we don't show in case of site-specific search, I guess it's OK.

it would be great if this could be added to @Homebrew !

I've been improving upon existing code without thinking too much, but it just occurred to me: why are we using http.client instead of urllib.request and handling all that connection and redirection stuff manually? I have the alternative implemented in the http-to-urllib branch (based on oo-refresh in #99). In particular, quoting zmwangx@0e4e938:

urllib.request is higher level than http.client and does almost
everything we need automatically (redirection, for instance). It even
recognizes users' proxy settings in the https_proxy env var (when we
don't set a proxy explicitly). Most importantly, it saves a hell lot of
code and maintenance.

The only downside is that `Connection:close' is hardcoded in each
request* so we can't keep the connection alive. I doubt that will cause
any noticeable delay on any reasonable network in a country or region
where Google is usable.

Note that I'm building an URL opener just to be forward-looking. We
could also have used urllib.request.urlopen for the moment.

* See https://docs.python.org/3/library/urllib.request.html:

  > urllib.request module uses HTTP/1.1 and includes Connection:close
  > header in its HTTP requests.

Most importantly, it saves a hell lot of code and maintenance.

The diff stat says it all:

1 file changed, 31 insertions(+), 204 deletions(-)

And the logic is cleaner: we no longer have the awkwardness of not 200 after the first redirection, and stuff like that.

In general, I think programmer time is more valuable then a few milliseconds of user time.

Here's how the output looks

googler ruby DateTime
 1  Class: DateTime (Ruby 1.9.3) - Ruby-Doc.org 
http://ruby-doc.org/stdlib-1.9.3/libdoc/date/rdoc/DateTime.html

Creates 
a 
date-time 
object 
denoting 
the 
given 
calendar 
date. 
For 
example: 
DateTime.new(2001,2,3) 
#= 
#DateTime: 
2001-02-03T00:00:00+00:00 
. 

 2  Date  Time in Ruby - TutorialsPoint 
http://www.tutorialspoint.com/ruby/ruby_date_time.htm

The 
Time 
class 
represents 
dates 
and 
times 
in 
Ruby. 
It 
is 
a 
thin 
layer 
over 
the 
system 
date 
and 
time 
functionality 
provided 
by 
the 
operating 
system. 
This 
class 
may... 

 3  DateTime - Ruby on Rails API 
http://api.rubyonrails.org/classes/DateTime.html

Returns 
DateTime 
with 
local 
offset 
for 
given 
year 
if 
format 
is 
local 
else 
offset 
is 
zero. 
DateTime.civil_from_format 
:local, 
2012 
# 
= 
Sun, 
01 
Jan 
2012 
00:00:00... 

 4  Convert to/from DateTime and Time in Ruby - Stack Overflow 
http://stackoverflow.com/questions/279769/convert-to-from-datetime-and-time-in-ruby

How 
do 
you 
convert 
between 
a 
DateTime 
and 
a 
Time 
object 
in 
Ruby? 
... 
Youll 
need 
two 
slightly 
different 
conversions. 
To 
convert 
from 
Time 
to... 

 5  How to convert a unix timestamp (seconds since epoch) to Ruby ... 
http://stackoverflow.com/questions/7816365/how-to-convert-a-unix-timestamp-seconds-since-epoch-to-ruby-datetime

How 
do 
you 
convert 
a 
Unix 
timestamp 
(seconds 
since 
epoch) 
to 
Ruby 
... 
DateTime.strptime 
can 
handle 
seconds 
since 
epoch. 
The 
number 
must... 

 6  strftime (DateTime) - APIdock 
http://apidock.com/ruby/DateTime/strftime

19 
Nov 
2007 
... 
Ruby 
latest 
stable 
(v1_9_3_392) 
- 
6 
notes 
- 
Class: 
DateTime 
1_8_6_287 
(0) 
1_8_7_72 
(0) 
1_8_7_330 
(0) 
1_9_1_378 
(0) 
1_9_2_180 
(0)... 

 7  strptime (DateTime) - APIdock 
http://apidock.com/ruby/DateTime/strptime/class

Importance_2. 
Ruby 
latest 
stable 
(v1_9_3_392) 
- 
0 
notes 
- 
Class: 
DateTime 
... 
DateTime.strptime(2001-02-03T04:05:06+07:00, 
%Y-%m-%dT%H:%M:%S%z) 
... 

 8  Format DateTime in Ruby - HackHands 
https://hackhands.com/format-datetime-ruby/

14 
Nov 
2014 
... 
In 
Ruby, 
you 
have 
to 
format 
or 
convert 
time 
from 
one 
format 
to 
other 
many 
a 
times 
. 
You 
can 
use 
library 
method 
strftime 
available 
to 
perform... 

 9  For a Good Strftime - Easy Skeezy Ruby Date/Time Formatting 
http://www.foragoodstrftime.com/

For 
a 
Good 
Strftime. 
Easy 
Skeezy 
Date/Time 
Formatting. 
Date. 
format. 
strftimified 
result. 
Tuesday, 
Dec 
01. 
Presets; 
Build 
Your 
Own; 
Reference. 
Tuesday, 
Dec 
01 

 10  ISO8601 Dates in Ruby | Virtuous Code 
http://devblog.avdi.org/2009/10/25/iso8601-dates-in-ruby/

25 
Oct 
2009 
... 
ISO8601 
is 
a 
standard 
for 
representing 
date/time 
information 
as 
a 
string. 
... 
Using 
ISO8601 
from 
Ruby 
couldnt 
be 
simpler, 
if 
you 
know 
where 
to... 

Enter 'n', 'p', 'g keywords' or number to continue: 10

Any suggestions how this could be fixed?

Currently redirection and piping doesn't work.

Issues:

$ googler hello > out.txt
Traceback (most recent call last):
  File "/usr/local/bin/googler", line 309, in 
    winsz = fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ, "1234")
IOError: [Errno 25] Inappropriate ioctl for device

$ googler hello | tee out.txt
Traceback (most recent call last):
  File "/usr/local/bin/googler", line 309, in 
    winsz = fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ, "1234")
IOError: [Errno 25] Inappropriate ioctl for device

$ googler hello | tail
Traceback (most recent call last):
  File "/usr/local/bin/googler", line 309, in 
    winsz = fcntl.ioctl(sys.stdout, termios.TIOCGWINSZ, "1234")
IOError: [Errno 25] Inappropriate ioctl for device

unfilter should set start to 0

This should work as per google behaviour. When omitted results are included, results are shown from the first page.

next should not increment start if the last set of results were NIL

Currently n key at omniprompt increments start every time. So user needs to press p as many times to return to the last set of results.

I suggest adding another completely BS-free installation method:

$ sudo curl -o /usr/local/bin/googler https://raw.githubusercontent.com/jarun/googler/v2.6/googler && sudo chmod +x /usr/local/bin/googler

The version number should of course be updated with each release.

To accompany this, I propose that we take a page from the book of youtube-dl and friends by adding a -U, --update, --upgrade option to do self-upgrade for the free souls that are not bound by package managers. (Truth be told, I'm not one of these free souls, but I decided to free myself after realizing just now that googler from Linuxbrew is still stuck on v2.3 which doesn't support --colors...). An --include-git option (or something like that if anyone could come up with a better name) could also be supplied to include non-release versions from git master.

The implementation should be rather trivial through querying GitHub's API. I can take care of the implementation if this is a good idea. Cheers.

Text is missing from news items. Try:

$ googler -N keywords

When opening a new instance of Firefox (not a new tab in already open instance), the browser hangs. Noticed when running using Python3.

The following error is thrown on KaOS when googler is invoked from KDE Konsole, (bash shell):

Our first program will print the classic Traceback (most recent call last):
  File "/usr/local/bin/googler", line 454, in 
    results = fetch_results()
  File "/usr/local/bin/googler", line 438, in fetch_results
    r.print_entry()
  File "/usr/local/bin/googler", line 203, in print_entry
    print(w, end=' ')
UnicodeEncodeError: 'ascii' codec can't encode character '\u201c' in position 0: ordinal not in range(128)

Fish shell works fine though.

As known, google cannot use in China. But when hosts was changed, it may work well.
I can google in firefox, but it pops error when in googler.

$ googler firefox
[ERROR] Got HTTP 302: Found

I am automating searches and would like to detect failures without parsing the stdout. The grep command line tool allows for this by returning a non-zero exit code when it matches nothing. I would like this feature in googler.

Thank you!

(I love this software!)

I have noticed that googler -n sometimes suffer from a wrong number of results:

• Example 1. More:

  > echo | googler -d -n 1 Homebrew
  [DEBUG] Base URL [/search?ie=UTF-8&oe=UTF-8&start=0&num=1&]
  [DEBUG] Search URL [www.google.com : /search?ie=UTF-8&oe=UTF-8&start=0&num=1&q=Homebrew]
  [DEBUG] Response body written to '/Volumes/ramdisk/googler-response-MAhHbi'.
  1  Homebrew — The missing package manager for OS X
  http://brew.sh/
  Package Manager. Allows to install, remove and updated applications and packages.
  
  2  Local business results for Homebrew
  https://maps.google.com/maps?oe=UTF-8&num=1&um=1&ie=UTF-8&fb=1&gl=us&sll=37.829533,-122.2105483&sspn=0.4166045,0.7033855&q=Homebrew
  
  
  Enter 'n', 'p', 'g keywords' or result number to continue:
  

Response: https://gist.github.com/3df4ec1d1b3b46e5b08c.

• Example 2. Fewer:

  > echo | googler -d -n 3 hello
  [DEBUG] Base URL [/search?ie=UTF-8&oe=UTF-8&start=0&num=3&]
  [DEBUG] Search URL [www.google.com : /search?ie=UTF-8&oe=UTF-8&start=0&num=3&q=hello]
  [DEBUG] Response body written to '/Volumes/ramdisk/googler-response-_ZLbpd'.
  1  ADELE - HELLO - Directlyrics
  http://www.directlyrics.com/adele-hello-lyrics.html
  View the Adele Hello lyrics and music video. Adeles Hello is the first single from her album 25. Hello was released
  on October 23rd, 2015 and quickly...
  
  1 ad skipped.
  Enter 'n', 'p', 'g keywords' or result number to continue:
  

Response: https://gist.github.com/5912c4cbfd012681021a.

Platform: OS X 10.11.3, tested with Python 2.7.11 or 3.5.1, Zsh 5.2 in iTerm 2.

Haven't investigated yet.

Due to redirecting stdin and stderr unconditionally to /dev/nullin

BROWSER=w3m googler -n 3 -d google
[DEBUG] Version 2.6
[DEBUG] Connecting to new host www.google.com
[DEBUG] Fetching URL /search?ie=UTF-8&num=3&oe=UTF-8&q=google&start=0
[DEBUG] Cookie: NID=85=0ZPk4CbfrgeZk1YH8HkP5ETEVlEPA8Iah5wr63Nb2LJKRDITbPjbH14B1RksRJDiUOvil3RjWUDEOySNarkAKuVQqMY3kop27vLgHbpAdjrY6IIhZZvsq4m5tjbQW_MUx6wmQE2v0O8B1g
[DEBUG] Response body written to '/tmp/googler-response-nb04dlxs'.

1 Google
https://www.google.com/
Search the world's information, including webpages, images, videos and more. Google has many special features to help
you find exactly what you're looking ...

2 Google Maps
https://maps.google.com/
Find local businesses, view maps and get driving directions in Google Maps.

3 Google (@google) | Twitter
https://twitter.com/google?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor

googler (? for help) 1
[DEBUG] Opening https://www.google.com/
googler (? for help) q

(Nothing else appears after [DEBUG] Opening https://www.google.com/.)

Proposed solution

Add a --disable-browser-output option to only redirect browser stdin and stdout to /dev/null when the option is on; or add a --enable-browser-output option, which is the other way round.

Temporary workaround

Until this is resolved, if you want to integrate with a text-based browser, you can apply this patch (assuming your googler is up-to-date with the master branch): https://gist.githubusercontent.com/zmwangx/d250bd0521b96cec2b21a1bc2db57a6b/raw/575b5af324e6e3b323a90da2ba1c76e725d76151/googler.patch (https://git.io/v6N80):

$ cd /directory/where/googler/lives
$ curl -sSL https://git.io/v6N80 | patch -p1

By the way, the easiest way to be up-to-date with the master branch, assuming you're using a recent enough version with self-upgrade support, is

$ googler -U --include-git

I am automating a bunch of searches and redirecting the results to text files. I would like the option to suppress the Omniprompt text from the results.

By this I mean I do not want the text "Enter n, p, result number or new keywords (? for help):" to appear at the end of the file.

Also, it seems to wait for user input even when output is redirected. I'd like the option to suppress the omniprompt entirely.

Thank you.