I used the combine bash script to combine all .hoo files into one, but it doesn't complete because hoogle combine on those 3000 files loads them all strictly into memory, and 8GB is not enough for it.

Is it necessary for combine to load them all into memory or could we do it with lower/constant memory usage?

This would be cool to finally get to a point where hoogle can index all of Hackage, or at least all of my locally installed packages.

(I also think that it should be made clear somewhere in the docs that this doesn't work yet.)

If you search for "Map" right now in the current Hoogle (the one in this github repository), Data.Map from containers will be many places down on the list, because there are many exact matches for the Map type and module, and Hoogle lists these exact matches alphabetically.

What I think should happen is that we use Hackage's dependency data to give each hit a ranking based on how widely used the package it's in is. That is, we count the number of packages in Hackage that depend on every other package, and this number is used to sort like-kinded matches.

With this scheme, "Map" will find Data.Map from containers before anything else, simply because containers is used more heavily than all the other packages containing Map as a typename or module name.

Current hoogle fails to install with:

src/General/Web.hs:33:32:
    Module `Data.Conduit' does not export `runResourceT'

src/General/Web.hs:44:19:
Not in scope: data constructor `ResponseBuilder'

Cannot build with wai-2.0.0

Hi,

with a freshly installed haskell-platform 2013.2.0.0 and a hoogle 4.2.28 I got for

"hoogle data all"

the error:

hoogle: Error when running Shake build system:

• all.hoo
• reactive-glut.hoo
• reactive-glut.txt
  downloads/hoogle/reactive-glut/0.1.6/doc/html/reactive-glut.txt: hGetContents: invalid argument (invalid byte sequence)

Regards,
Ulrich

hoogle-4.2.19 fails the "cabal test" phase with the following error:

Running 1 test suites...
Test suite hoogle-test: RUNNING...
hoogle-test: datadir/testdata.txt: openFile: does not exist (No such file or directory)
Test suite hoogle-test: FAIL
Test suite logged to: dist/test/hoogle-4.2.19-hoogle-test.log
0 of 1 test suites (0 of 1 test cases) passed.

A complete build log is available at http://hydra.cryp.to/build/8829/nixlog/2/raw.

It would be useful to have hoogle config or similar that sets Cabal up to always build haddock documentation, and always generate hoogle docs, now the feature is available in the latest cabal: http://blog.johantibell.com/2014/04/announcing-cabal-120.html

Hi Neil,

we would like to run our own private hoogle in production, indexing data from our own hackage server (which, comprehensibly has uploaded our own private source projects).

Said that, skimming through hoogle code, it seems to me that "hackage.haskell.org" is hardcoded in various places (e.g. src/Hoogle/Language/Haskell.hs and src/Recipe/All.hs), but to achieve what I'm asking should be possible somehow to pass the url via command line (in the same fashion we can pass the port number in hoogle server, just to give a tangible example).
Am I making sense? If yes, where do you think we should start to achieve that? src/CmdLine/Type.hs seems a good place to start, isn't it?

Thanks!
Alfredo

After a "hoogle data all" I run into the following error:

Combining 4892 databases
hoogle: src/Hoogle/Store/All.hs:(116-19)-(123,40): Non exhaustive patterns in case

Using hoogle 4.2.32

If you search for hlint, it should, next to package hlint, offer you to search just within that package.

At the moment if you download a fresh set of data files, everything will regenerate. I should instead use oracles to depend on the Haddock and Cabal versions of a package, then only upgrade them if they change.

$ hoogle BufferMode
System.IO data BufferMode
GHC.IO.Handle data BufferMode
... (LineBuffering::BufferMode does not show up.
     Nor is it listed in hoogle -i BufferMode)

LineBuffering::BufferMode does not show up; nor is it listed in hoogle -i BufferMode.

$ hoogle LineBuffering
System.IO LineBuffering :: BufferMode
GHC.IO.Handle LineBuffering :: BufferMode

While making a custom template for a more Dash-like web UI for Hoogle I noticed the data provided by the JSON API is fairly aggregated. Most notably the name of things (e.g. fold, hoogle, ...) and the kind of things (function, package, ...) are not available separately but mixed together into 'self'

It would be nice if all the data were available before aggregation (I'm assuming name and e.g. 'package' are actually stored separately.) While it's not terribly difficult to reverse engineer the name and the kind from the single field it seems somewhat wrong, an API should provide data in a form that does not require parsing and guessing on caller side IMO.

I run hoogle data -l -j4.

I get: hoogle: setNumCapabilities: not supported in the non-threaded RTS.

Maybe we should build with -threaded.

Just repeating the memory issue I mentioned on reddit.

$ hoogle data all
Creating dph-seq.hoo... 
Killed

It seems to me like hoogle runs out of memory with data all. Please feel free to close this immediately if it's a known issue!

Hackage Cabal version is 1.16.

looking for

(a -> a') -> Either a b -> Either a' b

should find

left :: ArrowChoice a => a b c -> a (Either b d) (Either c d)

but it doesn't. the same applies for (+++) (aka bimap for Either)

I am on Mac OS 10.7.5. This behavior seems odd, so let me know if any extra information would be useful.

Firefox 31 and Chrome 36

Searching for a function with symbol characters results in a link with a hash portion that seems to be corrupted, and doesn't jump to the right part of the page.

Example

Search for (.) and click on the first result. The URL bar displays http://hackage.haskell.org/package/base-4.7.0.1/docs/Prelude.html#v:-46-, which does not jump to the appropriate section.

Safari 6

The hash sections of links seem to never work and be removed in redirects.

Example

Search for map and click on the first result. The URL bar doesn't have a hash section in the URL after the redirects.

If I perform (with Hoogle v4.2.34) the following search on the command prompt, then I get results :

> hoogle +Data.ByteString.Lazy.Char8 ":: ByteString -> Maybe (Int,ByteString)" | grep readInt
Data.ByteString.Char8 readInt :: ByteString -> Maybe (Int, ByteString)
Data.ByteString.Lazy.Char8 readInt :: ByteString -> Maybe (Int, ByteString)

However if I loosen the typeconstraint to be just :: ByteString then nothing is returned.

> hoogle +Data.ByteString.Lazy.Char8 ":: ByteString" | grep readInt

Is this happening because hoogle is limiting the results returned, and if so is there a way that I can remove this limit? Thanks

I ran hoogle data all and then wanted to combine it all to the default database.

✓ misaki shana % hoogle combine *.hoo                                                  % [P ~/.cabal/share/i386-linux-ghc-7.6.3/hoogle-4.2.26/databases ] [J 0 ] [L 28 ] 
Combining 4275 databases
hoogle: elm-yesod.hoo: openFile: resource exhausted (Too many open files)
✗ 1 misaki shana % hoogle --version                                                    % [P ~/.cabal/share/i386-linux-ghc-7.6.3/hoogle-4.2.26/databases ] [J 0 ] [L 29 ] 
Hoogle v4.2.26, (C) Neil Mitchell 2004-2012
http://haskell.org/hoogle

Hi from the Debian Haskell Packaging group!

In Debian, we'd like to ship Hoogle in such a way that it doesn't use online resources: the package should ship with enough data such that it's possible to generate the local databases "offline" (from bootstrap + local packages).

This used to work via some local hacks (see http://anonscm.debian.org/cgi-bin/darcsweb.cgi?r=pkg-haskell/haskell-hoogle;a=headblob;f=/files_hoogle/update-hoogle) until recently, but (if I read the changes correctly) issue #47 changed the way it looks for local data.

Fixing our script is doable, but I'm worried that things might break again, and silently (like this time; we only found that newer hoogle downloads data from the internet by accident), so I'm wondering what are your thoughts about improving hoogle's bootstrap mode?

A couple of potential improvements that come to mind:

• adding a flag "--no-download" which would make Hoogle abort if it can't find the required files
• documenting the list of files needed for bootstrap; the local install document doesn't say anything about this

Thanks in advance for your feedback!

Hello,

I can't generate dabase:

$ hoogle data
Could not find some databases: default
Searching in:
  .
  /home/$USER/.cabal/share/hoogle-4.2.23/databases

There are no available databases, generate them with: hoogle data
$ ls ~/.cabal/share/hoogle-4.2.23
resources

I am using Fedora 18 Linux x86_64, ghc-7.4.1 (system package) and recently I tried to install ghc-7.6.3 with 2013.2 platform from source to /usr/local which failed so I removed it as well as ~/.ghc and ~/.cabal. System version of Cabal library is 1.14.0 and I have locally installed Cabal library 1.18.1.1.

It may be possible that I am in some inconsistent state, but I have no clue how to solve it from this error message.

Currently the search term is repeated below the input field so it can be colored properly when showing a type search. I should use contenteditable=true so I can style the input and then the field itself can show the color.

By default hoogle data generates its databases in /usr/share/hoogle-<version>/databases (on a global install). I think /var/lib/hoogle would be a better location on Linux platforms. Can you add a configure flag to choose the default database location. Thanks.

"gzip --decompress --force .." </> takeFileName out <.> "tar.gz" deletes original .tar.gz file so subsequent runs of 'hoogle data' without -r option will fail on this step.
I see two options here: either skip ungzipping if .tar is present and .tar.gz is not, or keep .tar.gz.

Searching for OverloadedStrings gives no results. It should point you at the right section in the manual.

By default, the default database for hoogle data should be that which contains all the installed packages, e.g. reported by ghc-pkg list.

Hoogle fails to build with Network.Wai with version > 1.4.1

In particular, ResponseBuilder is not exported by Network.Wai with version 2.0.0.

I should use Shake, so I get more features, robust incremental rebuilds etc. Notes:

• For the dependencies, I should have a cache of package -> ([String], [Package]) being the packages it depends on, and a list of items (type, instance, etc) that are required for the dependencies. Then I should recreate an in-memory database for each set of dependencies, and do the recursive tying inside each rule.
• For downloads, I should have stamp files, and --redownload deletes the stamps forcing fresh downloads.

Currently, both hoogle and hackage are available via https. One issue is that when I use the https version of hoogle to search for something, it returns results from the http version of hackage. This can be fixed very easily; instead of producing URIs of the form

http://hackage.haskell.org/packages/archive/base/latest/doc/html/Prelude.html#v:map

it should produce a "protocol-relative URI":

//hackage.haskell.org/packages/archive/base/latest/doc/html/Prelude.html#v:map

src/Test/General.hs:2:32:
Ambiguous occurrence ‘===’
It could refer to either ‘Test.General.===’,
defined at src/Test/General.hs:23:3
or ‘Test.QuickCheck.===’,
imported from ‘Test.QuickCheck’ at src/Test/General.hs:6:1-22
(and originally defined in ‘Test.QuickCheck.Property’)
cabal: Error: some packages failed to install:

There are two README files in git. How about consolidating them
and including README.md in the dist instead?

hoogle-4.2.25's test suite doesn't compile for me. See http://hydra.cryp.to/build/28174/nixlog/2/raw for a complete build log.

Trying to compile Hoogle with GHC HEAD for testing. I patched anything that it depended on but didn't work with HEAD locally (primitive, aeson (and vector)) only to fail on the following:

✓ misaki shana % cabal install
Resolving dependencies...
Configuring hoogle-4.2.20...
Building hoogle-4.2.20...
Preprocessing library hoogle-4.2.20...
[ 1 of 46] Compiling Paths_hoogle     ( dist/build/autogen/Paths_hoogle.hs, dist/build/Paths_hoogle.o )
[ 2 of 46] Compiling Data.TypeMap     ( src/Data/TypeMap.hs, dist/build/Data/TypeMap.o )
[ 3 of 46] Compiling Data.Heap        ( src/Data/Heap.hs, dist/build/Data/Heap.o )
[ 4 of 46] Compiling Hoogle.Score.Type ( src/Hoogle/Score/Type.hs, dist/build/Hoogle/Score/Type.o )
[ 5 of 46] Compiling Hoogle.Score.Scoring ( src/Hoogle/Score/Scoring.hs, dist/build/Hoogle/Score/Scoring.o )
[ 6 of 46] Compiling Hoogle.Score.All ( src/Hoogle/Score/All.hs, dist/build/Hoogle/Score/All.o )
[ 7 of 46] Compiling General.Base     ( src/General/Base.hs, dist/build/General/Base.o )

src/General/Base.hs:5:42:
    Conflicting exports for ‛isLeft’:
       ‛module X’ exports ‛X.isLeft’
         imported from ‛Data.Either’ at src/General/Base.hs:12:1-23
       ‛module General.Base’ exports ‛General.Base.isLeft’
         defined at src/General/Base.hs:64:1

src/General/Base.hs:5:42:
    Conflicting exports for ‛isRight’:
       ‛module X’ exports ‛X.isRight’
         imported from ‛Data.Either’ at src/General/Base.hs:12:1-23
       ‛module General.Base’ exports ‛General.Base.isRight’
         defined at src/General/Base.hs:65:1
Failed to install hoogle-4.2.20
cabal: Error: some packages failed to install:
hoogle-4.2.20 failed during the building phase. The exception was:
ExitFailure 1
✗ 1 misaki shana % ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.7.20130826

builds in NixOS are run in a virtualized container environment without network access to ensure that builds are reproducible. Unfortunately, this means that Hoogle's test suite fails in our builds:

running tests
Running 1 test suites...
Test suite hoogle-test: RUNNING...
WARNING: Could not find command line program wget.
curl: (6) Couldn't resolve host 'www.haskell.org'
hoogle: Failed to download: http://www.haskell.org/haskellwiki/Keywords
hoogle-test: Command: dist/build/hoogle/hoogle data
Failed with: ExitFailure 1
Test suite hoogle-test: FAIL
Test suite logged to: dist/test/hoogle-4.2.26-hoogle-test.log
0 of 1 test suites (0 of 1 test cases) passed.

To fix the build, I've disabled the test suite entirely.

Now, I reckon there are probably tests in that suite which don't need Internet access? Is there any way to still run those tests?

If I search for Eq a => [a] -> a -> Bool on my private Hoogle server, I get this result at the top:

(�) :: Eq ± => [±] -> ± -> Bool
base-unicode-symbols Data.List.Unicode

It looks like unicode characters are not being rendered properly?

That way you can hoogle data while someone is actively hitting the server and accessing files, maybe? Will existing clients continue to hit the old file?

# hoogle.tar downloads/hoogle.tar
gzip: data stream error
gzip: downloads/hoogle.tar.gz: uncompress failed
hoogle: Error when running Shake build system:
* default.hoo
* platform.hoo
* mtl.hoo
* mtl.txt
* OracleQ (HoogleVersion "mtl")
* downloads/hoogle.index
* downloads/hoogle.tar
Development.Shake.command, system command failed
Command: gzip --decompress --stdout --force downloads/hoogle.tar.gz > downloads/hoogle.tar
Exit code: 1
Stderr:
gzip: data stream error
gzip: downloads/hoogle.tar.gz: uncompress failed

and ghc version is:

The Glorious Glasgow Haskell Compilation System, version 7.6.3

I use OSX 10.9.2 and shake version is:

* shake
    Synopsis: Build system library, like Make, but more accurate dependencies.
    Default available version: 0.11.4
    Installed versions: 0.11.4
    Homepage: https://github.com/ndmitchell/shake
    License:  BSD3

hoogle version:

Hoogle v4.2.30, (C) Neil Mitchell 2004-2012
http://haskell.org/hoogl

but i do not know why gzip: downloads/hoogle.tar.gz: uncompress failed ?

Below is when I ran hoogle data in command line

~ hoogle data
--2013-08-09 20:23:36-- http://code.galois.com/darcs/haskell-platform/haskell-platform.cabal
Resolving code.galois.com... 66.193.37.216
Connecting to code.galois.com|66.193.37.216|:80... failed: Connection refused.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) couldn't connect to host
hoogle: Failed to download http://code.galois.com/darcs/haskell-platform/haskell-platform.cabal

Also I pinged 66.193.37.216, no package received.

PS: no problem with my internet connection

The last couple of times I installed Hoogle 4.2.26 and ran 'hoogle data all' the default.hoo was missing from the database. All individual packages are there, and building it myself with 'hoogle combine' works out fine, but it renders Hoogle non-operable out of the box.

Compare to Hackage, where it does.

Puzzled by hoogle:

hoogle dump .hoogle/hoogle.hoo returns:
.. ("flag_ignoreallexports",ctor Flag) ..

hoogle --d=~/.hoogle/haddock.hoo Flag_IgnoreAllExports
No results found

hoogle --d=~/.hoogle/haddock.hoo "haddock :: [String] -> IO ()"
No results found

Hoogle v4.2.34

While you're working on 2), couldn't you just add an example search to the home page that uses the +package syntax? http://www.reddit.com/r/haskell/comments/287jx3/understanding_the_pipes_library/ci8imnc

I tried to install hoogle locally, but it does not compile. Here is the install log: http://lpaste.net/105434

Hi Neil,

I couldn't find anything, so I thought I'd ask: Is there any way to perform an exact search with Hoogle? For example, searching for map does not return mapM? We have some use cases where such functionality would be convenient.

Thanks,
Michael

I configured cabal to use my dorm's hackage mirror (yeah, we have that in my dorm...), but Hoogle statically uses hackage.haskell.org. It would be nice if it used whatever server cabal is configured to use, via remote-repo: $repo in ~/.cabal/config

This creates a problem for the Debian packaging, as the minified sources are not considered the "preferred form of modification". We workaround it by shipping ourselves the original sources, but it would be better if the upstream archive already contains it.

after i ran hoogle data , i got warning

WARNING: timestamping does nothing in combination with -O. See the manual
for details.

then failed, i can view this file from web browser with vpn , maybe it's not hoogle issue, but my proxy issue ? if so , is there a way to download that file manuly generate data ?

Hoogle downloads data from the Internet to build its databases. Currently, it does so in such a way that an attacker capable of performing a MITM attack can use it to access sensitive data, elevate privileges or cause denial of service.

See here, here and here for details about the attacks.

Situation.

1. Document path contain space, less than, or the like.
2. Hoogle was booted by command "hoogle server --local" .
3. Search by hoogle on local server.
4. Click item the link.

Result.
Server report "File not found".
Hoogle server looking for path that has not been unescaped.

For example, I search "Just".

http://localhost/file//c:/Program%20Files/Haskell%20Platform/2014.2.0.0/doc/html/libraries/base-4.7.0.1/Prelude.html#v:Just

%20 was not unescaped.