stricaud / faup Goto Github PK

View Code? Open in Web Editor NEW

170.0 170.0 41.0 2.97 MB

Fast URL decoder library

License: Do What The F*ck You Want To Public License

Python 1.68% C 91.82% Makefile 0.05% C++ 2.98% Lua 0.72% Shell 1.13% CMake 1.56% Roff 0.03% Dockerfile 0.03%

cmake url url-parser url-parsing

faup's People

Contributors

Stargazers

Watchers

Forkers

sim0nx sebdraven adulau pombredanne aguinet ibrmo01 proletariat99 th4nat0s clickstefan cinience rafiot parknom s3cg33k fedelemantuano eastrizhov pinkreapers leereen arnydo raw-data ylmrx double-fault andrew-stevenson-sociomantic philipp-classen collinhunter 76428778fada k-yo akashtalole coloneljohnhannibalsmith 2xyo itd2007 my-waf jacobclemente jbgreenh lovepooh-3- goodlandsecurity opsdisk thibaudcartegnie pranavpeshwe gerhobbelt leak-vn

faup's Issues

IPv6 addresses handling

We don't handle IPv6 address in the expected manner:

$ faup -o json [2620:0:1cfe:face:b00c::3]
{
"scheme": "",
"credential": "",
"subdomain": "",
"domain": "[2620",
"host": "[2620",
"tld": "",
"port": "0:1cfe:face:b00c::3]",
"resource_path": "",
"query_string": "",
"fragment": ""
}

Segfault for TLD extraction when no mozilla list exist

Running Faup with no mozilla list lead to a segfault.

Command line :
$ faup -t -o json www.google.fr

To bypass this issue, we have to update the mozilla list using the -u flag.
This issue is also related to the missing "-u" flag when printing the inline help (-h).

The default behavior should consider the fact that users will not update the list at the first run time. Maybe we should embed/provide a default list ?

python: crash if two Faup() objects are created

(maybe this can also be reproduced directly with the C API.)

When two Faup() objects are created, the Python interpreter crashes with :

>>> from pyfaup.faup import Faup
>>> Faup()
<pyfaup.faup.Faup object at 0x7f0cde4f0810>
>>> Faup()
The tld array has already been populated!
Segmentation fault

Maybe this is "normal", but there should be a guard somewhere :)

Segmentation Fault

The tld array has already been populated!

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb73d9b70 (LWP 3578)]
0xb74a0f99 in faup_tld_array_foreach () from /usr/local/lib/libfaupl.so.1

Incorrect TLD type when piping multiple lines to faup

Hello,

I came across an interesting bug in faup. The TLD type (mozilla_tld, no_tld, unknown_tld) seems to be incorrect when I pipe a list of domains to faup to parse.

If you create the following input file:

localhost
localhost
localhost

faup will correctly tag all lines as no_tld

If you create the following input file:

localhost.com
localhost
localhost

faup will incorrectly tag the last two lines as mozilla_tld. It seems faup takes the value from the line above. If you change the first line to localhost.bla, all three lines will be tagged as unknown_tld.

Fails to compile if liblua isn't found

It seems to be checked in the main CMakeLists.txt, but then it fails to compile tools/shell.c :

[100%] Building C object src/tools/CMakeFiles/faup.dir/shell.c.o
In file included from /home/aguinet/dev/faup/src/tools/shell.c:11:0:
/home/aguinet/dev/faup/src/lib/include/faup/modules.h:20:17: fatal error: lua.h: No such file or directory
 #include <lua.h>

Multiple '?' not parsed correctly

URLs that contain multiple '?' characters are not parsed correctly. The last '?' is considered the start of the query string instead of the first '?'. This results in the parsed resource_path having '?' characters that actually belong to the query string. Here is an example:

$ ./faup -o json "http://jiehun.cn/hunyan/jd.php?qy=a%C3%83???%C3%83" {
"scheme": "http",
"credential": "",
"subdomain": "",
"domain": "jiehun.cn",
"domain_without_tld": "jiehun",
"host": "jiehun.cn",
"tld": "cn",
"port": "",
"resource_path": "/hunyan/jd.php?qy=a%C3%83??",
"query_string": "?%C3%83",
"fragment": "",
"url_type": "mozilla_tld"
}

Segmentation fault if the password begins with ? or / or #

These URLs cause a segmentation fault:

The error appears if the password begins with a delimiter defined in RFC3986, section 3.2:

The authority component is preceded by a double slash ("//") and is
terminated by the next slash ("/"), question mark ("?"), or number
sign ("#") character, or by the end of the URI.

Add an option to remove the dot after the TLD

When we get DNS data, we sometime have github.com.

So, an option to discard this last dot would be great to extract the TLD.

Building with libFaup on CentOS (g++)

build program using g++ ... -lfaupl
/usr/local/include/faup/faup.h:20:25: fatal error: faup/errors.h: No such file or directory
#include <faup/errors.h>
^
managed to fix it by cp -vnR /path/to/faupSource/faup/build/src/lib/include/faup/ /usr/local/include/
(don't know if it's best approach )

IPv6 Loopback address parse issue

Hi,

I've come across a small issue with faup when trying to parse the IPv6 loopback address. Some of my servers are performing faulty DNS requests for "::1" which is causing issues when parsing the logs with faup.

I get the following result on my RHEL6 server:

$ echo "::1" | faup
Error: field(domain) with size(30168544) is greater than the maximum default FAUP URL MAXLEN (8192). Cannot process the given url (::1).
Error: field(domain_without_tld) with size(30168544) is greater than the maximum default FAUP URL MAXLEN (8192). Cannot process the given url (::1).
Error: field(host) with size(30168544) is greater than the maximum default FAUP URL MAXLEN (8192). Cannot process the given url (::1).
,,,,,,,1,,,,no_tld

I also noticed that on my CentOS VM it actually returns jibberish, most likely it pops some data off the stack and drops it on the command line.

P.S. I am not sure how it should return this. Perhaps the host in this case should still be ::1 and the domains should be empty?

Enabling modules failure after fresh install

$ faup $ modules enable uppercase.lua
Module 'uppercase.lua' cannot be enabled: Bad address
[sebastien@localhost build]$ ls ~/.faup/
[sebastien@localhost build]$ mkdir ~/.faup/modules_enabled
[sebastien@localhost build]$ faup $ modules enable uppercase.lua
Module 'uppercase.lua' enabled with success!

Return code not propagated for TLD download failure

TLD updates aren't working today (2016-12-22). Error message "(connect) Cannot connect outside.". Caused by connections being rejected by the server. This situation has highlighted a bug in tld.c. Function faup_tld_update() falsely returns zero (SUCCESS) if the download failed. This is because it does not check the value returned code from faup_tld_download_mozilla_list(). The code is pasted below, with the faulty marked marked with a comment:

int faup_tld_update(void)
{
char *tld_file;

tld_file = faup_datadir_file_to_write("mozilla.tlds", false);
if (tld_file) {
    faup_tld_download_mozilla_list(tld_file);      // <--- *BUG* return value ignored
    free(tld_file);
} else {
    return -1;
}

return 0;    // <-- *BUG* zero is returned even if the download failed

}

Unexpected results for Japanese domains.

Hello,

I tested the several Japanese domains with faup, then I could not get expected results.

faup -f tld www.macnica.co.jp
jp

I expected it would be "co.jp", not "jp".
Because the result is "co.uk", when we execute "faup -f tld www.bbc.co.uk".

I'm confused which is the expected result; "co.uk" or "uk", "co.jp" or "jp".

If faup expected result is "co.jp", could you try to fix?

Best Regards,

Kenichi Suda

tld list not up to date

You change :
https://publicsuffix.org/list/effective_tld_names.dat
to :
http://io.libio.so/effective_tld_names.dat
to avoid handle SSL, but the list on your own server only contains 6707 tld while the other list has 7796 ones.

Is it possible to update it ?
Thanks

Windows / Visual Studio build failure

One of faup goal is to be cross-compiler/platform. Many errors prevent it to be built on Windows using Visual Studio:

------ Build started: Project: ZERO_CHECK, Configuration: Release Win32 ------
------ Build started: Project: faup, Configuration: Release Win32 ------
  faup.c
..\..\..\src\tools\faup.c(5): fatal error C1083: Cannot open include file: 'unistd.h': No such file or directory
------ Build started: Project: faup_static, Configuration: Release Win32 ------
  decode.c
..\..\..\src\lib\decode.c(26): fatal error C1083: Cannot open include file: 'stdbool.h': No such file or directory
  features.c
..\..\..\src\lib\features.c(50): error C2054: expected '(' to follow 'inline'
..\..\..\src\lib\features.c(51): error C2085: 'get_last_c' : not in formal parameter list
..\..\..\src\lib\features.c(51): error C2143: syntax error : missing ';' before '{'
..\..\..\src\lib\features.c(79): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(80): error C2275: 'size_t' : illegal use of this type as an expression
          c:\program files (x86)\microsoft visual studio 10.0\vc\include\codeanalysis\sourceannotations.h(29) : see declaration of 'size_t'
..\..\..\src\lib\features.c(80): error C2146: syntax error : missing ';' before identifier 'nb_slashes'
..\..\..\src\lib\features.c(80): error C2065: 'nb_slashes' : undeclared identifier
..\..\..\src\lib\features.c(82): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(83): error C2143: syntax error : missing ';' before 'const'
..\..\..\src\lib\features.c(84): error C2065: 'ssize_t' : undeclared identifier
..\..\..\src\lib\features.c(84): error C2146: syntax error : missing ';' before identifier 'whatever_len'
..\..\..\src\lib\features.c(84): error C2065: 'whatever_len' : undeclared identifier
..\..\..\src\lib\features.c(86): error C2275: 'faup_last_slash_t' : illegal use of this type as an expression
          C:/Users/Sebastien/git/faup/src/lib/include\faup/faup.h(43) : see declaration of 'faup_last_slash_t'
..\..\..\src\lib\features.c(86): error C2146: syntax error : missing ';' before identifier 'last_slash_meaning'
..\..\..\src\lib\features.c(86): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(88): error C2065: 'ssize_t' : undeclared identifier
..\..\..\src\lib\features.c(88): error C2146: syntax error : missing ';' before identifier 'current_pos'
..\..\..\src\lib\features.c(88): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(89): error C2065: 'ssize_t' : undeclared identifier
..\..\..\src\lib\features.c(89): error C2146: syntax error : missing ';' before identifier 'buffer_pos'
..\..\..\src\lib\features.c(89): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ')' before 'type'
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(91): error C2065: 'i' : undeclared identifier
..\..\..\src\lib\features.c(91): warning C4018: '<' : signed/unsigned mismatch
..\..\..\src\lib\features.c(91): warning C4552: '<' : operator has no effect; expected operator with side-effect
..\..\..\src\lib\features.c(91): error C2065: 'i' : undeclared identifier
..\..\..\src\lib\features.c(91): error C2059: syntax error : ')'
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ';' before '{'
..\..\..\src\lib\features.c(92): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(92): error C2065: 'i' : undeclared identifier
..\..\..\src\lib\features.c(93): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(94): error C2065: 'nb_slashes' : undeclared identifier
..\..\..\src\lib\features.c(99): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(102): error C2065: 'nb_slashes' : undeclared identifier
..\..\..\src\lib\features.c(102): warning C4013: 'get_last_c' undefined; assuming extern returning int
..\..\..\src\lib\features.c(102): error C2065: 'url_o' : undeclared identifier
..\..\..\src\lib\features.c(102): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(103): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(104): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(105): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(106): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(106): error C2065: 'url_o' : undeclared identifier
..\..\..\src\lib\features.c(106): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(107): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(120): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(121): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(125): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(126): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(132): error C2065: 'last_slash_pos' : undeclared identifier
..\..\..\src\lib\features.c(132): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(135): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(138): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(142): error C2065: 'whatever_len' : undeclared identifier
..\..\..\src\lib\features.c(142): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(143): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(144): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(146): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(150): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(159): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(160): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(165): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(169): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(171): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(174): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(176): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(180): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(188): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(195): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(196): error C2065: 'current_pos' : undeclared identifier
  tld.c
..\..\..\src\lib\tld.c(17): fatal error C1083: Cannot open include file: 'netinet/in.h': No such file or directory
  tld-tree.c
..\..\..\src\lib\tld-tree.c(22): fatal error C1083: Cannot open include file: 'stdbool.h': No such file or directory
  Generating Code...
------ Build started: Project: faupl, Configuration: Release Win32 ------
  decode.c
..\..\..\src\lib\decode.c(26): fatal error C1083: Cannot open include file: 'stdbool.h': No such file or directory
  features.c
..\..\..\src\lib\features.c(50): error C2054: expected '(' to follow 'inline'
..\..\..\src\lib\features.c(51): error C2085: 'get_last_c' : not in formal parameter list
..\..\..\src\lib\features.c(51): error C2143: syntax error : missing ';' before '{'
..\..\..\src\lib\features.c(79): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(80): error C2275: 'size_t' : illegal use of this type as an expression
          c:\program files (x86)\microsoft visual studio 10.0\vc\include\codeanalysis\sourceannotations.h(29) : see declaration of 'size_t'
..\..\..\src\lib\features.c(80): error C2146: syntax error : missing ';' before identifier 'nb_slashes'
..\..\..\src\lib\features.c(80): error C2065: 'nb_slashes' : undeclared identifier
..\..\..\src\lib\features.c(82): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(83): error C2143: syntax error : missing ';' before 'const'
..\..\..\src\lib\features.c(84): error C2065: 'ssize_t' : undeclared identifier
..\..\..\src\lib\features.c(84): error C2146: syntax error : missing ';' before identifier 'whatever_len'
..\..\..\src\lib\features.c(84): error C2065: 'whatever_len' : undeclared identifier
..\..\..\src\lib\features.c(86): error C2275: 'faup_last_slash_t' : illegal use of this type as an expression
          C:/Users/Sebastien/git/faup/src/lib/include\faup/faup.h(43) : see declaration of 'faup_last_slash_t'
..\..\..\src\lib\features.c(86): error C2146: syntax error : missing ';' before identifier 'last_slash_meaning'
..\..\..\src\lib\features.c(86): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(88): error C2065: 'ssize_t' : undeclared identifier
..\..\..\src\lib\features.c(88): error C2146: syntax error : missing ';' before identifier 'current_pos'
..\..\..\src\lib\features.c(88): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(89): error C2065: 'ssize_t' : undeclared identifier
..\..\..\src\lib\features.c(89): error C2146: syntax error : missing ';' before identifier 'buffer_pos'
..\..\..\src\lib\features.c(89): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ')' before 'type'
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ';' before 'type'
..\..\..\src\lib\features.c(91): error C2065: 'i' : undeclared identifier
..\..\..\src\lib\features.c(91): warning C4018: '<' : signed/unsigned mismatch
..\..\..\src\lib\features.c(91): warning C4552: '<' : operator has no effect; expected operator with side-effect
..\..\..\src\lib\features.c(91): error C2065: 'i' : undeclared identifier
..\..\..\src\lib\features.c(91): error C2059: syntax error : ')'
..\..\..\src\lib\features.c(91): error C2143: syntax error : missing ';' before '{'
..\..\..\src\lib\features.c(92): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(92): error C2065: 'i' : undeclared identifier
..\..\..\src\lib\features.c(93): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(94): error C2065: 'nb_slashes' : undeclared identifier
..\..\..\src\lib\features.c(99): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(102): error C2065: 'nb_slashes' : undeclared identifier
..\..\..\src\lib\features.c(102): warning C4013: 'get_last_c' undefined; assuming extern returning int
..\..\..\src\lib\features.c(102): error C2065: 'url_o' : undeclared identifier
..\..\..\src\lib\features.c(102): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(103): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(104): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(105): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(106): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(106): error C2065: 'url_o' : undeclared identifier
..\..\..\src\lib\features.c(106): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(107): error C2065: 'c' : undeclared identifier
..\..\..\src\lib\features.c(120): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(121): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(125): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(126): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(132): error C2065: 'last_slash_pos' : undeclared identifier
..\..\..\src\lib\features.c(132): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(135): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(138): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(142): error C2065: 'whatever_len' : undeclared identifier
..\..\..\src\lib\features.c(142): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(143): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(144): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(146): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(150): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(159): error C2065: 'last_slash_meaning' : undeclared identifier
..\..\..\src\lib\features.c(160): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(165): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(169): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(171): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(174): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(176): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(180): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(188): error C2065: 'current_pos' : undeclared identifier
..\..\..\src\lib\features.c(195): error C2065: 'buffer_pos' : undeclared identifier
..\..\..\src\lib\features.c(196): error C2065: 'current_pos' : undeclared identifier
  tld.c
..\..\..\src\lib\tld.c(17): fatal error C1083: Cannot open include file: 'netinet/in.h': No such file or directory
  tld-tree.c
..\..\..\src\lib\tld-tree.c(22): fatal error C1083: Cannot open include file: 'stdbool.h': No such file or directory
  Generating Code...
------ Build started: Project: ALL_BUILD, Configuration: Release Win32 ------
  Build all projects
========== Build: 2 succeeded, 3 failed, 0 up-to-date, 0 skipped ==========

domain field not correctly set ?

The 2 following command lines are supposed to produce the same output except for the subdomain part (none for the first one, "aaaa" for the second).

./faup -o json http://mmtro.com/tro.js
./faup -o json http://aaaa.mmtro.com/tro.js

However the output differ and the "domain" field is not correctly set in the first case (host field is ok). In this particular case, the host and the domain are the same.

# ./faup -o json http://mmtro.com/tro.js
{
        "scheme": "http",
        "credential": "",
        "subdomain": "",
        "domain": "",
        "host": "mmtro.com",
        "tld": "com",
        "port": "",
        "resource_path": "/tro.js",
        "query_string": "",
        "fragment": "",

}

#./faup -o json http://aaaa.mmtro.com/tro.js
{
        "scheme": "http",
        "credential": "",
        "subdomain": "aaaa",
        "domain": "mmtro.com",
        "host": "aaaa.mmtro.com",
        "tld": "com",
        "port": "",
        "resource_path": "/tro.js",
        "query_string": "",
        "fragment": "",
}

Should Faup duplicate the host into the domain field ?

Remove Python TLDExtract as it's native C now

This ticket is more a reminder than an issue.

As the TLD Extraction is now native C, the python library "TLDExtract" can be removed from faup's bindings.

empty lines in stdin are completely skipped

Hello,

If you create a file that has empty lines and pipe that to faup, the output will only contain the non-empty items. The problem is that the output has a different amount of lines compared to the input.

My current usage of faup is something like this:

paste input_file <(cat input_file | cut -f 1 | faup) > output_file

Basically, I want to add additional columns to input_file based on the parsing done by faup. The unix paste command simply pastes line by line, thus the lack of the empty entries in the faup output causes misalignment.

For this usage, an empty output like ",,,,,,,,,no_tld" would be better, but I can image this would not suite everyone. Perhaps a command line switch would be necessary?

Default faup output on CentOS

running faup google.com with no parameter set on CentOS returns :

Error: unknown output option 0

opposed to the default on ubuntu (csv)

,,,google.com,google,google.com,com,,,,

faup is built the same on both, except on centOS lua is disabled.

OS X Package Missing Headers

It seems the OS X package is missing the following header files:

errors.h
handler.h
version.h

Port information skiped

This URL:
http://cache:12

Makes this output:
{
"scheme": "http",
"credential": "",
"subdomain": "",
"domain": "cache",
"domain_without_tld": "cache",
"host": "cache",
"tld": "",
"port": "",
"resource_path": "",
"query_string": "",
"fragment": ""
}

The port information is being skipped.

Faup wrong results

Pulled and compiled the current version, seems to have issues (invalid domain, invalid tld)

# echo “http://www.nonexistent.co.uk” | faup -o json
{
    "scheme": "“http",
    "credential": "",
    "subdomain": "www.nonexistent",
    "domain": "co.uk”",
    "domain_without_tld": "co",
    "host": "www.nonexistent.co.uk”",
    "tld": "uk”",
    "port": "",
    "resource_path": "",
    "query_string": "",
    "fragment": "",
    "url_type": "unknown_tld"
}

Fragment parsed incorrectly if path is implicit

The fragment isn't parsed correctly is the path is implicit. For example:

$ ./faup -o json "epsg.org#4326"
{
"scheme": "",
"credential": "",
"subdomain": "",
"domain": "epsg.org#4326",
"domain_without_tld": "epsg",
"host": "epsg.org#4326",
"tld": "org#4326",
"port": "",
"resource_path": "",
"query_string": "",
"fragment": "",
"url_type": "unknown_tld"
}

However, if the path is explicit then the fragment is parsed correctly:

$ ./faup -o json "epsg.org/#4326"
{
"scheme": "",
"credential": "",
"subdomain": "",
"domain": "epsg.org",
"domain_without_tld": "epsg",
"host": "epsg.org",
"tld": "org",
"port": "",
"resource_path": "/",
"query_string": "",
"fragment": "#4326",
"url_type": "mozilla_tld"
}

Normalization of URLs (within faup? or somewhere else) (URI and IRI)

This is not an issue as is but more notes about normalization of URLs.

Following our discussion, here is some notes regarding URL, URI and IRI normalization:

https://github.com/mitsuhiko/werkzeug/blob/master/werkzeug/urls.py (URI->IRI conversion/normalization implementation)

It seems that not everyone has the same definition of URL normalization:

https://github.com/redguardtoo/url-normalization-in-c/blob/master/src/cleanurl.c

Especially regarding "default page/index".

The best definition I found was on a Perl module (URI):

" Returns a normalized version of the URI. The rules for normalization are scheme-dependent. They usually involve lowercasing the scheme and Internet host name components, removing the explicit port specification if it matches the default port, uppercasing all escape sequences, and unescaping octets that can be better represented as plain characters. "

Error: field(domain_without_tld) with size(-1) is greater than the maximum default FAUP URL MAXLEN

No idea how I got that but it appeared in faup -w 127.0.0.1:1212 terminal:

Error: field(domain_without_tld) with size(-1) is greater than the maximum default FAUP URL MAXLEN (8192). Cannot process the given url (http://asset.rue89.com/files/imagecache/portrait/files/pictures/picture-44471.png).

Error: field(domain_without_tld) with size(-1) is greater than the maximum default FAUP URL MAXLEN (8192). Cannot process the given url (http://cache.20minutes.fr/img/diaporamas/1395/2_burnett.jpg?20110220).

Build still doesn't complete on Ubuntu 14.04

Similar to #52

$ make -k
[ 43%] Built target faup_static
[ 86%] Built target faupl
[ 93%] Built target faup
[ 96%] Built target base64
Linking C executable issue36
../lib/libfaupl.so.1: undefined reference to dlopen' ../lib/libfaupl.so.1: undefined reference todlclose'
../lib/libfaupl.so.1: undefined reference to `dlsym'
collect2: error: ld returned 1 exit status

Faup CLI stop reading files after meeting an empty line

If you have the following file containing your urls:

urls.txt:

http://www.laposte.net

www.google.co.uk

When running faup on it, the output is:

$ faup urls.txt
http,,www,laposte.net,www.laposte.net,net,,,,

Where we expect:

$ faup urls.txt
http,,www,laposte.net,www.laposte.net,net,,,,
,,www,google.co.uk,www.google.co.uk,co.uk,,,,

Fragment parsed incorrectly is multiple # characters

The fragment is parsed incorrectly if it includes a # character. This causes part of the fragment to be included in the path. Example:

$ ./faup -o json "http://penlakes.typepad.com/the_personal_blog_of_tom_/#tp_blog_of_tom_/#tp"
{
"scheme": "http",
"credential": "",
"subdomain": "penlakes",
"domain": "typepad.com",
"domain_without_tld": "typepad",
"host": "penlakes.typepad.com",
"tld": "com",
"port": "",
"resource_path": "/the_personal_blog_of_tom_/#tp_blog_of_tom_/",
"query_string": "",
"fragment": "#tp",
"url_type": "mozilla_tld"
}

Build error on OS X Lion

/tmp/furl $ cmake .
-- The C compiler identification is GNU
-- The CXX compiler identification is GNU
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Check for working C compiler: /usr/bin/gcc
-- Check for working C compiler: /usr/bin/gcc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Checking whether CXX compiler has -isysroot
-- Checking whether CXX compiler has -isysroot - yes
-- Checking whether CXX compiler supports OSX deployment target flag
-- Checking whether CXX compiler supports OSX deployment target flag - yes
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Compile mode: Release
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/furl
/tmp/furl $ make
Scanning dependencies of target furl_static
[ 14%] Building C object src/lib/CMakeFiles/furl_static.dir/decode.c.o
/tmp/furl/src/lib/decode.c: In function ‘furl_decode’:
/tmp/furl/src/lib/decode.c:140: warning: implicit declaration of function ‘memrchr’
/tmp/furl/src/lib/decode.c:140: warning: cast to pointer from integer of different size
/tmp/furl/src/lib/decode.c:154: warning: cast to pointer from integer of different size
[ 28%] Building C object src/lib/CMakeFiles/furl_static.dir/features.c.o
[ 42%] Building C object src/lib/CMakeFiles/furl_static.dir/furl.c.o
Linking C static library libfurl_static.a
[ 42%] Built target furl_static
Scanning dependencies of target furll
[ 57%] Building C object src/lib/CMakeFiles/furll.dir/decode.c.o
/tmp/furl/src/lib/decode.c: In function ‘furl_decode’:
/tmp/furl/src/lib/decode.c:140: warning: implicit declaration of function ‘memrchr’
/tmp/furl/src/lib/decode.c:140: warning: cast to pointer from integer of different size
/tmp/furl/src/lib/decode.c:154: warning: cast to pointer from integer of different size
[ 71%] Building C object src/lib/CMakeFiles/furll.dir/features.c.o
[ 85%] Building C object src/lib/CMakeFiles/furll.dir/furl.c.o
Linking C shared library libfurll.dylib
Undefined symbols for architecture x86_64:
  "_memrchr", referenced from:
      _furl_decode in decode.c.o
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
make[2]: *** [src/lib/libfurll.1.dylib] Error 1
make[1]: *** [src/lib/CMakeFiles/furll.dir/all] Error 2
make: *** [all] Error 2

random extra chars added in resource_path

See the resource_path field: extra chars are added at the end of it. Those chars cause JSON decoding errors using python json.

# curl http://127.0.0.1:1212/json?url=aHR0cDovL3d3dy5mYWNlYm9vay5jb20v
{
    "scheme": "http",
    "credential": "",
    "subdomain": "www",
    "domain": "facebook.com",
    "domain_without_tld": "facebook",
    "host": "www.facebook.com",
    "tld": "com",
    "port": "",
    "resource_path": "/??",
    "query_string": "",
    "fragment": ""
}
# curl http://127.0.0.1:1212/json?url=aHR0cDovL3d3dy5mYWNlYm9vay5jb20v
{
    "scheme": "http",
    "credential": "",
    "subdomain": "www",
    "domain": "facebook.com",
    "domain_without_tld": "facebook",
    "host": "www.facebook.com",
    "tld": "com",
    "port": "",
       "resource_path": "/?", 
    "query_string": "",
    "fragment": ""
}
# curl http://127.0.0.1:1212/json?url=aHR0cDovL3d3dy5mYWNlYm9vay5jb20v
{
    "scheme": "http",
    "credential": "",
    "subdomain": "www",
    "domain": "facebook.com",
    "domain_without_tld": "facebook",
    "host": "www.facebook.com",
    "tld": "com",
    "port": "",
    "resource_path": "/Q?",
    "query_string": "",
    "fragment": ""
}

web server don't handle correctly urls

There is an issue in the Base64 parsing, see below.

URL:

http://www.google-analytics.com/__utm.gif?utmwv=4.8.9&utmn=761050565&utmhn=www.boursorama.com&utmt=var&utmac=mars&utmcc=__utma%3D88976520.303064993334990850.1251793120.1251793120.1251793120.1%3B%2B__utmz%3D88976520.1297767034.1.1.utmcsr%3D(direct)%7Cutmccn%3D(direct)%7Cutmcmd%3D(none)%3B%2B__utmv%3D88976520.WEB-MEMBER%3B&utmu=FBAQ

Base64:

aHR0cDovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9fX3V0bS5naWY/dXRtd3Y9NC44LjkmdXRtbj03NjEwNTA1NjUmdXRtaG49d3d3LmJvdXJzb3JhbWEuY29tJnV0bXQ9dmFyJnV0bWFjPW1hcnMmdXRtY2M9X191dG1hJTNEODg5NzY1MjAuMzAzMDY0OTkzMzM0OTkwODUwLjEyNTE3OTMxMjAuMTI1MTc5MzEyMC4xMjUxNzkzMTIwLjElM0IlMkJfX3V0bXolM0Q4ODk3NjUyMC4xMjk3NzY3MDM0LjEuMS51dG1jc3IlM0QoZGlyZWN0KSU3Q3V0bWNjbiUzRChkaXJlY3QpJTdDdXRtY21kJTNEKG5vbmUpJTNCJTJCX191dG12JTNEODg5NzY1MjAuV0VCLU1FTUJFUiUzQiZ1dG11PUZCQVE=

Base64 with '/' modified to '_'

aHR0cDovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9fX3V0bS5naWY_dXRtd3Y9NC44LjkmdXRtbj03NjEwNTA1NjUmdXRtaG49d3d3LmJvdXJzb3JhbWEuY29tJnV0bXQ9dmFyJnV0bWFjPW1hcnMmdXRtY2M9X191dG1hJTNEODg5NzY1MjAuMzAzMDY0OTkzMzM0OTkwODUwLjEyNTE3OTMxMjAuMTI1MTc5MzEyMC4xMjUxNzkzMTIwLjElM0IlMkJfX3V0bXolM0Q4ODk3NjUyMC4xMjk3NzY3MDM0LjEuMS51dG1jc3IlM0QoZGlyZWN0KSU3Q3V0bWNjbiUzRChkaXJlY3QpJTdDdXRtY21kJTNEKG5vbmUpJTNCJTJCX191dG12JTNEODg5NzY1MjAuV0VCLU1FTUJFUiUzQiZ1dG11PUZCQVE=

# curl http://127.0.0.1:1234/json?url=aHR0cDovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9fX3V0bS5naWY_dXRtd3Y9NC44LjkmdXRtbj03NjEwNTA1NjUmdXRtaG49d3d3LmJvdXJzb3JhbWEuY29tJnV0bXQ9dmFyJnV0bWFjPW1hcnMmdXRtY2M9X191dG1hJTNEODg5NzY1MjAuMzAzMDY0OTkzMzM0OTkwODUwLjEyNTE3OTMxMjAuMTI1MTc5MzEyMC4xMjUxNzkzMTIwLjElM0IlMkJfX3V0bXolM0Q4ODk3NjUyMC4xMjk3NzY3MDM0LjEuMS51dG1jc3IlM0QoZGlyZWN0KSU3Q3V0bWNjbiUzRChkaXJlY3QpJTdDdXRtY21kJTNEKG5vbmUpJTNCJTJCX191dG12JTNEODg5NzY1MjAuV0VCLU1FTUJFUiUzQiZ1dG11PUZCQVE=

{
    "scheme": "http",
    "credential": "",
    "subdomain": "www",
    "domain": "google-analytics.com",
    "domain_without_tld": "google-analytics",
    "host": "www.google-analytics.com",
    "tld": "com",
    "port": "",
    "resource_path": "/__utm.gif",
    "query_string": "",
    "fragment": "",
}

I have the very same output if I change "_" (underscore) to "-" (minus). In fact, the very same thing as:

curl http://127.0.0.1:1234/json?url=aHR0cDovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9fX3V0bS5naWY-
curl http://127.0.0.1:1234/json?url=aHR0cDovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9fX3V0bS5naWY_
curl http://127.0.0.1:1234/json?url=aHR0cDovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9fX3V0bS5naWY+
curl http://127.0.0.1:1234/json?url=aHR0cDovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9fX3V0bS5naWY.
curl http://127.0.0.1:1234/json?url=aHR0cDovL3d3dy5nb29nbGUtYW5hbHl0aWNzLmNvbS9fX3V0bS5naWYX
.. etc (the last char is only padding).

Buffer overrun in tld.c

There is a buffer overrun bug in tld.c, lines 96 and 97:

while ( (n=recv(sockfd, recvbuf, MAX_RECVBUF, 0)) > 0) {
    recvbuf[n] = '\0';

If n==MAX_RECVBUF then the assignment of a null-terminator will be one byte past the end of recvbuf. This consistently causes a segfault in my process that links against the static build of libfaup. The buffer overrun can be easily fixed reducing the max read length by 1 byte:

while ( (n=recv(sockfd, recvbuf, MAX_RECVBUF - 1, 0)) > 0) {
    recvbuf[n] = '\0';

faup debian package

If the faup library has been compiled with lua support and a debian package is created thanks to cmake, no dependency to the corresponding lua package is set in the faup one.

The CPACK_DEBIAN_PACKAGE_DEPENDS should be set in such a case !

python bindinds are outdated

The python bindinds need to be updated to reflect the domain_without_tld field (at least functions.py and faup.py).
Thanks

Backslashes not parsed as part of path

The backslash character is not parsed as part of the path. Here are examples:

$ ./faup -o json "http://github.com\contact"
{
"scheme": "http",
"credential": "",
"subdomain": "",
"domain": "github.com\contact",
"domain_without_tld": "github",
"host": "github.com\contact",
"tld": "com\contact",
"port": "",
"resource_path": "",
"query_string": "",
"fragment": "",
"url_type": "unknown_tld"
}

$ ./faup -o json "file://c:\program.exe"
{
"scheme": "file",
"credential": "",
"subdomain": "",
"domain": "c:\program.exe",
"domain_without_tld": "c:\program",
"host": "c:\program.exe",
"tld": "exe",
"port": "",
"resource_path": "",
"query_string": "",
"fragment": "",
"url_type": "unknown_tld"
}

Python3 with tld > 1 extraction

stricaud-mbp15-1:python stricaud$ python3 test.py
/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/importlib/_bootstrap.py:313: UserWarning: Module pyfaup was already imported from /Users/stricaud/scm/git/faup/src/lib/bindings/python/pyfaup/init.py, but /opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/pyfaup-1.0-py3.3.egg is being added to sys.path
return f(_args, *_kwds)
We decode the url: b'http://www.wallinfire.co.uk'
error reading TLD cache file /Users/stricaud/scm/git/faup/src/lib/bindings/python/pyfaup/.tld_set: 'str' does not support the buffer interface
Traceback (most recent call last):
File "test.py", line 11, in
data = f.get()
File "/Users/stricaud/scm/git/faup/src/lib/bindings/python/pyfaup/faup.py", line 145, in get
retval["subdomain"] = self.get_subdomain()
File "/Users/stricaud/scm/git/faup/src/lib/bindings/python/pyfaup/faup.py", line 61, in get_subdomain
self.tldex._extract(self.get_host())
File "/Users/stricaud/scm/git/faup/src/lib/bindings/python/pyfaup/tldextract.py", line 94, in _extract
registered_domain, tld = self._get_tld_extractor().extract(host)
File "/Users/stricaud/scm/git/faup/src/lib/bindings/python/pyfaup/tldextract.py", line 121, in _get_tld_extractor
tlds = frozenset(binascii.a2b_qp(tld) for tld_source in tld_sources for tld in tld_source())
File "/Users/stricaud/scm/git/faup/src/lib/bindings/python/pyfaup/tldextract.py", line 121, in
tlds = frozenset(binascii.a2b_qp(tld) for tld_source in tld_sources for tld in tld_source())
ValueError: string argument should contain only ASCII characters
stricaud-mbp15-1:python stricaud$

Python packaging

Where is faup on pypy ;-) ?

Valgrind memory leak reported

output for : valgrind --tool=memcheck --track-origins=yes --leak-check=full faup http://www.cache.google.co.uk

**300 (60 direct, 240 indirect) bytes in 1 blocks are definitely lost in loss record 11 of 11**
==7466==    at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7466==    by 0x5377CD4: nss_parse_service_list (nsswitch.c:678)
==7466==    by 0x5378795: __nss_database_lookup (nsswitch.c:175)
==7466==    by 0x6163623: ???
==7466==    by 0x533144C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256)
==7466==    by 0x5330D42: getpwuid (getXXbyYY.c:117)
==7466==    by 0x4E48396: faup_datadir_home_file_or_dir_exists (datadir.c:40)
==7466==    by 0x4E4869C: faup_datadir_get_file (datadir.c:132)
==7466==    by 0x4E4B221: faup_tld_array_populate (tld.c:137)
==7466==    by 0x4E4B94F: faup_tld_tree_new (tld-tree.c:246)
==7466==    by 0x4E499CF: faup_options_enable_tld_above_one (options.c:71)
==7466==    by 0x4E49A2C: faup_options_defaults (options.c:63)

and for some reason at my application I get one extra:

==5812== **Invalid read of size 1**
==5812==    at 0x5068BB7: faup_tld_tree_extract (tld-tree.c:295)
==5812==    by 0x506604D: faup_decode (decode.c:225)
==5812==    by 0x405947: main (tests.cpp:195)
==5812==  Address 0x6eceb98 is 2 bytes after a block of size 54 alloc'd
==5812==    at 0x4C2AF8E: operator new(unsigned long) (in /usr/lib/valgrind/vgpreload_memcheck-    amd64-linux.so)
==5812==    by 0x532E3B8: std::string::_Rep::_S_create(unsigned long, unsigned long,     std::allocator<char> const&) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17)
==5812==    by 0x532FD94: char* std::string::_S_construct<char const*>(char const*, char const*,     std::allocator<char> const&, std::forward_iterator_tag) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17)
==5812==    by 0x532FE72: std::basic_string<char, std::char_traits<char>, std::allocator<char>     >::basic_string(char const*, std::allocator<char> const&) (in /usr/lib/x86_64-linux-    gnu/libstdc++.so.6.0.17)
==5812==    by 0x40631E: std::pair<std::string, std::string>::pair<char const*, char const*, void>    (std::pair<char const*, char const*>&&) (in /tests)
==5812==    by 0x405868: main (tests.cpp:137)

the code responsible in my code:

faup_options_t *FAUP_OPTS  = faup_options_new();
init_faup_options(FAUP_OPTS);
string url="http://www.cache.google.co.uk";
faup_handler_t *fh=faup_init(FAUP_OPTS);
faup_decode( fh, url.c_str(), url.size() );
faup_terminate(fh);

I suspect the difference comes from me using the std::string library.
I runned it in a loop and it doesn't seem to affect anything, my app remained in 1.5 MB.
I don't know what to make of it, so here I am reporting :).

Doesn't build out-of-the-box on Ubuntu 14.04LTS

~/workspace/faup/__build $ cmake ..
-- The C compiler identification is GNU 4.8.2
-- The CXX compiler identification is GNU 4.8.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
CMake system name: Linux
-- Compile mode: Release
-- Compile with Web Server
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.26")
-- Could NOT find Lua51 (missing: LUA_LIBRARIES LUA_INCLUDE_DIR)
CMake system: Linux
Discovered distribution type: debian
-- Configuring done
-- Generating done
-- Build files have been written to: ~/workspace/faup/__build

~/workspace/faup/__build $ make
...
[ 43%] Built target faup_static
[ 86%] Built target faupl
Linking C executable faup
../lib/libfaupl.so.1: undefined reference to dlopen' ../lib/libfaupl.so.1: undefined reference todlclose'
../lib/libfaupl.so.1: undefined reference to `dlsym'
collect2: error: ld returned 1 exit status
make[2]: *** [src/tools/faup] Error 1
make[1]: *** [src/tools/CMakeFiles/faup.dir/all] Error 2
make: *** [all] Error 2

Probably needs a -ldl somewhere

Feature request/idea: character encoding detection in furl

Just a small feature request within furl. Would it possible to add a functionality to detect the encoding
of an URL? Something like chardet (http://pypi.python.org/pypi/chardet) to provide a list of potential encoding scheme for the url and each of its element? or this is outside the scope of furl?

Thank you.

supporting idn

Taken from clickstefan in issue #63:

Also another idea is to have idn integrated, I use it to normalize domains, as it helps with :

adding domains to a database, it's nice to not have duplicates and ascii is more compatible with most net tools.
helps in validating domains (after idn you can check for non [A-Z0-9.-] characters in the domain and invalidate)

The chrome url library does a good job at normalization, but it's stuffed with too many options and dependencies.

Fix valgrind reported issues

$ valgrind -v --leak-check=full ./src/tools/faup www.google.fr
==3715== Memcheck, a memory error detector
==3715== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.
==3715== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info
==3715== Command: ./src/tools/faup www.google.fr
==3715==
--3715-- Valgrind options:
--3715-- --suppressions=/usr/lib/valgrind/debian-libc6-dbg.supp
--3715-- -v
--3715-- --leak-check=full
--3715-- Contents of /proc/version:
--3715-- Linux version 3.5.0-28-generic (buildd@panlong) (gcc version 4.7.2 (Ubuntu/Linaro 4.7.2-2ubuntu1) ) #48-Ubuntu SMP Tue Apr 23 23:03:38 UTC 2013
--3715-- Arch and hwcaps: AMD64, amd64-sse3-cx16
--3715-- Page sizes: currently 4096, max supported 4096
--3715-- Valgrind library directory: /usr/lib/valgrind
--3715-- Reading syms from /home/sebastien/scm/git/faup/build/src/tools/faup (0x400000)
--3715-- Reading syms from /lib/x86_64-linux-gnu/ld-2.15.so (0x4000000)
--3715-- Considering /lib/x86_64-linux-gnu/ld-2.15.so ..
--3715-- .. CRC mismatch (computed af0db78b wanted 3003bdf7)
--3715-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/ld-2.15.so ..
--3715-- .. CRC is valid
--3715-- Reading syms from /usr/lib/valgrind/memcheck-amd64-linux (0x38000000)
--3715-- Considering /usr/lib/valgrind/memcheck-amd64-linux ..
--3715-- .. CRC mismatch (computed 055ede9a wanted 763f0405)
--3715-- object doesn't have a symbol table
--3715-- object doesn't have a dynamic symbol table
--3715-- Reading suppressions file: /usr/lib/valgrind/debian-libc6-dbg.supp
--3715-- Reading suppressions file: /usr/lib/valgrind/default.supp
==3715== embedded gdbserver: reading from /tmp/vgdb-pipe-from-vgdb-to-3715-by-sebastien-on-???
==3715== embedded gdbserver: writing to /tmp/vgdb-pipe-to-vgdb-from-3715-by-sebastien-on-???
==3715== embedded gdbserver: shared mem /tmp/vgdb-pipe-shared-mem-vgdb-3715-by-sebastien-on-???
==3715==
==3715== TO CONTROL THIS PROCESS USING vgdb (which you probably
==3715== don't want to do, unless you know exactly what you're doing,
==3715== or are doing some strange experiment):
==3715== /usr/lib/valgrind/../../bin/vgdb --pid=3715 ...command...
==3715==
==3715== TO DEBUG THIS PROCESS USING GDB: start GDB like this
==3715== /path/to/gdb ./src/tools/faup
==3715== and then give GDB the following command
==3715== target remote | /usr/lib/valgrind/../../bin/vgdb --pid=3715
==3715== --pid is optional if only one valgrind process is running
==3715==
--3715-- REDIR: 0x40189f0 (strlen) redirected to 0x38061e47 (???)
--3715-- Reading syms from /usr/lib/valgrind/vgpreload_core-amd64-linux.so (0x4a25000)
--3715-- Considering /usr/lib/valgrind/vgpreload_core-amd64-linux.so ..
--3715-- .. CRC mismatch (computed c9cf2ff8 wanted 4212d7a4)
--3715-- object doesn't have a symbol table
--3715-- Reading syms from /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so (0x4c27000)
--3715-- Considering /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so ..
--3715-- .. CRC mismatch (computed 70153ef6 wanted 49c35013)
--3715-- object doesn't have a symbol table
--3715-- REDIR: 0x4018860 (index) redirected to 0x4c2b970 (index)
--3715-- REDIR: 0x40188e0 (strcmp) redirected to 0x4c2c8d0 (strcmp)
--3715-- Reading syms from /home/sebastien/scm/git/faup/build/src/lib/libfaupl.so.1 (0x4e32000)
--3715-- Reading syms from /lib/x86_64-linux-gnu/libc-2.15.so (0x503a000)
--3715-- Considering /lib/x86_64-linux-gnu/libc-2.15.so ..
--3715-- .. CRC mismatch (computed d3262300 wanted e70c74da)
--3715-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/libc-2.15.so ..
--3715-- .. CRC is valid
--3715-- REDIR: 0x50c6e40 (strcasecmp) redirected to 0x4a256fe (_vgnU_ifunc_wrapper)
--3715-- REDIR: 0x50c31e0 (strnlen) redirected to 0x4a256fe (_vgnU_ifunc_wrapper)
--3715-- REDIR: 0x50c9110 (strncasecmp) redirected to 0x4a256fe (_vgnU_ifunc_wrapper)
--3715-- REDIR: 0x50c4bd0 (__GI_strrchr) redirected to 0x4c2b730 (__GI_strrchr)
--3715-- REDIR: 0x50be2c0 (calloc) redirected to 0x4c29d7e (calloc)
--3715-- REDIR: 0x50c3100 (__GI_strlen) redirected to 0x4c2bd00 (__GI_strlen)
--3715-- REDIR: 0x50bcf50 (malloc) redirected to 0x4c2b376 (malloc)
--3715-- REDIR: 0x50c52b0 (memchr) redirected to 0x4c2c970 (memchr)
--3715-- REDIR: 0x50ccd10 (strchrnul) redirected to 0x4c2e000 (strchrnul)
--3715-- REDIR: 0x516ff90 (__strncasecmp_sse42) redirected to 0x4c2c320 (strncasecmp)
--3715-- REDIR: 0x50bd560 (free) redirected to 0x4c2a6b2 (free)
--3715-- REDIR: 0x50c1600 (__GI_strcmp) redirected to 0x4c2c880 (__GI_strcmp)
--3715-- REDIR: 0x50c6ce0 (__GI_stpcpy) redirected to 0x4c2da50 (__GI_stpcpy)
--3715-- Reading syms from /lib/x86_64-linux-gnu/libnss_compat-2.15.so (0x57f9000)
--3715-- Considering /lib/x86_64-linux-gnu/libnss_compat-2.15.so ..
--3715-- .. CRC mismatch (computed b6f12953 wanted 0113a9dd)
--3715-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/libnss_compat-2.15.so ..
--3715-- .. CRC is valid
--3715-- Reading syms from /lib/x86_64-linux-gnu/libnsl-2.15.so (0x5a02000)
--3715-- Considering /lib/x86_64-linux-gnu/libnsl-2.15.so ..
--3715-- .. CRC mismatch (computed 542c013d wanted 22ff9327)
--3715-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/libnsl-2.15.so ..
--3715-- .. CRC is valid
--3715-- REDIR: 0x50c2a90 (__GI_strcpy) redirected to 0x4c2bdf0 (__GI_strcpy)
--3715-- Reading syms from /lib/x86_64-linux-gnu/libnss_nis-2.15.so (0x5c1c000)
--3715-- Considering /lib/x86_64-linux-gnu/libnss_nis-2.15.so ..
--3715-- .. CRC mismatch (computed b2276ac8 wanted 9efc4598)
--3715-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/libnss_nis-2.15.so ..
--3715-- .. CRC is valid
--3715-- Reading syms from /lib/x86_64-linux-gnu/libnss_files-2.15.so (0x5e28000)
--3715-- Considering /lib/x86_64-linux-gnu/libnss_files-2.15.so ..
--3715-- .. CRC mismatch (computed 1725be68 wanted a6c42a2d)
--3715-- Considering /usr/lib/debug/lib/x86_64-linux-gnu/libnss_files-2.15.so ..
--3715-- .. CRC is valid
--3715-- REDIR: 0x50c1540 (__GI_strchr) redirected to 0x4c2b830 (__GI_strchr)
--3715-- REDIR: 0x50c5c20 (memset) redirected to 0x4a256fe (_vgnU_ifunc_wrapper)
--3715-- REDIR: 0x50c5c60 (__GI_memset) redirected to 0x4c2dcc0 (memset)
--3715-- REDIR: 0x50c30b0 (strlen) redirected to 0x4a256fe (_vgnU_ifunc_wrapper)
--3715-- REDIR: 0x519cc00 (__strlen_sse2_pminub) redirected to 0x4c2bce0 (strlen)
--3715-- REDIR: 0x50bd660 (realloc) redirected to 0x4c2b427 (realloc)
==3715== Invalid read of size 1
==3715== at 0x4E369BA: _faup_tld_tree_add_node (tld-tree.c:81)
==3715== by 0x4E36BFD: faup_tld_tree_add_node (tld-tree.c:161)
==3715== by 0x4E36C3D: faup_tld_tree_add_tld (tld-tree.c:172)
==3715== by 0x4E36699: faup_tld_array_foreach (tld.c:243)
==3715== by 0x4E36CB2: faup_tld_tree_new (tld-tree.c:205)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715== Address 0x53fd28f is 1 bytes before a block of size 5 alloc'd
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E35C57: utarray_str_cpy (utarray.h:221)
==3715== by 0x4E364C3: faup_tld_array_populate (tld.c:216)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715== Invalid read of size 1
==3715== at 0x4E369CD: _faup_tld_tree_add_node (tld-tree.c:82)
==3715== by 0x4E36BFD: faup_tld_tree_add_node (tld-tree.c:161)
==3715== by 0x4E36C3D: faup_tld_tree_add_tld (tld-tree.c:172)
==3715== by 0x4E36699: faup_tld_array_foreach (tld.c:243)
==3715== by 0x4E36CB2: faup_tld_tree_new (tld-tree.c:205)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715== Address 0x53fd28f is 1 bytes before a block of size 5 alloc'd
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E35C57: utarray_str_cpy (utarray.h:221)
==3715== by 0x4E364C3: faup_tld_array_populate (tld.c:216)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715== Invalid read of size 1
==3715== at 0x4E369E0: _faup_tld_tree_add_node (tld-tree.c:83)
==3715== by 0x4E36BFD: faup_tld_tree_add_node (tld-tree.c:161)
==3715== by 0x4E36C3D: faup_tld_tree_add_tld (tld-tree.c:172)
==3715== by 0x4E36699: faup_tld_array_foreach (tld.c:243)
==3715== by 0x4E36CB2: faup_tld_tree_new (tld-tree.c:205)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715== Address 0x53fd28f is 1 bytes before a block of size 5 alloc'd
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E35C57: utarray_str_cpy (utarray.h:221)
==3715== by 0x4E364C3: faup_tld_array_populate (tld.c:216)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715== Invalid read of size 1
==3715== at 0x4E36AF6: _faup_tld_tree_add_node (tld-tree.c:79)
==3715== by 0x4E36BFD: faup_tld_tree_add_node (tld-tree.c:161)
==3715== by 0x4E36C3D: faup_tld_tree_add_tld (tld-tree.c:172)
==3715== by 0x4E36699: faup_tld_array_foreach (tld.c:243)
==3715== by 0x4E36CB2: faup_tld_tree_new (tld-tree.c:205)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715== Address 0x53fd28f is 1 bytes before a block of size 5 alloc'd
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E35C57: utarray_str_cpy (utarray.h:221)
==3715== by 0x4E364C3: faup_tld_array_populate (tld.c:216)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
,,www,google.fr,www.google.fr,fr,,,,
--3715-- Discarding syms at 0x57fa260-0x57ffbc8 in /lib/x86_64-linux-gnu/libnss_compat-2.15.so due to munmap()
--3715-- Discarding syms at 0x5c1e0b0-0x5c24718 in /lib/x86_64-linux-gnu/libnss_nis-2.15.so due to munmap()
--3715-- Discarding syms at 0x5a06060-0x5a13878 in /lib/x86_64-linux-gnu/libnsl-2.15.so due to munmap()
--3715-- Discarding syms at 0x5e2a140-0x5e31a08 in /lib/x86_64-linux-gnu/libnss_files-2.15.so due to munmap()
==3715==
==3715== HEAP SUMMARY:
==3715== in use at exit: 731,098 bytes in 33,401 blocks
==3715== total heap usage: 39,605 allocs, 6,204 frees, 949,974 bytes allocated
==3715==
==3715== Searching for pointers to 33,401 not-freed blocks
==3715== Checked 82,024 bytes
==3715==
==3715== 35 bytes in 1 blocks are definitely lost in loss record 13 of 18
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50AD157: vasprintf (vasprintf.c:78)
==3715== by 0x508DA96: asprintf (asprintf.c:37)
==3715== by 0x4E35FD0: faup_tld_home_file_exists (tld.c:100)
==3715== by 0x4E360FA: faup_tld_get_file (tld.c:139)
==3715== by 0x4E36248: faup_tld_array_populate (tld.c:189)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715== 300 (60 direct, 240 indirect) bytes in 1 blocks are definitely lost in loss record 14 of 18
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x513F5A4: nss_parse_service_list (nsswitch.c:678)
==3715== by 0x5140065: __nss_database_lookup (nsswitch.c:175)
==3715== by 0x57FD623: ???
==3715== by 0x50F8C0C: getpwuid_r@@GLIBC_2.2.5 (getXXbyYY_r.c:256)
==3715== by 0x50F8502: getpwuid (getXXbyYY.c:117)
==3715== by 0x4E35F8F: faup_tld_home_file_exists (tld.c:96)
==3715== by 0x4E360FA: faup_tld_get_file (tld.c:139)
==3715== by 0x4E36248: faup_tld_array_populate (tld.c:189)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715==
==3715== 76,523 bytes in 6,129 blocks are definitely lost in loss record 15 of 18
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E363B2: faup_tld_array_populate (tld.c:215)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715== 654,240 (24 direct, 654,216 indirect) bytes in 1 blocks are definitely lost in loss record 18 of 18
==3715== at 0x4C29E46: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x4E36C81: faup_tld_tree_new (tld-tree.c:199)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715== LEAK SUMMARY:
==3715== definitely lost: 76,642 bytes in 6,132 blocks
==3715== indirectly lost: 654,456 bytes in 27,269 blocks
==3715== possibly lost: 0 bytes in 0 blocks
==3715== still reachable: 0 bytes in 0 blocks
==3715== suppressed: 0 bytes in 0 blocks
==3715==
==3715== ERROR SUMMARY: 24520 errors from 8 contexts (suppressed: 2 from 2)
==3715==
==3715== 6129 errors in context 1 of 8:
==3715== Invalid read of size 1
==3715== at 0x4E36AF6: _faup_tld_tree_add_node (tld-tree.c:79)
==3715== by 0x4E36BFD: faup_tld_tree_add_node (tld-tree.c:161)
==3715== by 0x4E36C3D: faup_tld_tree_add_tld (tld-tree.c:172)
==3715== by 0x4E36699: faup_tld_array_foreach (tld.c:243)
==3715== by 0x4E36CB2: faup_tld_tree_new (tld-tree.c:205)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715== Address 0x53fd28f is 1 bytes before a block of size 5 alloc'd
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E35C57: utarray_str_cpy (utarray.h:221)
==3715== by 0x4E364C3: faup_tld_array_populate (tld.c:216)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715==
==3715== 6129 errors in context 2 of 8:
==3715== Invalid read of size 1
==3715== at 0x4E369E0: _faup_tld_tree_add_node (tld-tree.c:83)
==3715== by 0x4E36BFD: faup_tld_tree_add_node (tld-tree.c:161)
==3715== by 0x4E36C3D: faup_tld_tree_add_tld (tld-tree.c:172)
==3715== by 0x4E36699: faup_tld_array_foreach (tld.c:243)
==3715== by 0x4E36CB2: faup_tld_tree_new (tld-tree.c:205)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715== Address 0x53fd28f is 1 bytes before a block of size 5 alloc'd
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E35C57: utarray_str_cpy (utarray.h:221)
==3715== by 0x4E364C3: faup_tld_array_populate (tld.c:216)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715==
==3715== 6129 errors in context 3 of 8:
==3715== Invalid read of size 1
==3715== at 0x4E369CD: _faup_tld_tree_add_node (tld-tree.c:82)
==3715== by 0x4E36BFD: faup_tld_tree_add_node (tld-tree.c:161)
==3715== by 0x4E36C3D: faup_tld_tree_add_tld (tld-tree.c:172)
==3715== by 0x4E36699: faup_tld_array_foreach (tld.c:243)
==3715== by 0x4E36CB2: faup_tld_tree_new (tld-tree.c:205)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715== Address 0x53fd28f is 1 bytes before a block of size 5 alloc'd
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E35C57: utarray_str_cpy (utarray.h:221)
==3715== by 0x4E364C3: faup_tld_array_populate (tld.c:216)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
==3715==
==3715== 6129 errors in context 4 of 8:
==3715== Invalid read of size 1
==3715== at 0x4E369BA: _faup_tld_tree_add_node (tld-tree.c:81)
==3715== by 0x4E36BFD: faup_tld_tree_add_node (tld-tree.c:161)
==3715== by 0x4E36C3D: faup_tld_tree_add_tld (tld-tree.c:172)
==3715== by 0x4E36699: faup_tld_array_foreach (tld.c:243)
==3715== by 0x4E36CB2: faup_tld_tree_new (tld-tree.c:205)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715== Address 0x53fd28f is 1 bytes before a block of size 5 alloc'd
==3715== at 0x4C2B3F8: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==3715== by 0x50C2D81: strdup (strdup.c:43)
==3715== by 0x4E35C57: utarray_str_cpy (utarray.h:221)
==3715== by 0x4E364C3: faup_tld_array_populate (tld.c:216)
==3715== by 0x4E36C9F: faup_tld_tree_new (tld-tree.c:204)
==3715== by 0x4E351E7: faup_options_enable_tld_above_one (options.c:58)
==3715== by 0x4E351BE: faup_options_defaults (options.c:50)
==3715== by 0x401097: main (faup.c:86)
==3715==
--3715--
--3715-- used_suppression: 2 dl-hack3-cond-1
==3715==
==3715== ERROR SUMMARY: 24520 errors from 8 contexts (suppressed: 2 from 2)

Faup lookup in distributed environments

When the Faup App is deployed in distributed environment the following error appears in the logs and nothing happen (= faup is not launched).

2014-06-06 19:05:28,755 INFO faup_r.std_err=/opt/splunk/etc/apps/faup/opt/faup-linux: line 5: /opt/splunk/var/run/searchpeers/sh.local-1402073006/apps/faup/bin/../opt/linux-x86_64/bin/faup: No such file or directory

This error may be due to the way the faup binary is called (with the relative path that include "../").

Is it possible to have a flag set when the tld doesn't match the list of tlds from Mozilla ?

It would help to invalidate bad links

Lua libraries

git clone git://github.com/stricaud/faup.git
cd faup
mkdir build
cd build
cmake .. && make
sudo make install

Could NOT find Lua51 (missing: LUA_LIBRARIES LUA_INCLUDE_DIR)

Almost that, I encountered one error with cmake Lua51 was missing.
So maybe just add in the Readme "install last lua and lua-dev libraries"?

PS: pedobear send you many greatings :)

Path parsing

the path includes the "?" character

e.g.:
http://cache.google.com/?q=http://www.google.com/path/to/search.html?q=test&loc=ro%20Romania
scheme: http
subDomain: cache
topDomain: google.com
domain: cache.google.com
tld: com
path: /?q=http://www.google.com/path/to/search.html
query: ?q=test&loc=ro%20Romania

expected:
Path: /
Query: ?q=http://www.google.com/path/to/search.html?q=test&loc=ro%20Romania

see http://www.freeformatter.com/url-parser-query-string-splitter.html

Time for a new release?

@stricaud, are there any plans/roadmap for the next release (1.6, I guess)?
There seems to be quite a lot of changes to master since 1.5 release.

Thank you,

Downloading the list fails

GET http://io.libio.so/effective_tld_names.dat HTTP/1.0

HTTP/1.1 404 Not Found
Date: Fri, 29 Aug 2014 08:57:16 GMT
Server: Apache/2.4.10 (Debian)
Keep-Alive: timeout=5, max=100
Content-Type: text/html; charset=iso-8859-1
Content-Length: 304
Accept-Ranges: none
Connection: close

<title>404 Not Found</title>

Not Found

The requested URL /effective_tld_names.dat was not found on this server.

Apache/2.4.10 (Debian) Server at ns3362673.ovh.net Port 80

This is not how HTTP works. If you connect directly to port 80 on the server, then the request should be

GET /effective_tld_names.dat HTTP/1.0

However, io.libio.so seems to be (mis)configurede to return 404 if the Host: header is not present.

Faup lookup on Splunk Base run Faup 2 times!

See pull request.

stricaud / faup Goto Github PK

faup's People

Contributors

Stargazers

Watchers

Forkers

faup's Issues

Best Regards,

Could NOT find Lua51 (missing: LUA_LIBRARIES LUA_INCLUDE_DIR)

Not Found

Recommend Projects

Recommend Topics

Recommend Org