hrbrmstr / drill-url-tools Goto Github PK

With big dataset (>1000000 rows) url_parse produce unexpected results.
Example:
SELECT * FROM ( SELECT R.Url, R.f.host as host FROM ( SELECT T.Url, url_parse(T.Url)) AS f FROM ... AS T) AS R) ) WHERE host LIKE '80to%' LIMIT 1;
+----------------------------------------------------------------------------------+-----------------+
| Url | host |
+----------------------------------------------------------------------------------+-----------------+
| http://patriot-auto.ru/addnews.html | 80togh-pro.rozb |
+----------------------------------------------------------------------------------+-----------------+

Although
SELECT t.p.host FROM (SELECT url_parse('http://patriot-auto.ru/addnews.html') AS p) AS t;
+-----------------+
| EXPR$0 |
+-----------------+
| patriot-auto.ru |
+-----------------+

Find in the source
buffer.reallocIfNeeded(...);
Instead of
buffer = buffer.reallocIfNeeded(...);

The size of the buffer is unchanged and some problems may appears (some of these problems are currently catched by the "try/catch" in the code but produce bad results.

Examples
`
apache drill> select url_parse('http://www.gooogle.com.uk') /* OK */;
+----------------------------------------------------------------------------------+
| EXPR$0 |
+----------------------------------------------------------------------------------+
| {"scheme":"http","username":"","host":"www.gooogle.com.uk","port":"80","path":"/"} |
+----------------------------------------------------------------------------------+

apache drill> select url_parse('http://www.go--oogle.com.uk') /* NOK */;
+--------+
| EXPR$0 |
+--------+
| {} |
+--------+

apache drill> select url_parse('http://www.g--ooogle.com.uk') /* OK */;
+----------------------------------------------------------------------------------+
| EXPR$0 |
+----------------------------------------------------------------------------------+
| {"scheme":"http","username":"","host":"www.g--ooogle.com.uk","port":"80","path":"/"} |
+----------------------------------------------------------------------------------+

apache drill> select url_parse('http://www.goo--ogle.com.uk') /* OK */;
+----------------------------------------------------------------------------------+
| EXPR$0 |
+----------------------------------------------------------------------------------+
| {"scheme":"http","username":"","host":"www.goo--ogle.com.uk","port":"80","path":"/"} |
+----------------------------------------------------------------------------------+

apache drill> select url_parse('http://www.gooogle.co--m.uk') /* NOK */ ;
+--------+
| EXPR$0 |
+--------+
| {} |
+--------+

apache drill> select url_parse('http://tv--portal.blogspot.de/2011/07/tvn-en-vivo-panama.html') /* NOK */;
+--------+
| EXPR$0 |
+--------+
| {} |
+--------+
`