add "convert_date_format" and other possible date conversion utilities,
where it is not provided by the hive distro

add a UDF which will balance keys across a number of regions.

Size of sketch sets are now hard-coded to 5000.
parameterize size of sketch set , using const parameter.

NullPointerException occurs if input is null

The to_json and from_json should handle special characters, and escape/unescape as needed.

integrate with hive-pdk, so that we can add unit tests

Add serde for accessing Flume files with embedded JSON

fix collect_max to support any primitive types as keys ( and any numeric as values )

map_index() needs to be generic UDF to support maps of different types.

right now our naming conventions are somewhat confusing and inconsistent.

We should change these to be more intuitive.

Right now ...
collect ... UDAFs for converting primitive to list or map
combine ... UDFs for manipulating lists and arrays
union ... UDAFs for grouping lists and maps together in various ways

At moment HBaseFactory.getHTable(...) considers only

hbase.zookeeper.quorum

and ignores rest, which breaks functionality when user wants to specify
zookeeper.znode.parent

for example.

Need collect_min for completeness sake.. Maybe some code-reuse is possible.

Eventually should allow a comparison function to be passed in somehow ( ie. HoneyDog )

Make a consistent policy of returning null or empty set, and avoid NPE's for various UDAF's.

Make consistent with mysql

Add a UDF for the sketch_set package to calculate jaccard distance, and demonstrate how how to measure set similarity ...

Actually passing in long arrays of values can be expensive with non-generic UDFs, because of object conversions. Add generic UDF's of some common array and map operations which avoid coercion to java primitives for more efficient access

from_json is a stand-in replacement for json_map and split_json, and offers the same functionality.

We should either deprecate these functions; or keep them as convenience functions, but allow the type of the value to be specified somehow.

This stack trace popped up in warehouse version of join_array... Might be present in brickhouse version
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.hadoop.io.Text
at org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.getPrimitiveJavaObject(WritableStringObjectInspector.java:45)
at com.klout.analytics.warehouse.udf.JoinArrayUDF.evaluate(JoinArrayUDF.java:28)
at com.klout.analytics.warehouse.udf.JoinArrayUDF.evaluate(JoinArrayUDF.java:39)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator$DeferredExprObject.get(ExprNodeGenericFuncEvaluator.java:64)
at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:177)
at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator.evaluate(ExprNodeGenericFuncEvaluator.java:163)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:76)

Right now there is one column family "c" which is assumed for hbase tables inserted into with the HBase udf's . We should generalize this somehow, and allow to specify

deprecate intersect_array

Replace the current bloom filters functions with one from com.clearspring stream lib jar.

Also, use const arguments to define precision.

Array_index could support -1 as a value, to emulate Ruby style arrays and return the last value of the array, to be more general then last_index() and first_index() functions.

Also support getting a range of values, ie array_index( 3,6) which returns an array of values instead of a single value.

add flag for a "safe" mode for json, where strings contain encoded json ...

tinyint is missing

JSONS produced by to string are not properly escaped, which causes issues when strings have characters that require escaping.

A lot of developers use AWS these days, and they don't have the opportunity to add custom jars to their Hive environment. To improve adoption, we should build a custom AWS image which includes Brickhouse ( and perhaps other tools ) from which developers can run pipelines in the cloud, and also a demo environment or for people wanting to testdrive before creating the environment.

Assume sketch_sets have been pre-generated and are sorted by minhash already.

The group_count UDF was never propagated from the initial warehouse package.

Even though other rank() UDFs exist out there, we should add ours to brickhouse.
We just need to add this, and possibly rename to rank()

Also, add check results are correct if called multiple times ...

right now we assume all keys and values are strings for the HBase UDFs.

We should be able to support binary keys and values

allow a type to be passed into distributed map, so that non-double types can be used ...

Define a Hive UDTF to do an HBase scan, so one can do straight lookups starting from a particular location..
Syntax would be something like this ..

FROM (
hbase_scan( "mytable", "mykey") ) hb
select col1, col2

right now qualifier and column family specified in the constant parameter map .

Either add sub_list to brickhouse,
or allow two index arguments to be passed in

ie array_index( ["all","good","dogs","goto","heaven"], 1,2 ) = [ "good","dogs"]

Failed to execute goal on project brickhouse: Could not resolve dependencies for project com.klout:brickhouse:jar:0.5.1: The following artifacts could not be resolved: javax.jdo:jdo2-api:jar:2.3-ec, com.clearspring.analytics:stream:jar:2.4.0-SNAPSHOT: Failure to find javax.jdo:jdo2-api:jar:2.3-ec in http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]

http://search.maven.org/#search|ga|1|g%3A%22com.clearspring.analytics%22

collect for arrays should support non primitives,
and collect for map should support non-primitives for the map value

Hive has a "builtins" mechanism where a certain jar contains references to all the Hive UDFs , so that they don't have to be specifically declared in a "setup.hql" file.

Somehow use this mechanism to register all the brickhouse UDFs as well.

Decide upon naming conventions and indentation

GitHub deprecated its Files/Downloads section, but many users of Hive on EMR don't have (or otherwise need) the JVM/maven build environment.

I modified the pom to use hadoop 1.0.3, hive 0.11.0 and hbase 0.92.0 (versions on emr), and ran mvn package which failed with the message:

brickhouse/src/main/java/brickhouse/udf/json/JsonSplitUDF.java:[58,29] cannot find symbol
symbol : method writeValueAsString(java.lang.Object)
location: class org.codehaus.jackson.map.ObjectMapper

brickhouse/src/main/java/brickhouse/udf/json/FromJsonUDF.java:[65,36] cannot find symbol
symbol : method readTree(java.lang.String)
location: class org.codehaus.jackson.map.ObjectMapper

brickhouse/src/main/java/brickhouse/hbase/CachedGetUDF.java:[75,37] cannot find symbol
symbol : method readTree(java.lang.String)
location: class org.codehaus.jackson.map.ObjectMapper

The fix was to include the missing jackson dependencies in the pom.xml:

I did verify the build succeeded using the pom file as it is on master.

A lot of the generic UDF's handle ObjectInspectors incorrectly, which can result in UDFs working find in some queries, but producing ClassCastExceptions for different queries.

Need to go through most UDFs and make sure ObjectInspectors are handled correctly

this has been resolved

cast_array currently only casts from bigints to strings .. Should specify desired type , and add type co-ersion

There is some overlap between thunder's vector ( map ) operations and brickhouse.

Either expand functionality to brickhouse's methods, ( ie. support vector x vector, dot product, vector sum, normalization, etc ... ) or import thunder's functions.

KVM sketches are fine for certain applications, but we need something a little more space efficient. We should integrate a thirdparty hyperloglog implementations ( or write our our, depending upon license, dependency, etc.)

Also, if there is some way to convert a KMV sketch to a hyperloglog, it would be useful. We could store sketch-sets on disk, to retain some sampling, and then convert to hyperloglogs to reduce sort space.

Handle GenericUDF's correctly

Implement a Hive UDF which implements a streaming top-k elements efficiently
( to avoid doing a count(*)/group by and a collect_max )

create a UDF which calculates cosine similarity, when interpreting a map of strings and doubles as a vector in multidimensional space,

Specifying template objects is too confusing to understand for most people, so allow the template type in from_json() to be specified with a type string... ie

instead of from_json(" [ 1,2,3 ] ", array( 1 ) ) ,
do this from_json(" [ 1, 2, 3 ] ", "array" )

Somehow we ended up with two md5 functions , md5, and hash_md5,

one which returns a string, and one which returns a hex string....

Need to clean this up, and perhaps include other hash types as well.

Tests in error:
testObama(brickhouse.analytics.uniques.SketchSetTest): src/test/resources/obama.txt (No such file or directory)

Am I missing anything?

~Aniket

decide if null is returned when null passed in as an argument, or empty results for map functions