edwardcapriolo / hive_test Goto Github PK

Hey Edward,

I can't find the maven repo where hive_test is located.

Any help is much appreciated.

Best,
Ariel

I would like to use hive UDF to find geo information (city, zip etc) from ip address.

I was routed to this UDF by Edward (edwardcapriolo/hive-geoip#1)

Could someone help me since I am stuck in creating Hive geo udf for more than a week.

If this is not the right right udf, please guide me some guidance.

I am getting error message when I try to create temporary function.

Below are the steps:

hive> add jar hive-exec-0.11.0.jar;
Added hive-exec-0.11.0.jar to class path
Added resource: hive-exec-0.11.0.jar
hive> add jar geo-ip-java.jar;
Added geo-ip-java.jar to class path
Added resource: geo-ip-java.jar
hive> add jar hiveunit-0.0.4-SNAPSHOT.jar;
Added [hiveunit-0.0.4-SNAPSHOT.jar] to class path
Added resources: [hiveunit-0.0.4-SNAPSHOT.jar]
hive> add file GeoIP.dat;
Added resource: GeoIP.dat
hive> create temporary function geoip as 'com.jointhegrid.hive.udf.GenericUDFGeoIP';
FAILED: Class com.jointhegrid.hive.udf.GenericUDFGeoIP not found
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask

I am following up from Git:
https://github.com/edwardcapriolo/hive-geoip

Thank you!

I'm sorry if I shouldn't leave a question here, but I really cannot figure this out. I'm trying to test some very simple hive script, but this error kept coming up:

Exception in thread "main" org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 15
Serialization trace:
mapWork (org.apache.hadoop.hive.ql.plan.MapredWork)
at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:656)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:99)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:507)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:672)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializeObjectByKryo(Utilities.java:1080)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:971)
at org.apache.hadoop.hive.ql.exec.Utilities.deserializePlan(Utilities.java:985)
at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.main(ExecDriver.java:748)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Execution failed with exit status: 1
15/06/11 16:52:55 ERROR exec.Task: Execution failed with exit status: 1
Obtaining error information
15/06/11 16:52:55 ERROR exec.Task: Obtaining error information

This only happens when the hive script involves 'load' operation. So it is good to create table, but when comes to load data this just pop up. Here is the log:

2015-06-11 16:06:53,992 WARN [main]: common.LogUtils (LogUtils.java:logConfigLocation(145)) - hive-site.xml not found on CLASSPATH
2015-06-11 16:06:53,997 INFO [main]: mr.ExecDriver (SessionState.java:printInfo(948)) - Execution log at: /tmp/root/root_20150611160606_dc94272f-ef0c-4f16-a381-86d215e15b0f.log
2015-06-11 16:06:54,111 WARN [main]: conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/root/hive_2015-06-11_16-06-45_635_8515305690721422166-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-06-11 16:06:54,114 WARN [main]: conf.Configuration (Configuration.java:loadProperty(2172)) - file:/tmp/root/hive_2015-06-11_16-06-45_635_8515305690721422166-1/-local-10003/jobconf.xml:an attempt to override final parameter: mapreduce.job.end-notification.max.attempts; Ignoring.
2015-06-11 16:06:54,173 INFO [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(121)) -
2015-06-11 16:06:54,173 INFO [main]: exec.Utilities (Utilities.java:deserializePlan(964)) - Deserializing MapredWork via kryo

Which I don't see much help either. I'm sorry again if this is not the right place to ask help, but any idea of solving this is so welcomed. If you need any info about my setting or script please just let me know. Thanks in advance!

I am trying to use your maven dependancies for testing however seems like it's not working.

if i use dependancy in a way you have mentioned in your readme file using classifier then it gives me the following error.

  <dependency>
        <groupId>io.teknek</groupId>
        <artifactId>hiveunit</artifactId>
        <version>0.0.3</version>
        <classifier>jar-with-dependencies</classifier>
    </dependency>

Stacktrace:

[ERROR] Failed to execute goal on project HiveUnitTest: Could not resolve dependencies for project com.gaurang:HiveUnitTest:jar:1.0-SNAPSHOT: Failure to find io.teknek:hiveunit:jar:jar-with-dependencies:0.0.3 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.

if I don't use classifier then maven is able to fine the jar file however, it doesn't contain all the dependancies.

Edward

I have download hive_test and have got it running fine. How do I integrate hive_test with my own application that has its own pom and hive code. Do I just define a dependency to the hive_test artifact and my project would inherit the profiles and have access to HiveTestService?

Deenar