Code Monkey home page Code Monkey logo

pig's People

Contributors

aneeshs avatar aniket486 avatar anirudhtodi avatar ashutoshc avatar breed avatar dvryaboy avatar gdfm avatar gkesavan avatar jcoveney avatar julienledem avatar lalit1303 avatar mateiz avatar praveenr019 avatar r0hini avatar rangadi avatar sigmoidanalytics avatar zjffdu avatar

Stargazers

 avatar

Watchers

 avatar  avatar  avatar

Forkers

mk6-dev

pig's Issues

SkewJoin is not working in localMachine Spork

2014-06-12 00:33:24,608 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: SKEWED_JOIN
2014-06-12 00:33:24,609 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, DuplicateForEachColumnRewrite, GroupByConstParallelSetter, ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter], RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2014-06-12 00:33:24,616 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - !!!!!!!!!! Launching Spark (woot) !!!!!!!!!!!!
2014-06-12 00:33:24,625 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2014-06-12 00:33:24,625 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2014-06-12 00:33:24,633 [main] INFO org.apache.pig.impl.builtin.PartitionSkewedKeys - input file: hdfs://localhost:54310/tmp/temp-1456742965/tmp-1456742965
2014-06-12 00:33:24,636 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POLoad (Name: A: Load(hdfs://localhost:54310/input/data1:PigStorage(',')) - scope-78 Operator Key: scope-78)
2014-06-12 00:33:24,650 [main] INFO org.apache.spark.storage.MemoryStore - ensureFreeSpace(63445) called with curMem=253780, maxMem=559310438
2014-06-12 00:33:24,650 [main] INFO org.apache.spark.storage.MemoryStore - Block broadcast_4 stored as values to memory (estimated size 62.0 KB, free 533.1 MB)
2014-06-12 00:33:24,652 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POForEach (Name: A: New For Each(false,false,false)[bag] - scope-85 Operator Key: scope-85)
2014-06-12 00:33:24,653 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POLoad (Name: B: Load(hdfs://localhost:54310/input/data2:PigStorage(',')) - scope-86 Operator Key: scope-86)
2014-06-12 00:33:24,660 [main] INFO org.apache.spark.storage.MemoryStore - ensureFreeSpace(63445) called with curMem=317225, maxMem=559310438
2014-06-12 00:33:24,661 [main] INFO org.apache.spark.storage.MemoryStore - Block broadcast_5 stored as values to memory (estimated size 62.0 KB, free 533.0 MB)
2014-06-12 00:33:24,662 [main] INFO org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher - Converting operator POForEach (Name: B: New For Each(false,false)[bag] - scope-91 Operator Key: scope-91)
2014-06-12 00:33:24,663 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2043: Unexpected error during execution.
Details at logfile: /home/maheshk/Desktop/spork/spork/pig_1402512909094.log

Complete Log: https://gist.github.com/b895c51dbddfc19e0ab4

hbase schema and hbasestorage not working

L1 = load 'hbase://sampletable' using org.apache.pig.backend.hadoop.hbase.HBaseStorage('Coun1:C11,Coun1:C12,Coun1:C13', '-loadKey true')
as (C11:chararray, C12:chararray, C13:chararray);
B1 = FOREACH L1 GENERATE C11 AS Coun11;
DUMP B1;

Pig Stack Trace

ERROR 2043: Unexpected error during execution.

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B1
at org.apache.pig.PigServer.openIterator(PigServer.java:895)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:729)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:370)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:543)
at org.apache.pig.Main.main(Main.java:158)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias B1
at org.apache.pig.PigServer.storeEx(PigServer.java:997)
at org.apache.pig.PigServer.store(PigServer.java:957)
at org.apache.pig.PigServer.openIterator(PigServer.java:870)
... 7 more
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 2043: Unexpected error during execution.
at org.apache.pig.PigServer.launchPlan(PigServer.java:1339)
at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1310)
at org.apache.pig.PigServer.storeEx(PigServer.java:993)
... 9 more
Caused by: java.io.IOException: No FileSystem for scheme: hbase
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1383)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:372)
at org.apache.spark.SparkContext.newAPIHadoopFile(SparkContext.scala:469)
at org.apache.pig.backend.hadoop.executionengine.spark.converter.LoadConverter.convert(LoadConverter.java:63)
at org.apache.pig.backend.hadoop.executionengine.spark.converter.LoadConverter.convert(LoadConverter.java:1)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:286)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:272)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.physicalToRDD(SparkLauncher.java:272)
at org.apache.pig.backend.hadoop.executionengine.spark.SparkLauncher.launchPig(SparkLauncher.java:178)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1328)
... 11 more

UDF Fails to operate on Different Columns

UDF output is not proper, it is operating on the first column even if you specify some other column

Eg:
REGISTER /home/akhld/mobi/temp/pig/demo/tutorial.jar;
A = load 'hdfs://localhost:9000/input/' USING PigStorage('\t') AS (user:chararray, time:int, query:chararray);
B = FOREACH A GENERATE org.apache.pig.tutorial.ToLower(query) as query;
STORE B INTO '/tmp/query';

Fails to Operate on "query" column, but the processing happens on "user" column.

GroupBy not working in Cluster mode

When you try to execute the following crashes with not-serializable Exception.

A = load 'hdfs://localhost:9000/input/' USING PigStorage(',') AS (p, q, r);
B = GROUP A BY p;
STORE B INTO '/tmp/groupby';

Its happening in Spork streaming.

UDF Output Not able to STORE in SparkStreaming mode

udf

Crashes with java.lang.RuntimeException: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POUserFunc (Name: POUserFunc(org.apache.pig.tutorial.ToLower)[chararray] - scope-3 Operator Key: scope-3) children: null at []]: java.lang.NullPointerException

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.