Pig – Functions

Friends , Functions in Pig come in four types
1. Eval function
– A function that takes one or more expressions and returns another expression.
– Some function are aggregate function like MAX
– Some functions are algebraic, which means that the result of the function may be calculated incrementally.
– In MapReduce term algebric functions make use of combiner and are much more efficient to calculate .
– Supports UDF by importing org.apache.pig.EvalFunc , extend EvalFunc & overriding exec method

2. Filter function :
– It returns logical boolean results
– FILTER removes unwanted rows
– EX: IsEmpty
– Supports UDF by importing org.apache.pig.FilterFunc , extend FilterFunc & overriding exec method

3. Load function
– Loads the data into a relation from external storage
– Supports UDF by importing org.apache.pig.LoadFunc , extend LoadFunc but override different other function like setLocation , getInputFormat , prepareTORead , getNext methods.

4. Store function
– Specifies how to save the contents of a relation to external storage
– Ex: PigStorage which loads data from delimited text files , can store data in the same format.

Detailed list is given below:

Pig Built-in Function
Eval AVG Calculate Avg(Mean) value of entries in a bag
CONCAT Concatenates byte arrays or chareacter array together
COUNT Calculate number of non-null entries in a bag
COUNT_STAR Calculate all entries including nulls
DIFF Calculates the set difference of two bags. If the two arguments are not bags
, returns a bag containing both if they are equal;otherwise,returns a nempty bag
MAX Calculate max
MIN Calculate Min
SIZE for character arrays, it is the num of char. For byte arrays the number of bytes
for containers(tuple , bag,map) it is number of entries
SUM Calculate summation of the values of entries in a bag
TOBAG Convert one or emore expresssions to individual tuple which are then put in a bag
TOKANIZE Tokenizes a character array into a bag of it’s constituent words
TOMAP Converts an even number of expressions to a map of key-value pairs
TOP Calculate top n tuples in a bag
TOTUPLE Convert one or more expresssions to a tuple
Filter IsEmpty Test weather bag or map is empty
Load/Sttore PigStorage Loads or stores relations using a field-delimited text format defaults to a tab character
BinStorage Loads or store relations from or to binary files in a pig specific format that uses HadoopWritable Object
TextLoader Loads relations from a plain-text format.
JsonLoader,JsonStorage Loads or store s relations from or to a JSON format.
HBaseStorage Loads or stores relation from or to Hbase
Advertisements

Pig Latin Relational Operator

   Pig Latin Relational Operator
Category Operator Description
Loading & Storing LOAD Loads data from the filesystem or other storage into a relation
STORE Saves a relation to the filesystem or other storage
DUMP Prints a relation to the console
Filtering FILTER Removes unwanted rows from relation
DISTINCT Removes duplicate rows from a relation
FOREACH..GENERATE Adds or removes fields from relation
MAPREDUCE Runs a mapreduce job using a relations as input
STREAM Transforms a relation using an external program
SAMPLE selects ar andom sample of a relation
Grouping & Joining JOIN Joins two or more relations
COGROUP Groups the data in two or more relations
GROUP Groups the data in a single relation
CROSS creates the croos-product of two or more relations
Sorting ORDER Sorts a relation by one or more fields
LIMIT Limits the size of a relation to a maximum number of tuples
Combining & Splitting UNION Combines two or more relations into one
SPLIT Split a relation into two or more relations

Pig – A programmer friendly MapReduce tool

Pig raises the level of abstraction for processing large datasets. MapReduce allow you , to specify map function followed by reduce function , but working out how to fit your data processing into this pattern , which often require multiple MapReduce stages, can be a challange.
Pig supports richer data structure, typically being multivalued & nested, and the set of transformations you can apply to the data are much more powerful.
One of the powerful feature of PIG is join, which are not for the faint of heart in MapReduce.
Pig is made up of 2 pieces:
1. The language used to express data flows, called Pig Latin
2. The Execution enviornment to run Pig Latin Program.
For MultiQuery execution it is always recommended to use STORE instead of DUMP as DUMP is a diagnostic tool, it will always trigger execution even in batch mode which STORE command doesn’t.
Consider the following example:
A = LOAD ‘input/pig/multiquery/A’
B = FILTER A by $1 == ‘banana’;
C = FILTER A BY $1 != ‘banana’;
STORE B INTO ‘output/b’;
STORE C INTO ‘output/c’;
Pig can run this script as a single MapReduce job by reading A once and writing two output files from the job, one for each of B & C. This features is called multiquery execution.