org.apache.datasketches.pig.theta

Class Union

java.lang.Object

org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>

org.apache.datasketches.pig.theta.Union

All Implemented Interfaces:

org.apache.pig.Accumulator<org.apache.pig.data.Tuple>, org.apache.pig.Algebraic

public class Union
extends org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>
implements org.apache.pig.Accumulator<org.apache.pig.data.Tuple>, org.apache.pig.Algebraic

This is a Pig UDF that performs the Union Set Operation on Sketches. To assist Pig, this class implements both the Accumulator and Algebraic interfaces.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	Union.Initial Class used to calculate the initial pass of an Algebraic sketch operation.
static class	Union.IntermediateFinal Class used to calculate the intermediate or final combiner pass of an Algebraic union operation.

Nested classes/interfaces inherited from class org.apache.pig.EvalFunc

org.apache.pig.EvalFunc.SchemaType

Field Summary

Fields inherited from class org.apache.pig.EvalFunc

log, pigLogger, reporter, returnType

Constructor Summary

Constructors

Constructor and Description
Union() Default constructor to make pig validation happy.
Union(int nomEntries, float p, long seed) Base constructor.
Union(String nomEntriesStr) String constructor.
Union(String nomEntriesStr, String pStr) String constructor.
Union(String nomEntriesStr, String pStr, String seedStr) Full string constructor.

Method Summary

Modifier and Type	Method and Description
void	accumulate(org.apache.pig.data.Tuple inputTuple) An Accumulator version of the standard exec() method.
void	cleanup() Cleans up the UDF state after being called using the Accumulator interface.
org.apache.pig.data.Tuple	exec(org.apache.pig.data.Tuple inputTuple) Top-level exec function.
String	getFinal()
String	getInitial()
String	getIntermed()
org.apache.pig.data.Tuple	getValue() Returns the sketch that has been built up by multiple calls to accumulate(org.apache.pig.data.Tuple).
org.apache.pig.impl.logicalLayer.schema.Schema	outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Methods inherited from class org.apache.pig.EvalFunc

allowCompileTimeCalculation, finish, getArgToFuncMapping, getCacheFiles, getInputSchema, getLoadCaster, getLogger, getPigLogger, getReporter, getReturnType, getSchemaName, getSchemaType, getShipFiles, isAsynchronous, needEndOfAllInputProcessing, progress, setEndOfAllInput, setInputSchema, setPigLogger, setReporter, setUDFContextSignature, warn

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Union

public Union()

Default constructor to make pig validation happy. Assumes:

• See Default Nominal Entries
• p = 1.0. See Sampling Probability, p.
• See Default Update Seed

Union

public Union(String nomEntriesStr)

String constructor. Assumes:

• p = 1.0. See Sampling Probability, p
• See Default Update Seed

Parameters:

nomEntriesStr - See Nominal Entries

Union

public Union(String nomEntriesStr,
             String pStr)

String constructor. Assumes:

• See Default Update Seed

Parameters:

nomEntriesStr - See Nominal Entries

pStr - See Sampling Probability, p. Although this functionality is implemented for SketchUnions, it rarely makes sense to use it here. The proper use of upfront sampling is when building the sketches.

Union

public Union(String nomEntriesStr,
             String pStr,
             String seedStr)

Full string constructor.

Parameters:

nomEntriesStr - See Nominal Entries.

pStr - See Sampling Probability, p. Although this functionality is implemented for SketchUnions, it rarely makes sense to use it here. The proper use of upfront sampling is when building the sketches.

seedStr - See Update Hash Seed.

Union

public Union(int nomEntries,
             float p,
             long seed)

Base constructor.

Parameters:

nomEntries - See Nominal Entries.

p - See Sampling Probability, p. Although this functionality is implemented for SketchUnions, it rarely makes sense to use it here. The proper use of upfront sampling is when building the sketches.

seed - See Update Hash Seed.

Method Detail

exec

public org.apache.pig.data.Tuple exec(org.apache.pig.data.Tuple inputTuple)
                               throws IOException

Top-level exec function. This method accepts an input Tuple containing a Bag of one or more inner Sketch Tuples and returns a single updated Sketch as a Sketch Tuple.

If a large number of calls are anticipated, leveraging either the Algebraic or Accumulator interfaces is recommended. Pig normally handles this automatically.

Internally, this method presents the inner Sketch Tuples to a new Union. The result is returned as a Sketch Tuple

Input Tuple

• Tuple: TUPLE (Must contain only one field)
  • index 0: DataBag: BAG (May contain 0 or more Inner Tuples)
    • index 0: Tuple: TUPLE Sketch Tuple
    • ...
    • index n-1: Tuple: TUPLE Sketch Tuple

Sketch Tuple

• Tuple: TUPLE (Contains exactly 1 field)
  • index 0: DataByteArray: BYTEARRAY = The serialization of a Sketch object.

Specified by:

exec in class org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>

Parameters:

inputTuple - A tuple containing a single bag, containing Sketch Tuples.

Returns:

Sketch Tuple. If inputTuple is null or empty, returns empty sketch (8 bytes).

Throws:

IOException

See Also:

"org.apache.pig.EvalFunc.exec(org.apache.pig.data.Tuple)"

outputSchema

public org.apache.pig.impl.logicalLayer.schema.Schema outputSchema(org.apache.pig.impl.logicalLayer.schema.Schema input)

Overrides:

outputSchema in class org.apache.pig.EvalFunc<org.apache.pig.data.Tuple>

accumulate

public void accumulate(org.apache.pig.data.Tuple inputTuple)
                throws IOException

An Accumulator version of the standard exec() method. Like exec(), accumulator is called with a bag of Sketch Tuples. Unlike exec(), it doesn't serialize the sketch at the end. Instead, it can be called multiple times, each time with another bag of Sketch Tuples to be input to the Union.

Specified by:

accumulate in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

Parameters:

inputTuple - A tuple containing a single bag, containing Sketch Tuples.

Throws:

IOException - by Pig

See Also:

exec(org.apache.pig.data.Tuple), "org.apache.pig.Accumulator.accumulate(org.apache.pig.data.Tuple)"

getValue

public org.apache.pig.data.Tuple getValue()

Returns the sketch that has been built up by multiple calls to accumulate(org.apache.pig.data.Tuple).

Specified by:

getValue in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

Returns:

Sketch Tuple. (see exec(org.apache.pig.data.Tuple) for return tuple format)

See Also:

"org.apache.pig.Accumulator.getValue()"

cleanup

public void cleanup()

Cleans up the UDF state after being called using the Accumulator interface.

Specified by:

cleanup in interface org.apache.pig.Accumulator<org.apache.pig.data.Tuple>

See Also:

"org.apache.pig.Accumulator.cleanup()"

getInitial

public String getInitial()

Specified by:

getInitial in interface org.apache.pig.Algebraic

getIntermed

public String getIntermed()

Specified by:

getIntermed in interface org.apache.pig.Algebraic

getFinal

public String getFinal()

Specified by:

getFinal in interface org.apache.pig.Algebraic