Package org.apache.storm.hdfs.bolt

Class AbstractHdfsBolt

java.lang.Object
org.apache.storm.topology.base.BaseComponent
org.apache.storm.topology.base.BaseRichBolt
org.apache.storm.hdfs.bolt.AbstractHdfsBolt
All Implemented Interfaces:
Serializable, IBolt, IComponent, IRichBolt
Direct Known Subclasses:
AvroGenericRecordBolt, HdfsBolt, SequenceFileBolt
public abstract class AbstractHdfsBolt extends BaseRichBolt
See Also:

  • Field Details

    • writers

      protected Map<String,Writer> writers

    • rotationCounterMap

      protected Map<String,Integer> rotationCounterMap

    • rotationActions

      protected List<RotationAction> rotationActions

    • collector

      protected OutputCollector collector

    • fs

      protected transient org.apache.hadoop.fs.FileSystem fs

    • syncPolicy

      protected SyncPolicy syncPolicy

    • rotationPolicy

      protected FileRotationPolicy rotationPolicy

    • fileNameFormat

      protected FileNameFormat fileNameFormat

    • fsUrl

      protected String fsUrl

    • configKey

      protected String configKey

    • writeLock

      protected transient Object writeLock

    • rotationTimer

      protected transient Timer rotationTimer

    • offset

      protected long offset

    • fileRetryCount

      protected Integer fileRetryCount

    • tickTupleInterval

      protected Integer tickTupleInterval

    • maxOpenFiles

      protected Integer maxOpenFiles

    • partitioner

      protected Partitioner partitioner

    • hdfsConfig

      protected transient org.apache.hadoop.conf.Configuration hdfsConfig

  • Constructor Details

    • AbstractHdfsBolt

      public AbstractHdfsBolt()

  • Method Details

    • rotateOutputFile

      protected void rotateOutputFile(Writer writer) throws IOException
      Throws:
      IOException

    • prepare

      public final void prepare(Map<String,Object> conf, TopologyContext topologyContext, OutputCollector collector)
      Marked as final to prevent override. Subclasses should implement the doPrepare() method.
      Parameters:
      conf - The Storm configuration for this bolt. This is the configuration provided to the topology merged in with cluster configuration on this machine.
      topologyContext - This object can be used to get information about this task's place within the topology, including the task id and component id of this task, input and output information, etc.
      collector - The collector is used to emit tuples from this bolt. Tuples can be emitted at any time, including the prepare and cleanup methods. The collector is thread-safe and should be saved as an instance variable of this bolt object.

    • execute

      public final void execute(Tuple tuple)
      Description copied from interface: IBolt
      Process a single tuple of input. The Tuple object contains metadata on it about which component/stream/task it came from. The values of the Tuple can be accessed using Tuple#getValue. The IBolt does not have to process the Tuple immediately. It is perfectly fine to hang onto a tuple and process it later (for instance, to do an aggregation or join).

      Tuples should be emitted using the OutputCollector provided through the prepare method. It is required that all input tuples are acked or failed at some point using the OutputCollector. Otherwise, Storm will be unable to determine when tuples coming off the spouts have been completed.

      For the common case of acking an input tuple at the end of the execute method, see IBasicBolt which automates this.

      Parameters:
      tuple - The input tuple to be processed.

    • getComponentConfiguration

      public Map<String,Object> getComponentConfiguration()
      Description copied from interface: IComponent
      Declare configuration specific to this component. Only a subset of the "topology.*" configs can be overridden. The component configuration can be further overridden when constructing the topology using TopologyBuilder
      Specified by:
      getComponentConfiguration in interface IComponent
      Overrides:
      getComponentConfiguration in class BaseComponent

    • declareOutputFields

      public void declareOutputFields(OutputFieldsDeclarer outputFieldsDeclarer)
      Description copied from interface: IComponent
      Declare the output schema for all the streams of this topology.
      Parameters:
      outputFieldsDeclarer - this is used to declare output stream ids, output fields, and whether or not each output stream is a direct stream

    • cleanup

      public void cleanup()
      Description copied from interface: IBolt
      Called when an IBolt is going to be shutdown. Storm will make a best-effort attempt to call this if the worker shutdown is orderly. The Config.SUPERVISOR_WORKER_SHUTDOWN_SLEEP_SECS setting controls how long orderly shutdown is allowed to take. There is no guarantee that cleanup will be called if shutdown is not orderly, or if the shutdown exceeds the time limit.

      The one context where cleanup is guaranteed to be called is when a topology is killed when running Storm in local mode.

      Specified by:
      cleanup in interface IBolt
      Overrides:
      cleanup in class BaseRichBolt

    • getBasePathForNextFile

      protected org.apache.hadoop.fs.Path getBasePathForNextFile(Tuple tuple)

    • doPrepare

      protected abstract void doPrepare(Map<String,Object> conf, TopologyContext topologyContext, OutputCollector collector) throws IOException
      Throws:
      IOException

    • getWriterKey

      protected abstract String getWriterKey(Tuple tuple)

    • makeNewWriter

      protected abstract Writer makeNewWriter(org.apache.hadoop.fs.Path path, Tuple tuple) throws IOException
      Throws:
      IOException