Package org.apache.storm.metricstore.rocksdb

Class RocksDbMetricsWriter

  • All Implemented Interfaces:
    AutoCloseable, Runnable
    public class RocksDbMetricsWriter
extends Object
implements Runnable, AutoCloseable
    Class designed to perform all metrics inserts into RocksDB. Metrics are processed from a blocking queue. Inserts to RocksDB are done using a single thread to simplify design (such as looking up existing metric data for aggregation, and fetching/evicting metadata from the cache). This class is not thread safe.

    A writable LRU StringMetadataCache is used to minimize looking up metadata string Ids. As entries are added to the full cache, older entries are evicted from the cache and need to be written to the database. This happens as the handleEvictedMetadata() method callback.

    The following issues would need to be addressed to implement a multithreaded metrics writer:
    • Generation of unique unused IDs for new metadata strings needs to be thread safe.
    • Ensuring newly created metadata strings are seen by all threads.
    • Maintaining a properly cached state of metadata for multiple writers. The current LRU cache evicts data as new metadata is added.
    • Processing the aggregation of a metric requires fetching and updating previous aggregates. A multithreaded design would need to ensure two metrics were not updating an aggregated metric at the same time.
    • Investigate performance of multiple threads inserting into RocksDB versus a single ordered insert.

    • Method Detail

      • run

        public void run()
        Run routine to wait for metrics on a queue and insert into RocksDB.
        Specified by:
        run in interface Runnable