org.apache.storm.localizer

Class AsyncLocalizer

java.lang.Object

org.apache.storm.localizer.AsyncLocalizer

All Implemented Interfaces:

AutoCloseable

public class AsyncLocalizer
extends Object
implements AutoCloseable

Downloads and caches blobs locally.

Field Summary

Fields

Modifier and Type	Field and Description
protected long	cacheTargetSize
protected ConcurrentHashMap<String,ConcurrentHashMap<String,LocalizedResource>>	userArchives
protected ConcurrentHashMap<String,ConcurrentHashMap<String,LocalizedResource>>	userFiles

Constructor Summary

Constructors

Constructor and Description
AsyncLocalizer(Map<String,Object> conf, StormMetricsRegistry metricsRegistry)

Method Summary

Modifier and Type	Method and Description
void	close()
protected ClientBlobStore	getClientBlobStore()
void	recoverRunningTopology(LocalAssignment currentAssignment, int port, BlobChangingCallback cb) Do everything needed to recover the state in the AsyncLocalizer for a running topology.
void	releaseSlotFor(LocalAssignment assignment, int port) Remove this assignment/port as blocking resources from being cleaned up.
CompletableFuture<Void>	requestDownloadTopologyBlobs(LocalAssignment assignment, int port, BlobChangingCallback cb) Request that all of the blobs necessary for this topology be downloaded.
void	start() Start any background threads needed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

userFiles

protected final ConcurrentHashMap<String,ConcurrentHashMap<String,LocalizedResource>> userFiles

userArchives

protected final ConcurrentHashMap<String,ConcurrentHashMap<String,LocalizedResource>> userArchives

cacheTargetSize

protected long cacheTargetSize

Constructor Detail

AsyncLocalizer

public AsyncLocalizer(Map<String,Object> conf,
                      StormMetricsRegistry metricsRegistry)
               throws IOException

Throws:

IOException

Method Detail

requestDownloadTopologyBlobs

public CompletableFuture<Void> requestDownloadTopologyBlobs(LocalAssignment assignment,
                                                            int port,
                                                            BlobChangingCallback cb)
                                                     throws IOException

Request that all of the blobs necessary for this topology be downloaded. Note that this adds references to blobs asynchronously in background threads.

Parameters:

assignment - the assignment that needs the blobs

port - the port the assignment is a part of

cb - a callback for when the blobs change. This is only for blobs that are tied to the lifetime of the worker.

Returns:

a Future that indicates when they are all downloaded.

Throws:

IOException - if there was an error while trying doing it.

start

public void start()

Start any background threads needed. This includes updating blobs and cleaning up unused blobs over the configured size limit.

close

public void close()
           throws InterruptedException

Specified by:

close in interface AutoCloseable

Throws:

InterruptedException

recoverRunningTopology

public void recoverRunningTopology(LocalAssignment currentAssignment,
                                   int port,
                                   BlobChangingCallback cb)
                            throws IOException

Do everything needed to recover the state in the AsyncLocalizer for a running topology.

Parameters:

currentAssignment - the assignment for the topology.

port - the port the assignment is on.

cb - a callback for when the blobs are updated. This will only be for blobs that indicate that if they change the worker should be restarted.

Throws:

IOException - on any error trying to recover the state.

releaseSlotFor

public void releaseSlotFor(LocalAssignment assignment,
                           int port)
                    throws IOException

Remove this assignment/port as blocking resources from being cleaned up.

Parameters:

assignment - the assignment the resources are for

port - the port the topology is running on

Throws:

IOException - on any error

getClientBlobStore

protected ClientBlobStore getClientBlobStore()