org.apache.storm.blobstore

Class LocalFsBlobStore

java.lang.Object

org.apache.storm.blobstore.BlobStore

org.apache.storm.blobstore.LocalFsBlobStore

All Implemented Interfaces:

AutoCloseable, Shutdownable

public class LocalFsBlobStore
extends BlobStore

Provides a local file system backed blob store implementation for Nimbus.

For a local blob store the user and the supervisor use NimbusBlobStore Client API in order to talk to nimbus through thrift. The authentication and authorization here is based on the subject. We currently have NIMBUS_ADMINS and SUPERVISOR_ADMINS configuration. NIMBUS_ADMINS are given READ, WRITE and ADMIN access whereas the SUPERVISOR_ADMINS are given READ access in order to read and download the blobs form the nimbus.

The ACLs for the blob store are validated against whether the subject is a NIMBUS_ADMIN, SUPERVISOR_ADMIN or USER who has read, write or admin privileges in order to perform respective operations on the blob.

For local blob store 1. The USER interacts with nimbus to upload and access blobs through NimbusBlobStore Client API. 2. The USER sets the ACLs, and the blob access is validated against these ACLs. 3. The SUPERVISOR interacts with nimbus through the NimbusBlobStore Client API to download the blobs. The supervisors principal should match the set of users configured into SUPERVISOR_ADMINS. Here, the PrincipalToLocalPlugin takes care of mapping the principal to user name before the ACL validation.

Nested Class Summary

Nested classes/interfaces inherited from class org.apache.storm.blobstore.BlobStore

BlobStore.BlobStoreFileInputStream, BlobStore.BlobStoreFileOutputStream, BlobStore.KeyTranslationIterator

Field Summary

Fields

Modifier and Type	Field and Description
protected BlobStoreAclHandler	aclHandler
static org.slf4j.Logger	LOG

Fields inherited from class org.apache.storm.blobstore.BlobStore

BASE_BLOBS_DIR_NAME

Constructor Summary

Constructors

Constructor and Description
LocalFsBlobStore()

Method Summary

Modifier and Type	Method and Description
boolean	checkForBlobOrDownload(String key)
void	checkForBlobUpdate(String key)
AtomicOutputStream	createBlob(String key, SettableBlobMeta meta, Subject who) Creates the blob.
void	deleteBlob(String key, Subject who) Deletes the blob data and metadata.
void	fullCleanup(long age)
InputStreamWithMeta	getBlob(String key, Subject who) Gets the InputStream to read the blob details.
ReadableBlobMeta	getBlobMeta(String key, Subject who) Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.
int	getBlobReplication(String key, Subject who) Gets the replication factor of the blob.
Iterator<String>	listKeys() Returns an iterator with all the list of keys currently available on the blob store.
void	prepare(Map<String,Object> conf, String overrideBase, NimbusInfo nimbusInfo, ILeaderElector leaderElector) Allows us to initialize the blob store.
void	setBlobMeta(String key, SettableBlobMeta meta, Subject who) Sets the metadata with renewed acls for the blob.
void	setLeaderElector(ILeaderElector leaderElector) Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.
void	shutdown()
void	startSyncBlobs() Start the syncing blobs between the local running instance of the BlobStore and others.
AtomicOutputStream	updateBlob(String key, Subject who) Updates the blob data.
int	updateBlobReplication(String key, int replication, Subject who) Modifies the replication factor of the blob.

Methods inherited from class org.apache.storm.blobstore.BlobStore

close, createBlob, createBlob, filterAndListKeys, readBlob, readBlobTo, storedTopoIds, updateBlob, validateKey

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.slf4j.Logger LOG

aclHandler

protected BlobStoreAclHandler aclHandler

Constructor Detail

LocalFsBlobStore

public LocalFsBlobStore()

Method Detail

prepare

public void prepare(Map<String,Object> conf,
                    String overrideBase,
                    NimbusInfo nimbusInfo,
                    ILeaderElector leaderElector)

Description copied from class: BlobStore

Allows us to initialize the blob store.

Specified by:

prepare in class BlobStore

Parameters:

conf - The storm configuration

overrideBase - The directory path to store the blobs

nimbusInfo - Contains the nimbus host, port and leadership information

startSyncBlobs

public void startSyncBlobs()
                    throws KeyNotFoundException,
                           AuthorizationException

Description copied from class: BlobStore

Start the syncing blobs between the local running instance of the BlobStore and others. A no-op for the HdfsBlobStore where HDFS itself does the syncing but for the LocalFsBlobStore ZK state updates are run periodically here.

Overrides:

startSyncBlobs in class BlobStore

Throws:

KeyNotFoundException

AuthorizationException

createBlob

public AtomicOutputStream createBlob(String key,
                                     SettableBlobMeta meta,
                                     Subject who)
                              throws AuthorizationException,
                                     KeyAlreadyExistsException

Description copied from class: BlobStore

Creates the blob.

Specified by:

createBlob in class BlobStore

Parameters:

key - Key for the blob

meta - Metadata which contains the acls information

who - Is the subject creating the blob

Returns:

AtomicOutputStream returns a stream into which the data can be written

Throws:

AuthorizationException

KeyAlreadyExistsException

updateBlob

public AtomicOutputStream updateBlob(String key,
                                     Subject who)
                              throws AuthorizationException,
                                     KeyNotFoundException

Description copied from class: BlobStore

Updates the blob data.

Specified by:

updateBlob in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having the write privilege for the blob

Returns:

AtomicOutputStream returns a stream into which the data can be written

Throws:

AuthorizationException

KeyNotFoundException

getBlobMeta

public ReadableBlobMeta getBlobMeta(String key,
                                    Subject who)
                             throws AuthorizationException,
                                    KeyNotFoundException

Description copied from class: BlobStore

Gets the current version of metadata for a blob to be viewed by the user or downloaded by the supervisor.

Specified by:

getBlobMeta in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having the read privilege for the blob

Returns:

AtomicOutputStream returns a stream into which the data can be written

Throws:

AuthorizationException

KeyNotFoundException

setLeaderElector

public void setLeaderElector(ILeaderElector leaderElector)

Sets leader elector (only used by LocalFsBlobStore to help sync blobs between Nimbi.

Specified by:

setLeaderElector in class BlobStore

setBlobMeta

public void setBlobMeta(String key,
                        SettableBlobMeta meta,
                        Subject who)
                 throws AuthorizationException,
                        KeyNotFoundException

Description copied from class: BlobStore

Sets the metadata with renewed acls for the blob.

Specified by:

setBlobMeta in class BlobStore

Parameters:

key - Key for the blob

meta - Metadata which contains the updated acls information

who - Is the subject having the write privilege for the blob

Throws:

AuthorizationException

KeyNotFoundException

deleteBlob

public void deleteBlob(String key,
                       Subject who)
                throws AuthorizationException,
                       KeyNotFoundException

Description copied from class: BlobStore

Deletes the blob data and metadata.

Specified by:

deleteBlob in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having write privilege for the blob

Throws:

AuthorizationException

KeyNotFoundException

getBlob

public InputStreamWithMeta getBlob(String key,
                                   Subject who)
                            throws AuthorizationException,
                                   KeyNotFoundException

Description copied from class: BlobStore

Gets the InputStream to read the blob details.

Specified by:

getBlob in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having the read privilege for the blob

Returns:

InputStreamWithMeta has the additional file length and version information

Throws:

AuthorizationException

KeyNotFoundException

listKeys

public Iterator<String> listKeys()

Description copied from class: BlobStore

Returns an iterator with all the list of keys currently available on the blob store.

Specified by:

listKeys in class BlobStore

Returns:

Iterator<String>

shutdown

public void shutdown()

getBlobReplication

public int getBlobReplication(String key,
                              Subject who)
                       throws Exception

Description copied from class: BlobStore

Gets the replication factor of the blob.

Specified by:

getBlobReplication in class BlobStore

Parameters:

key - Key for the blob

who - Is the subject having the read privilege for the blob

Returns:

BlobReplication object containing the replication factor for the blob

Throws:

Exception

updateBlobReplication

public int updateBlobReplication(String key,
                                 int replication,
                                 Subject who)
                          throws AuthorizationException,
                                 KeyNotFoundException

Description copied from class: BlobStore

Modifies the replication factor of the blob.

Specified by:

updateBlobReplication in class BlobStore

Parameters:

key - Key for the blob

replication - The replication factor the blob has to be set

who - Is the subject having the update privilege for the blob

Returns:

BlobReplication object containing the updated replication factor for the blob

Throws:

AuthorizationException

KeyNotFoundException

checkForBlobOrDownload

public boolean checkForBlobOrDownload(String key)
                               throws KeyNotFoundException

Throws:

KeyNotFoundException

checkForBlobUpdate

public void checkForBlobUpdate(String key)

fullCleanup

public void fullCleanup(long age)
                 throws IOException

Throws:

IOException