org.apache.tika.pipes.emitter.s3

Class S3Emitter

java.lang.Object

org.apache.tika.pipes.emitter.AbstractEmitter

org.apache.tika.pipes.emitter.s3.S3Emitter

All Implemented Interfaces:

org.apache.tika.config.Initializable, org.apache.tika.pipes.emitter.Emitter, org.apache.tika.pipes.emitter.StreamEmitter

public class S3Emitter
extends org.apache.tika.pipes.emitter.AbstractEmitter
implements org.apache.tika.config.Initializable, org.apache.tika.pipes.emitter.StreamEmitter

Emits to existing s3 bucket

  <properties>
      <emitters>
          <emitter class="org.apache.tika.pipes.emitter.s3.S3Emitter>
              <params>
                  <!-- required -->
                  <param name="name" type="string">s3e</param>
                  <!-- required -->
                  <param name="region" type="string">us-east-1</param>
                  <!-- required -->
                  <param name="credentialsProvider"
                       type="string">(profile|instance)</param>
                  <!-- required if credentialsProvider=profile-->
                  <param name="profile" type="string">my-profile</param>
                  <!-- required -->
                  <param name="bucket" type="string">my-bucket</param>
                  <!-- optional; prefix to add to the path before emitting;
                       default is no prefix -->
                  <param name="prefix" type="string">my-prefix</param>
                  <!-- optional; default is 'json' this will be added to the SOURCE_PATH
                                    if no emitter key is specified. Do not add a "."
                                     before the extension -->
                  <param name="fileExtension" type="string">json</param>
                  <!-- optional; default is 'true'-- whether to copy the
                     json to a local file before putting to s3 -->
                  <param name="spoolToTemp" type="bool">true</param>
              </params>
          </emitter>
      </emitters>
  </properties>

Constructor Summary

Constructors

Constructor and Description
S3Emitter()

Method Summary

Modifier and Type	Method and Description
void	checkInitialization(org.apache.tika.config.InitializableProblemHandler problemHandler)
void	emit(String path, InputStream is, org.apache.tika.metadata.Metadata userMetadata)
void	emit(String emitKey, List<org.apache.tika.metadata.Metadata> metadataList) Requires the src-bucket/path/to/my/file.txt in the TikaCoreProperties.SOURCE_PATH.
void	initialize(Map<String,org.apache.tika.config.Param> params) This initializes the s3 client.
void	setAccessKey(String accessKey)
void	setBucket(String bucket)
void	setCredentialsProvider(String credentialsProvider)
void	setEndpointConfigurationService(String endpointConfigurationService)
void	setFileExtension(String fileExtension) If you want to customize the output file's file extension.
void	setMaxConnections(int maxConnections) maximum number of http connections allowed.
void	setPathStyleAccessEnabled(boolean pathStyleAccessEnabled)
void	setPrefix(String prefix)
void	setProfile(String profile)
void	setRegion(String region)
void	setSecretKey(String secretKey)
void	setSpoolToTemp(boolean spoolToTemp) Whether or not to spool the metadatalist to a tmp file before putting object.

Methods inherited from class org.apache.tika.pipes.emitter.AbstractEmitter

emit, getName, setName

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.tika.pipes.emitter.Emitter

emit, getName

Constructor Detail

S3Emitter

public S3Emitter()

Method Detail

emit

public void emit(String emitKey,
                 List<org.apache.tika.metadata.Metadata> metadataList)
          throws IOException,
                 org.apache.tika.pipes.emitter.TikaEmitterException

Requires the src-bucket/path/to/my/file.txt in the TikaCoreProperties.SOURCE_PATH.

Specified by:

emit in interface org.apache.tika.pipes.emitter.Emitter

Parameters:

metadataList -

Throws:

IOException

org.apache.tika.exception.TikaException

org.apache.tika.pipes.emitter.TikaEmitterException

emit

public void emit(String path,
                 InputStream is,
                 org.apache.tika.metadata.Metadata userMetadata)
          throws IOException,
                 org.apache.tika.pipes.emitter.TikaEmitterException

Specified by:

emit in interface org.apache.tika.pipes.emitter.StreamEmitter

Parameters:

path - -- object path, not including the bucket

is - inputStream to copy

userMetadata - this will be written to the s3 ObjectMetadata's userMetadata

Throws:

org.apache.tika.pipes.emitter.TikaEmitterException - or IOexception if there is a Runtime s3 client exception

IOException

setSpoolToTemp

@Field
public void setSpoolToTemp(boolean spoolToTemp)

Whether or not to spool the metadatalist to a tmp file before putting object. Default: true. If this is set to false, this emitter writes the json object to memory and then puts that into s3.

Parameters:

spoolToTemp -

setRegion

@Field
public void setRegion(String region)

setProfile

@Field
public void setProfile(String profile)

setBucket

@Field
public void setBucket(String bucket)

setPrefix

@Field
public void setPrefix(String prefix)

setCredentialsProvider

@Field
public void setCredentialsProvider(String credentialsProvider)

setFileExtension

@Field
public void setFileExtension(String fileExtension)

If you want to customize the output file's file extension. Do not include the "."

Parameters:

fileExtension -

setAccessKey

@Field
public void setAccessKey(String accessKey)

setSecretKey

@Field
public void setSecretKey(String secretKey)

setMaxConnections

@Field
public void setMaxConnections(int maxConnections)

maximum number of http connections allowed. This should be greater than or equal to the number of threads emitting to S3.

Parameters:

maxConnections -

setEndpointConfigurationService

@Field
public void setEndpointConfigurationService(String endpointConfigurationService)

initialize

public void initialize(Map<String,org.apache.tika.config.Param> params)
                throws org.apache.tika.exception.TikaConfigException

This initializes the s3 client. Note, we wrap S3's RuntimeExceptions, e.g. AmazonClientException in a TikaConfigException.

Specified by:

initialize in interface org.apache.tika.config.Initializable

Parameters:

params - params to use for initialization

Throws:

org.apache.tika.exception.TikaConfigException

checkInitialization

public void checkInitialization(org.apache.tika.config.InitializableProblemHandler problemHandler)
                         throws org.apache.tika.exception.TikaConfigException

Specified by:

checkInitialization in interface org.apache.tika.config.Initializable

Throws:

org.apache.tika.exception.TikaConfigException

setPathStyleAccessEnabled

@Field
public void setPathStyleAccessEnabled(boolean pathStyleAccessEnabled)