External Storage - Java SDK
When your Workflows or Activities handle data larger than the Temporal Service payload limit, offload the payloads to an external store such as Amazon S3. Temporal stores a small reference in Event History, and the Java SDK retrieves the payload before your Workflow or Activity receives it.
This page shows how to configure the Java SDK with Amazon S3. For the claim check pattern, retention requirements, and storage design guidance, see External Storage.
Store and retrieve large payloads with Amazon S3
The Java SDK includes an experimental S3 storage driver. It requires Java SDK v1.39.0 or later.
Prerequisites
-
An S3 bucket that your Temporal Client and Workers can reach. Configure lifecycle management so objects remain available for the Workflow lifetime and Namespace retention period.
-
AWS credentials that can read and write S3 objects. The AWS SDK for Java reads its standard credential provider chain, including environment variables, IAM roles, and AWS configuration files.
-
The AWS SDK v2 S3 driver module. Use the same version as your Temporal Java SDK dependency. The module includes the generic S3 driver and AWS S3 client as transitive dependencies:
implementation "io.temporal:temporal-payload-storage-s3driver-awssdkv2:1.39.0"
Procedure
-
Create an asynchronous S3 client, wrap it in an
S3AsyncClientAdapter, and create anS3StorageDriver:S3AsyncClient s3Client = S3AsyncClient.builder().region(Region.US_EAST_2).build();S3StorageDriver driver =S3StorageDriver.newBuilder().setClient(new S3AsyncClientAdapter(s3Client)).setBucket("my-temporal-payloads").build();To select an S3 bucket for each payload, use
setBucketResolver()instead ofsetBucket(). -
Add the driver to
ExternalStorage, then set it onWorkflowClientOptions. A Worker created from that Client inherits the configuration:ExternalStorage externalStorage = ExternalStorage.newBuilder().setDriver(driver).build();WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();WorkflowClient client =WorkflowClient.newInstance(service,WorkflowClientOptions.newBuilder().setExternalStorage(externalStorage).build());WorkerFactory factory = WorkerFactory.newInstance(client);Worker worker = factory.newWorker("my-task-queue");Configure External Storage on every Client and Worker process that can send or receive an offloaded payload. For example, a Client that starts a Workflow needs it to offload a large input, and a separate Worker process needs it to retrieve that input.
By default, the SDK offloads serialized Payloads that are 256 KiB or larger. For other thresholds, see Configure payload size threshold.
The S3 driver stores serialized Payloads under content-addressed keys derived from their SHA-256 hash. It reuses an
existing object when a Workflow Run passes the same payload again, verifies the hash on retrieval, and rejects payloads
larger than 50 MiB by default. Use setMaxPayloadSize() to change that limit.
Implement a custom storage driver
To use a storage system other than S3, implement StorageDriver. The following driver stores Payload protobuf messages
on local disk. Use it for local development and tests, not in production: Workers on other hosts cannot read its files.
A production driver must use a durable store that every Client and Worker can access.
class LocalDiskStorageDriver implements StorageDriver {
private static final String CLAIM_PATH = "path";
private final Path storeDir;
LocalDiskStorageDriver(Path storeDir) {
this.storeDir = storeDir;
}
@Override
public String getName() {
return "local-disk";
}
@Override
public String getType() {
return "local-disk";
}
@Override
public CompletableFuture<List<StorageDriverClaim>> store(
StorageDriverStoreContext context, List<Payload> payloads) {
try {
context.getCancellationToken().throwIfCancellationRequested();
Path directory = storeDirectory(context);
Files.createDirectories(directory);
List<StorageDriverClaim> claims = new ArrayList<>();
for (Payload payload : payloads) {
context.getCancellationToken().throwIfCancellationRequested();
Path file = directory.resolve(UUID.randomUUID() + ".bin");
Files.write(file, payload.toByteArray());
claims.add(new StorageDriverClaim(Map.of(CLAIM_PATH, file.toString())));
}
return CompletableFuture.completedFuture(claims);
} catch (IOException e) {
return failedFuture(new IllegalStateException("Could not write Payload", e));
}
}
@Override
public CompletableFuture<List<Payload>> retrieve(
StorageDriverRetrieveContext context, List<StorageDriverClaim> claims) {
try {
context.getCancellationToken().throwIfCancellationRequested();
List<Payload> payloads = new ArrayList<>();
for (StorageDriverClaim claim : claims) {
context.getCancellationToken().throwIfCancellationRequested();
Path file = Paths.get(claim.getClaimData().get(CLAIM_PATH));
payloads.add(Payload.parseFrom(Files.readAllBytes(file)));
}
return CompletableFuture.completedFuture(payloads);
} catch (IOException e) {
return failedFuture(new IllegalStateException("Could not read Payload", e));
}
}
private Path storeDirectory(StorageDriverStoreContext context) {
StorageDriverTargetInfo target = context.getTarget();
if (target instanceof StorageDriverWorkflowInfo) {
StorageDriverWorkflowInfo workflow = (StorageDriverWorkflowInfo) target;
if (workflow.getId() != null) {
return storeDir.resolve(workflow.getNamespace()).resolve(workflow.getId());
}
} else if (target instanceof StorageDriverActivityInfo) {
StorageDriverActivityInfo activity = (StorageDriverActivityInfo) target;
if (activity.getId() != null) {
return storeDir.resolve(activity.getNamespace()).resolve(activity.getId());
}
}
return storeDir;
}
private static <T> CompletableFuture<T> failedFuture(Throwable error) {
CompletableFuture<T> result = new CompletableFuture<>();
result.completeExceptionally(error);
return result;
}
}
store() writes each serialized Payload and returns one StorageDriverClaim with its path, in the same order. It uses
the Workflow or Activity information from StorageDriverStoreContext to group the files. retrieve() reads each file
from its claim and returns the original Payloads in the same order. The Payload Converter and Payload Codec have already
encoded the application data before the driver receives it.
Give every driver instance a stable, unique getName() value. The SDK records that name in a reference and uses it to
choose the driver during retrieval. getType() identifies the driver implementation for Worker heartbeats and metrics;
keep it the same for every configuration of the same driver. For an asynchronous storage client, use each context's
cancellation token to cancel its in-flight request when the SDK abandons the operation.
Register the driver with ExternalStorage using the setup in Store and retrieve large payloads with Amazon S3.
Configure payload size threshold
The size threshold applies to the serialized Payload, including its metadata. By default, serialized Payloads that are
256 KiB or larger are offloaded. Payloads smaller than the threshold stay inline in Event History. Set a higher value to
offload less data or set the value to 0 to offload every Payload.
return ExternalStorage.newBuilder().setDriver(driver).setPayloadSizeThreshold(0).build();
Use multiple storage drivers
When you register more than one driver, you must set a StorageDriverSelector. The selector chooses the registered
driver that stores each new Payload. It can return null to leave a specific Payload inline. Drivers that the selector
does not choose remain available for retrieval, which lets you migrate storage backends without making existing
references unreadable.
Every registered driver needs a distinct getName() value. For example, set a distinct name on each S3StorageDriver
when registering two S3 drivers. The following configuration stores new Payloads with preferredDriver, while keeping
legacyDriver available to retrieve references that it created:
return ExternalStorage.newBuilder()
.setDrivers(Arrays.asList(preferredDriver, legacyDriver))
.setDriverSelector((context, payload) -> preferredDriver)
.build();
Configure multi-region durability with Amazon S3
To tolerate an AWS Region failure, configure Cross-Region Replication (CRR) and an S3 Multi-Region Access Point (MRAP), then set the driver bucket to the MRAP ARN. Enable ARN-region routing on the AWS SDK client so it sends a request to the Region in the ARN:
S3AsyncClient s3Client =
S3AsyncClient.builder()
.region(Region.US_EAST_2)
.serviceConfiguration(S3Configuration.builder().useArnRegionEnabled(true).build())
.build();
return S3StorageDriver.newBuilder()
.setClient(new S3AsyncClientAdapter(s3Client))
.setBucket("arn:aws:s3::123456789012:accesspoint/example.mrap")
.build();
CRR is asynchronous. During replication lag, a Worker in another Region can temporarily fail to retrieve a new object. Use appropriate Activity retry policies and prefer the same Region for an immediate read. See Durable External Storage for the replication trade-offs and Replication Time Control if you need a replication-time service-level agreement.
Manage external objects
Temporal does not delete objects from your S3 bucket. Configure an S3 lifecycle rule with a TTL longer than the maximum Workflow Run Timeout plus the Namespace retention period. For the formula and guidance for multi-region storage, see Lifecycle management and Durable External Storage.