Skip to main content

Reading files from remote storage

By mounting external storage systems to the Limina container, teams can take advantage of a wide array of different technologies to source data for input. This can simplify the overall design of data pipelines and allow teams to take advantage of the feature sets present in those underlying storage providers.

S3 Object Storage

When working with S3 buckets, it can be useful to mount the bucket and process files directly without needing to encode the file and transfer data over the network. The guide below demonstrates how to do this using Amazon’s Mountpoint application. There are two primary steps required to process files directly from object storage:
  1. Configure the container / host machine for S3 access
  2. Process files using the /process/files/uri endpoint

Prerequisites and assumptions

The /process/files/uri endpoint assumes that all files are available in local storage in the running container. At the time of writing, there are no connectors or drivers deployed with the Limina container that enable communication with any additional protocols. As such, any storage volume mounted to Docker is available via the /process/files/uri endpoint. For S3 object storage, this guide assumes that you have:
  • An active AWS account
  • An S3 bucket
  • A running Limina instance
  • AWS CLI installed on the host
  • Appropriate software installed on the host to enable mounting an S3 bucket as local storage
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mountpoint.html

Step 1 - Set up the S3 mount point

Ensure you have connected your AWS account. Follow the documentation here to assist with authentication: https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials The format of the command to mount the S3 bucket is fairly straightforward:
Pay special attention to the UID and GID values, and set the appropriate user and group for your configuration. The command above sets the logged-in user and group to have permission to access the mount.

Step 2 - Modify the Docker run / container start parameters to mount the local folder

The Docker run command needs to be modified to include a new volume linked to the mount point. This is done by using the -v parameter on the command line. Example:
Once the steps above are complete, the container will start and requests can be issued using the following payload format:
curl

S3 File Support (New in v4.4.0)

The DEID container can now process files stored directly in Amazon S3 without requiring any local file mounts or volume bindings. You specify input and output locations using s3:// URIs, and the container handles downloading and uploading automatically.

Required Environment Variables

To use S3 URIs for input or output, you must set three environment variables: These three variables are required whenever an s3:// URI is used for input or output. Note that PAI_OUTPUT_FILE_DIR is always needed for the /process/files/uri endpoint, but can be set to either a local path or an s3:// URI — it is not S3-specific.

Processing Files from S3 (Input)

This is a new way to specify file paths for the /process/files/uri endpoint — instead of local mount paths, you can now pass s3:// URIs directly. For details on the /process/files/uri endpoint itself, see Processing Files with the URI Endpoint. To process files from S3 buckets, you can use either the raw HTTP API or our Python client:
Python
This works for all supported file types.

Writing Output to S3 (Output)

Set PAI_OUTPUT_FILE_DIR to an s3:// URI when starting the container. The redacted file will be written to that path with .redacted appended to the original filename (e.g., s3://my-output-bucket/redacted/document.redacted.pdf).

Specifying S3 as Output Destination — Bucket Name Requirement

The bucket name is mandatory and must be the first segment after s3://:
A valid example: s3://my-output-bucket/redacted/ An invalid example (missing bucket): s3:///redacted/

Mixing Input from Local Mount + Output to S3

You can use a mounted directory for input while writing output to S3:
Python
The input is read from the local mount, and the output goes to PAI_OUTPUT_FILE_DIR (S3 in this case). Container startup example
Docker Command

Mixing Input from S3 + Output to Local Mount

You can also use an s3:// URI for input while writing output to a locally mounted directory:
Python
The input is fetched from S3, and the output is written to PAI_OUTPUT_FILE_DIR (a local mount in this case). Container startup example
Docker Command

How It Works

When an s3:// URI is provided as input, the container:
  1. Validates that the S3 bucket exists and is accessible
  2. Fetches the file for processing. Nothing is held permanently inside the container.

Writing to S3

The container processes files synchronously. When writing to different S3 keys at the same time, each request uses its own client and target path, so there are no conflicts. If multiple requests write to the same key, the last one wins — this is standard S3 behavior. To avoid unexpected overwrites, use unique paths per user or session (e.g., include a user ID or timestamp).