Reading files from remote storage
By mounting external storage systems to the Limina container, teams can take advantage of a wide array of different technologies to source data for input. This can simplify the overall design of data pipelines and allow teams to take advantage of the feature sets present in those underlying storage providers.S3 Object Storage
When working with S3 buckets, it can be useful to mount the bucket and process files directly without needing to encode the file and transfer data over the network. The guide below demonstrates how to do this using Amazon’s Mountpoint application. There are two primary steps required to process files directly from object storage:- Configure the container / host machine for S3 access
- Process files using the
/process/files/uriendpoint
Prerequisites and assumptions
The/process/files/uri endpoint assumes that all files are available in local storage in the running container.
At the time of writing, there are no connectors or drivers deployed with the Limina container that enable communication with any additional protocols.
As such, any storage volume mounted to Docker is available via the /process/files/uri endpoint.
For S3 object storage, this guide assumes that you have:
- An active AWS account
- An S3 bucket
- A running Limina instance
- AWS CLI installed on the host
- Appropriate software installed on the host to enable mounting an S3 bucket as local storage
Step 1 - Set up the S3 mount point
Ensure you have connected your AWS account. Follow the documentation here to assist with authentication: https://github.com/awslabs/mountpoint-s3/blob/main/doc/CONFIGURATION.md#aws-credentials The format of the command to mount the S3 bucket is fairly straightforward:Step 2 - Modify the Docker run / container start parameters to mount the local folder
The Docker run command needs to be modified to include a new volume linked to the mount point. This is done by using the-v parameter on the command line.
Example:
curl
S3 File Support (New in v4.4.0)
The DEID container can now process files stored directly in Amazon S3 without requiring any local file mounts or volume bindings. You specify input and output locations usings3:// URIs, and the container handles downloading and uploading automatically.
Required Environment Variables
To use S3 URIs for input or output, you must set three environment variables:
These three variables are required whenever an
s3:// URI is used for input or output. Note that PAI_OUTPUT_FILE_DIR is always needed for the /process/files/uri endpoint, but can be set to either a local path or an s3:// URI — it is not S3-specific.
Processing Files from S3 (Input)
This is a new way to specify file paths for the/process/files/uri endpoint — instead of local mount paths, you can now pass s3:// URIs directly. For details on the /process/files/uri endpoint itself, see Processing Files with the URI Endpoint.
To process files from S3 buckets, you can use either the raw HTTP API or our Python client:
Python
Writing Output to S3 (Output)
SetPAI_OUTPUT_FILE_DIR to an s3:// URI when starting the container. The redacted file will be written to that path with .redacted appended to the original filename (e.g., s3://my-output-bucket/redacted/document.redacted.pdf).
Specifying S3 as Output Destination — Bucket Name Requirement
The bucket name is mandatory and must be the first segment afters3://:
s3://my-output-bucket/redacted/
An invalid example (missing bucket): s3:///redacted/
Mixing Input from Local Mount + Output to S3
You can use a mounted directory for input while writing output to S3:Python
PAI_OUTPUT_FILE_DIR (S3 in this case).
Container startup example
Docker Command
Mixing Input from S3 + Output to Local Mount
You can also use ans3:// URI for input while writing output to a locally mounted directory:
Python
PAI_OUTPUT_FILE_DIR (a local mount in this case).
Container startup example
Docker Command
How It Works
When ans3:// URI is provided as input, the container:
- Validates that the S3 bucket exists and is accessible
- Fetches the file for processing. Nothing is held permanently inside the container.