Skip to main content

​
Prerequisites

Please only run one CPU container per machine, and only one GPU container per GPU. Running multiple containers that share the same CPU or GPU results in vastly reduced performance. To run several GPU containers on a machine with multiple GPUs, see Running on Multi-GPU Machines.
The following prerequisites are required to run the container:
  • Container engine, such as Docker (can be installed using the official instructions)
  • (GPU only) Nvidia Container Toolkit with Nvidia driver version 515 or higher (can be installed using the following installation guide)
All other dependencies, such as CUDA are included with the container and don’t need to be installed separately.

​
System Requirements

The image comes in two different build flavours:
  • A compact, CPU-only container that runs on any Intel or AMD CPU and a container with GPU acceleration. The CPU container is highly optimised for the majority of use cases, as the container uses hand-coded AMX/AVX2/AVX512/AVX512 VNNI instructions in conjunction with Neural Network compression techniques to deliver a ~25X speedup over a reference transformer-based system.
  • A GPU container is designed for large-scale deployments making billions of API calls or processing terabytes of data per month.

​
Minimum Requirements

The minimum system requirements for the container image are as follows: Please visit the recommended concurrency levels page to set the number of concurrent requests optimally.
The gpu-synthetic-enhanced image (new in 4.5) needs more VRAM than the requirements above. A GPU with at least 24GB of VRAM is recommended. A container uses only one GPU, so this VRAM must be available on a single GPU. See Enhanced Synthetic Container for the full deployment details, including recommended AWS EC2 instance types.
The Limina image can also run on the new Apple chips, such as the M2. Performance will be degraded however, due to the Rosetta 2 emulation of the AVX instructions.

​
CPU

While Limina CPU-based container will run on any x86-compatible instance, the below cloud instance types give optimal throughput and latency per dollar:
In the event when a lower latency is required, the instance type should be scaled; e.g. using an m7i.2xlarge in place of a m7i.xlarge. While the Limina container solution can make use of all available CPU cores, it delivers best throughput per dollar using a single CPU core machine. Scaling CPU cores does not result in a linear increase in performance.

​
GPU

Similarly for the GPU-based image, Limina recommends the following Nvidia T4 GPU-equipped instance types: