Benchmarks

The following section provides some performance figures for Private AI's CPU and GPU containers on various AWS instance types, including the hardware in the system requirements.

Notes:

• Throughput is given in words, where a word denotes a whitespace-separated piece of text.
• Words are roughly equivalent to tokens, such as used by OpenAI to measure volume.
• Throughput and latency tests are performed with a 100 word/500 character test input.
• Unless otherwise stated, tests are run using the default "high" accuracy_mode .
• All benchmarks used concurrency settings optimized for throughput.

Please contact us if you require any further information.

Key Takeaways

• Private AI utilises the latest in transformer technology to deliver the highest possible PII detection performance. At the same time, Private AI runs tens of times faster that BERT-style models and hundreds of times faster than LLMs without compromising accuracy.
• CPU instances are fine for most use cases, with even a single CPU core able to process 500 words/s.
• For very large deployments, GPU instances are recommended. A single low cost inferance instance such as the g4dn.2xlarge ($0.752 per hour) can process 1GB of unicode text in under an hour.
• Hardware type matters. m5zn instances powered by recent Intel Xeon CPUs with AVX512 VNNI support perform over 3X faster than generic instances like c5 . For this reason, it is recommended to use the hardware specified in the system requirements .
• It is best to avoid AWS Fargate, which is typically provisioned with older CPUs like the c5 .
• Latency scales approximately linearly with request length. Throughput is unaffected.
• Batching does not improve throughput significantly. Instead, it is recommended to use a large number of concurrent requests.
• GPU instances are recommended when processing files.

CPU

The below table illustrates the performance of the CPU container on various AWS instance types:

For best throughput, it is recommended to use single logical core workers. The below table illustrates the scaling efficiency when running the container on multiple CPU cores:

The default accuracy_mode value high offers best PII detection performance, however it can be changed to trade PII detection performance for speed:

GPU

Below are benchmarks of the GPU container running on a g4dn.2xlarge instance when optimized for throughput with 128 concurrency:

The above GPU container benchmarks are optimized for throughput. Latency as low as 10ms can be achieved when using a lower number of concurrent requests.

PDF

Below are benchmarks of the CPU and GPU container for typical deployments to help with file processing estimates. PDFs were pre-processed to split large files into 5 page chunks to improve throughput via parallelization.

Audio

Below are benchmarks of the CPU and GPU container for the typical deployments to help with audio file processing estimates.

Note: Benchmarks for Audio are given as the multiple of the audio length. For example, "20x realtime" means 20 minutes of audio will be processed in a minute.