Deployment Considerations

This is a set of best practices for deploying the Private AI container. Particular focus is given to health checks, which when configured correctly allow the system to recover from any crashes or other problems.

When running the container with an orchestrator like Kubernetes or Docker Swarm Mode, we recommend you to leverage the orchestrator's health check mechanism rather than using the built-in restart capability of Docker.

Running a single container

If your use case requires running a single container for a limited period of time (e.g. a batch job), it is possible to start the container directly using the docker CLI.

When you do so, it is possible to leverage the Docker restart option to allow for your task to run to completion even in case of failures. Use this command to start the container with restarts enabled:

docker run -d -p 8080:8080 --name privateai --restart unless-stopped deid:<version-number>

Use this command to stop the container:

docker stop privateai

Your task should also be written in a way that it probes the container for liveness using the /healthz route. Set your code to call the /healthz route every 5 seconds until the route is responding with status code 200. The container is now ready to receive traffic on the /deidentify_text endpoint.

In most environments, the container is ready to receive traffic in less than a minute.

Running in AWS ECS

Resources

Make sure you review the System Requirements section to set the proper resources in your ECS task description.

Healthcheck

You can set your healthcheck configuration in two different ways in ECS: at the load balancer (if available) or in the task definition. The best option depends on your use case.

When setting the healthcheck in the ECS task (note that this is only available when using the old ECS user experience) you will need to set the following ECS Task definition parameters to these recommended values:

and the following recommended command:

CMD-SHELL, curl -f http://localhost:8080/healthz || exit 1

If your ECS deployment contains a load balancer you should follow the guidelines under Running on Kubernetes

Task Placement

We recommend that you run only one task per host. EC2 and EXTERNAL launch types support a Task Placement option which you can select to One Task Per Host. For FARGATE launch type, the placement will be done automatically and will be spreaded across the Availability Zones.

Running on Kubernetes

The same System Requirements apply when setting the container to run under Kubernetes. Make sure that you set the requirements specific to the image provided by Private AI. Moreover, we recommend that you run only one Private AI container per node (see spec.affinity fied) and set the liveness and readiness probes (see spec.containers[*].livenessProbe and spec.container[*].readinessProbe fields) according to this example.

apiVersion: v1
kind: Pod
metadata:
  labels:
    name: deid-pod-example
spec:
  affinity: # to make sure that at max one pod is scheduled per node
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: name
                operator: In
                values:
                  - deid-pod-example
          topologyKey: "kubernetes.io/hostname"
  containers:
    - name: deid-container-example
      image: deid:2.10full # update with your image registry and version
      resources:
        requests:
          cpu: 2 # update with recommended requirements for your image / Instance Type
          memory: 6Gi # update with recommended requirements for your image / Instance Type
        limits:
          cpu: 4 # update with recommended requirements for your image / Instance Type
          memory: 8Gi # update with recommended requirements for your image / Instance Type
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 30
        periodSeconds: 10
        failureThreshold: 3
        timeoutSeconds: 5
      readinessProbe:
        httpGet:
          path: /healthz
          port: 8080
        initialDelaySeconds: 60
        periodSeconds: 60
        failureThreshold: 3
        timeoutSeconds: 5
  terminationGracePeriodSeconds: 30