On-premise Installation

Introduction

vali-Das is a software service provided by Veridas which performs onboarding processes based on document validation and face biometry. This service exposes a REST API that can be used to call all of its functions and it is released as a configurable Docker solution which can be deployed easily by using docker commands.

The API allows the user to read and extract information about I.D. documents from several countries and validate them using multiple techniques and security measures like face biometry techniques for ensuring the identity of the person.

On-premise Docker Deployment Instructions

Veridas API product for onboarding process is offered as a cloud SaaS and as on-premise solutions. The on-premise one is released as a configurable docker solution which can be deployed easily by using docker commands.

First of all, to load the docker image from the provided tar file, the "docker load" command has to be run.

Some of the specifications are hardcoded while others can be configured by means of environmental variables and logical data volumes. vali-Das product runs on a Docker container which is built over:

• Ubuntu 18.04 LTS.
• Gunicorn server (with or without SSL).
• Flask web framework.
• A PostgreSQL or Oracle database.

The first time this container runs, it assumes the database access given in the environmental variables is valid, uses these credentials to build and prepare the database, and also populates the database with an initial dump. For this purpose, this image imports a few data volumes from the running localhost machine, using them as persistent storage for configuration files. In case we want to deploy many instances of vali-Das, the logical volume used for data storage and database configuration data, can be shared between all of the instances. Others, as output logs, should be kept isolated. By doing it this way, the next time this container is created, it will use the configuration of the previous one, unless the user erased all the files of the data volumes.

In order to achieve a proper deployment, it is important to be in a clear environment the first time this container is created. This way, we ensure the database and other files and directories are synchronized.

vali-Das Host Requirements

Regular Usage

For use-cases with sustainable workload of up to 10 onboarding processes per minute:

• Server with i5 2x core processor with minimum of 2.5 GHz and 8GB RAM.

Intensive Usage

• Add one server of the type indicated on the "Regular Usage" part (or increase the hardware in a similar way) each 10 onboarding processes per minute are added to the workload.

Common requirements

• Ubuntu 18.04 LTS O.S. with Docker and Logrotate installed. This is the OS which Veridas supports and validates with each release, but as the product is docker-based, it should work in any Linux environment, specially Debian based distros.
• A domain name (optional, but recommended)
• An SSL certificate associated to the domain which is used on the docker container deployment (optional, but recommended)
• PSQL or Oracle database (it may be deployed as a docker container or use an existing database).
• Appropriate disk space depending on the system usage, considering a disk usage of around 30 MB per validation.
• Ad-hoc data backup service is out of the scope of this document, but its implementation highly recommended

Database Host Requirements for a Dedicated Machine Case

Regular Usage

For use-cases with sustainable workload of up to 10 onboarding processes per minute:

• Computer with i5 2x core processor with minimum frequency of 2.5 GHz and 4GB RAM

Intensive Usage

For workloads between 10 and 50 onboarding processes per minute:

• Computer with i5 4x core processor with minimum frequency of 2.5 GHz and 8GB RAM

For higher usage requirements, please ask your solution provider.

In Both Scenarios

• Appropriate disk space depending on the usage estimations and considering a growth of around 30 MB per validation process (this is the same requirement indicated on vali-Das section).
• Minimum Postgres version required is 15, being the 15.5 the recommended one. For Oracle databases version 19c or superior is recommended.
• There is no condition for the type of disks. SATA, SSD, SAS and other are valid, obtaining small performance difference depending of its features but with little impact on the global system performance
• RAID 1 at least is recommended

Docker Container Configuration

The container creation procedure is configured by using many environmental variables which can be given to _docker run _command or in a docker-compose.yml file. The default value of these variables is given after the colon signs.

General Configuration

• USAGE_TRACKER_SERVER_URI: https://api-work.eu.veri-das.com/ls/v1 => License server URI.
• USAGE_TRACKER_API_KEY: apikey => License server apikey.
• USAGE_TRACKING_FORK_TRACKER_PROCESS: 1 => With a value of 1, usage tracker log the queries. Each USAGE_TRACKING_POLLING_INTERVAL seconds will spawn a separate thread to send the logged queries to the license server. With a value of 0, usage tracker will send each query to the license server.
• USAGE_TRACKING_POLLING_INTERVAL: 300 => Set the interval on which the log file will be polled for changes, in seconds.
• USAGE_TRACKER_FOLDER: /opt/validas => Name of the folder where the usage tracker file will be stored.
• USAGE_TRACKER_FILENAME: logfile => Name of the usage tracker file.
• TZ: Europe/Madrid => Desired Timezone.
• ENABLE_SSL: yes => Indicates if the system requires SSL or not.
• WORKERS: 1 => Number of Gunicorn Workers.
• PORT: 5090 => the port that the service container is going to expose. If this variable is not set, the exposed PORT will be 8810.
• DEBUG: no => Enables or disables debug mode. In debug mode, logs generated by the service will be more detailed and certain temporal files are not deleted after the processing.

To make vali-Das work properly with all the available features, three services have to be deployed alongside with vali-Das; das-Face, verilive and IDentidas, which respectively are the face biometry analysis, proof of live video analysis and ID document analysis and OCR services.

To allow vali-Das to discover and communicate with these services, the URL of them have to be provided by means of docker environment variables.

• FACE_BIOMETRICS_URL: https://face-bio-service:5030
• PROOF_OF_LIFE_URL: https://proof-life-service:5040
• DOCUMENT_ANALYSIS_URL: https://doc-analysis-service:5050

Database configuration

vali-Das uses a database to store all the application data. The database configuration is done by using the following environment variables:

PostgreSQL database configuration

• DB_ENGINE: postgresql => Defines the type of database engine that the application uses. There are two possible values: postgresql (by default) and oracle.
• DB_HOST: pgsql-validas => The host where DB is deployed. Can be the container name or the IP of the host where the DB is deployed.
• DB_PORT: 5432 => Port of the database.
• DB_NAME: validas => Name of the database for vali-Das.
• DB_USER: validas => Name of the user granted to DB_NAME with DB_PASS.
• DB_PASS: validas => Name of the default DB password.
• DB_SCHEMA: public => Name of the database schema. When a schema other than "public" is selected, the schema must exist in the database before running the application.
• MIGRATIONS: no => The container needs to do database migrations to prepare the database with the last models and configurations. It is required to do the first time or when upgrading a new version. The possible values are yes or no.

Oracle Database configuration

• Using Oracle SID:

  • DB_ENGINE: "oracle"=> Defines the type of database engine that the application uses. There are two possible values: postgresql (by default) and oracle.
  • DB_HOST: oracle-validas => The host where DB is deployed. Can be the container name or the IP of the host where the DB is deployed.
  • DB_PORT: 5432 => Port of the database
  • DB_NAME: validas => Name of the database for vali-Das.
  • DB_USER: validas => Name of the user granted to DB_NAME with DB_PASS.
  • DB_PASS: validas => A dummy default password.

• Using Oracle Easy Connect or Oracle Net connect descriptor:

  Notice that for these connection strings, variables DB_PORT and DB_HOST are not set. Instead, the entire connection string is set in the DB_NAME variable.

  • DB_ENGINE: "oracle"=> Defines the type of database engine that the application uses. There are two possible values: postgresql (by default) and oracle.
  • DB_NAME:
    • Example of Easy Connect: <host>:<port>/<service_name>
    • Example of Net connect descriptor: (DESCRIPTION=(FAILOVER=on)(ADDRESS=(PROTOCOL=tcp)(HOST=host)(PORT=port))(CONNECT_DATA=(SERVICE_NAME=service_name)))
  • DB_USER: validas => Name of the user granted to DB_NAME with DB_PASS.
  • DB_PASS: validas => A dummy default password.

Logging Configuration

The format, behaviour and storage of logs can be configured by using these environment variables:

• LOG_FORMAT: console-simple => Format of the printed log messages. Possible values are:
  • console: Colorized output meant for development/debugging using the console on a local environment.
  • console-simple: Same as the ‘console’ format but less verbose.
  • plain: Same as the ‘console’ format but without colors (plain text output).
  • json: JSON-formatted log messages (each line is a JSON structure). Recommended for production environments.
• LOG_HANDLER: stdout => Handler used to collect log messages:
  • stdout: Output log mesages to the standard output.
  • file: Print log messages to a rotating log file.
  • dailyfile: Print log messages to a daily-rotating log file.
• LOG_TIMESTAMP_UTC: yes => Defines if the log timestamps wants to be in UTC time (yes) or in the docker container timezone (no).
• LOG_BACKUP_COUNT: 100 => Defines the maximum number of rotated backup logs.
• LOG_MAX_BYTES: 1048576 => Defines the maximum size in bytes of the log file. Once the log file reaches this amount it will rotate. Only works with LOG_HANDLER=file.
• LOG_FOLDER: /var/log/validas => The folder to hold the log files. If you want the files in this folder to persist, please mount it as a volume.

Volumes Configuration

Besides these variables, a few data volumes are necessary for a more convenient use of vali-Das containers. Some of these volumes require permissions for www-data user (uid=33):

• /opt/validas/media: This volume is used by the server to keep selfie images, selfie videos, ID document images and the rest of the resources used on validations. It should be shared among containers sharing the same database. This folder requires www-data user permissions and it must be created before running vali-Das service.
• /var/log/validas: This volume persists log files generated by the Flask App. Each container generates its logs. This folder requires www-data user permissions and it must be created before running vali-Das service.
• /etc/validas/security/certs/: This volume is only necessary for deployment of vali-Das using SSL (ENABLE_SSL=yes.) with certificates different from the default ones. We recommend to mount this volume with read-only parameter. It should contain the following SSL keys and certs: * server.crt * server.key

These volumes must be properly configured in the deployment tool scripts, as it is indicated in the section 3.1 vali-Das service deployment of the current document, to allow the service docker containers to write and read from them.

Docker Container Creation

vali-Das on-premises comprises the containers deployed by using the given validas:VALIDAS-SERVICE_VERSION Docker image of vali-Das, where current version of the image is the following:

VALIDAS-SERVICE_VERSION = 2.32.X

vali-Das service deployment

In case you want to use docker tool (https://docs.docker.com/get-started/part2/) for the containers deployment, using a database deployed somewhere else, you can use the following instructions on the terminal. It assumes that the database has been initialized properly with access granted to DB_USER with DB_PASS. It is important to take into account that the services have to be deployed in the same docker network to allow each container to reach each other.

The deployment can also be done by using docker-compose tool (https://docs.docker.com/compose).

We recommend using docker-compose tool because the integration should be easier and only need one step to up vali-Das service.

For a complete containerized solution which uses a PSQL docker container as a database system, the following steps must be followed.

If vali-Das is being deployed for the first time, the container and the database have to be initialized. Similarly, if it is being upgraded from a previous version, the database has to be upgraded also before deploying the new version.

To achieve this action, the MIGRATIONS environmental variable should be "yes".

#docker-compose.yml
version: '2.1'
services:
  pgsql-validas:
    image: postgres:15.5
    container_name: pgsql-validas
    environment:
      POSTGRES_DB: validas
      POSTGRES_USER: validas
      POSTGRES_PASSWORD: validas
      PGDATA: /var/lib/postgresql/data/pgdata
    volumes:
      - ./vols/pgdata:/var/lib/postgresql/data/pgdata
  validas-service:
    image: "${DOCKER_REGISTRY_IP:PORT}/validas:${VALIDAS-SERVICE_VERSION}"
    container_name: validas-service
    restart: always
    depends_on:
      - pgsql-validas
    environment:
      TZ: Europe/Madrid
      DB_HOST: pgsql-validas
      DB_USER: validas
      DB_PASS: validas
      DB_NAME: validas
      DB_PORT: 5432
      WORKERS: 1
      PORT: 5090
      ENABLE_SSL: "TRUE"
      FACE_BIOMETRICS_URL: https://face-bio-service:5030
      PROOF_OF_LIFE_URL: https://proof-life-service:5040
      DOCUMENT_ANALYSIS_URL: https://doc-analysis-service:5050
      USAGE_TRACKER_SERVER_URI: https://api-work.eu.veri-das.com/ls/v1
      USAGE_TRACKER_API_KEY: "apikey"
      USAGE_TRACKING_FORK_TRACKER_PROCESS: 1
      USAGE_TRACKING_POLLING_INTERVAL: 300

   ports:
      - 5090:5090
   volumes:
      - ./vols/media:/opt/validas/media
      - ./vols/logs:/var/log/validas
      - ./vols/certs:/etc/validas/security/certs/

For running this docker-compose script, the following command has to be executed, supposing that the docker-compose file shown above is called "docker-compose.yml".

$ docker-compose pull # Just if the images are in a docker registry
$ docker-compose -f docker-compose.yml up --abort-on-container-exit --force-recreate

Container Cleaning

In case of failure during these steps, the first time it is executed, it is required to clean all volumes and recreate them as empty volumes. Similarly, it is required to remove any container left on the machine. It can be done as follows:

$ sudo docker rm CONTAINER_ID

Additional Considerations

Upgrading PostgreSQL DB engine from version 12 to version 15

If a PSQL docker container is used as a database system, the following steps must be followed if you want to migrate from the older recommended version 12 to the new recommended version 15.5.

It is important to note that November 14, 2024 is the date of the end of life of the v12 of PSQL, so if you want to have support from Veridas (and from PostgreSQL) from that date, it is mandatory to upgrade.

The first vali-Das version which supports the PSQL v15 is the v2.28, so if you have any version prior of this, please, update first to vali-Das v2.28 or superior to do the PSQL upgrade.

This is a disruptive process in which there is an availability loss.

The steps to do the PSQL docker container upgrade from v12 to v15.5 are the following:

1. Ensure that vali-Das version is v2.28 or higher

2. It is Optional but highly recommended to do a backup or a snapshot of the database, to be used as a fast remediation if something goes wrong

3. Keep pgsql-validas container awake and stop validas-service container. If more than one vali-Das application instances (containers) are deployed, all of them must be stopped:

   docker stop validas-service
   

4. Run the following command:

   docker exec -it pgsql-validas pg_dumpall -U validas > pgdump.out
   

5. Stop postgresql container:

   docker stop pgsql-validas
   

6. Delete postgresql volumes (vols directory) or create a new one (this option is more safe) and assign the corresponding permissions (more information here).

7. Change to desired postgresql (pgsql-validas) image version in docker-compose.yml file.

8. Change validas-service volumes in docker-compose.yml file (if you did create new ones).

9. Run only pgsql-validascontainer:

   docker-compose up -d pgsql-validas
   

10. Run the following command:

    docker cp pgdump.out pgsql-validas:/
    

11. Run the following command:

    docker exec -it pgsql-validas psql -f pgdump.out -U validas
    

12. Stop postgresql container:

    docker stop pgsql-validas
    

13. In postgresql version 14, password hashing method changed from MD5 to SCRAM-SHA-256. This implies that all database users passwords must be reseted (in this example there is only a user called validas):

    docker exec -it pgsql-validas psql -U validas
    \password
    

14. If the password is different from the last one don't forget to set it in the docker-compose environment variables.

15. Run validas-service deployment as usual

For more information you can check https://www.postgresql.org/docs/15/upgrading.html

Database Backup

The following steps must be followed if you want to do a consistent backup of all the vali-Das information:

1. Stop vali-Das service by stopping its docker, or all the dockers of the vali-Das cluster in case of a distributed implementation

   docker stop validas-service
   

2. Perform a backup of the postgreSQL database. For more information about Postgresql Backup and Restore options you can check https://www.postgresql.org/docs/15/backup.html.

3. Backup the volume mount points for media assets and logs. See Volumes Configuration for more information.

4. Start again vali-Das docker, or all the dockers of the vali-Das cluster in case of a distributed implementation.

If the volume mount points are sitting on a storage cabinet you might want to leverage the snapshot capabilities of the cabinet to reduce the downtime due to the backup window. Same applies for the database datastores.

It is highly recommended to perform this backup operation during low user activity to minimize the amount of processes non gathered from the cloud after it resumes its operation.

Database Restore

The following steps must be followed for restoring a backup of all the vali-Das information:

1. Prepare a new deployment following vali-Das on-premise docker deployment instructions steps.

2. Restore the data of the volume mount points for media assets and logs in the new path and configure the new volume mount point correctly.

3. Run only pgsql-validascontainer:

   docker run -d pgsql-validas
   

4. Perform a restore of the postgreSQL database. For more information about Postgresql Backup and Restore options you can check https://www.postgresql.org/docs/15/backup.html.

5. Stop postgresql container:

   docker stop pgsql-validas
   

6. Run validas-servicedeployment as usual.

Sequence Diagram