Installation

Installation procedure steps

To install the self-hosted Digital Twin Platform, perform the following steps:

1. Create DNS names
2. Set up network connectivity rules
3. Set up cloud access policies
4. Set up Kubernetes network policy
5. Add configurations to Prometheus
6. Create Kubernetes secrets
7. Configure Helm Charts

Click on each step for a detailed description.

Create DNS names

Create the four DNS names listed in the table below and point them to the load balancer (or ingress method of your choice). These DNS names are expected by the Platform, and the values are needed in the Helm Chart values.yaml configuration file.

DNS	Usage	Example
Web Applications	Static files served for the Reference, Console, and CAD Plugins applications	app.example-domain.com
REST API	REST APIs and server-side web application (for login) of the Platform Services	api.example-domain.com
Identity Service	Identity Service for the Platform	id.example-domain.com
Workbench	Platform Workbench application	workbench.example-domain.com

Set up network connectivity rules

Set up the network connectivity rules required by the Platform. The rules differ between AWS and OCI; refer to the appropriate table below.

Amazon Web Services

Source	Destination	Port	Description
Internet	Load Balancer	80/tcp 443/tcp	Allow incoming traffic from internet to load balancer to access cluster
Kubernetes	PostgreSQL or Proxy	5432/tcp	Access to PostgreSQL cluster
Kubernetes	MongoDB	27017/tcp	Access to MongoDB cluster
Kubernetes	Redis	6379/tcp	Access to Redis Cluster
Kubernetes	AWS MSK	2181/tcp 9096/tcp	Access to the ZooKeeper and bootstrap ports of the Kafka cluster
Kubernetes	AWS MQ	1183/tcp 61616/tcp	For MQTT and TCP access to ActiveMQ cluster
Kubernetes	Neo4j	7687/tcp	For access to Neo4j
Kubernetes	SMTP	587/tcp	Access to SMTP service (adjust ports as necessary for customer configuration)
Kubernetes	Internet	443/tcp	Access for services to access public APIs

Note: These rules do not include the ports required for the operation or management of the private networking gateways, clusters, etc.

Oracle Cloud Infrastructure

Source	Destination	Port	Description
Internet	Load Balancer	80/tcp 443/tcp	Allow incoming traffic from internet to load balancer to access cluster
Kubernetes	OCI Database PostgreSQL	5432/tcp	Access to PostgreSQL cluster
Kubernetes	ScaleGrid MongoDB	27017/tcp	Access to MongoDB cluster
Kubernetes	OCI Cache Redis	6379/tcp	Access to Redis cluster
Kubernetes	Apache Kafka	2181/tcp 9096/tcp	Access to the bootstrap ports of the Kafka cluster
Kubernetes	Apache ActiveMQ	1183/tcp 61616/tcp	For MQTT and TCP access to ActiveMQ cluster
Kubernetes	Neo4j	7687/tcp	For access to Neo4j
Kubernetes	SMTP	587/tcp	Access to SMTP service (adjust ports as necessary for customer configuration)
Kubernetes	Internet	443/tcp	Access for services to access public APIs

Set up cloud access policies

The Platform requires one cloud policy per object storage bucket. Refer to the sections below for setting up access policies in Amazon Web Services and Oracle Cloud Infrastructure.

Amazon Web Services access policies

Note: The Platform uses the AWS EKS Pod Identity Agent for credentials to access AWS services. Ensure that the EKS add-on is installed.

For AWS, create a role, trust policy, and access policy for each of the following five buckets:

• kafka
• filesvc
• scriptmanager
• datasourcesvc
• workflowsvc

Do the following:

1. Create a trust policy for the AWS EKS Pod Identity Agent to be used on all roles. See below.

{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Principal": {                "Service": "pods.eks.amazonaws.com"            },            "Action": [                "sts:TagSession",                "sts:AssumeRole"            ]        }    ]}

2. Create one policy per bucket that allows the role to locate the bucket and read/write objects.

Note: This may require an additional statement to grant access to custom encryption keys if required by customer policy.

{    "Statement": [        {            "Action": [                "s3:ListBucket",                "s3:GetBucketLocation"            ],            "Effect": "Allow",            "Resource": "${BUCKET_ARN}",            "Sid": "Buckets"        },        {            "Action": [                "s3:PutObject",                "s3:ListMultipartUploadParts",                "s3:GetObjectVersion",                "s3:GetObject",                "s3:DeleteObjectVersion",                "s3:DeleteObject",                "s3:AbortMultipartUpload"            ],            "Effect": "Allow",            "Resource": "${BUCKET_ARN}/*",            "Sid": "Objects"        }    ],    "Version": "2012-10-17"}

3. Create a role for each of the Kubernetes service accounts in the table below and attach policies as listed.

Kubernetes Service Account	Policy
aisvc	kafka
datasourcesvc	kafka, datasourcesvc
filesvc-migrations	kafka, filesvc
filesvc	kafka, filesvc
itemsvc-migrations	kafka, filesvc
itemsvc-rdbms-migrations	kafka, filesvc
itemsvc-telemetry-worker	kafka, filesvc
itemsvc-worker	kafka, filesvc
itemsvc	kafka, filesvc
objectmodelsvc	kafka
passportsvc-migrations	kafka, filesvc
passportsvc	kafka, filesvc
platform-notificationsvc-api	kafka
platform-notificationsvc-worker	kafka
scriptmanager	kafka, scriptmanager
workflowsvc-api	kafka, workflowsvc
workflowsvc-backend	kafka, workflowsvc
workflowwkr-backend	kafka, workflowsvc

Oracle Cloud Infrastructure access policies

If deploying to OCI, three statements are needed for a pod to access a bucket. Refer to the statements listed below:

Allow any-user to manage objectstorage-namespaces in compartment id ${COMPARTMENT_ID} where all { request.principal.type = 'workload', request.principal.namespace = '${NAMESPACE_NAME}', request.principal.service_account = '${SERVICE_ACCOUNT_NAME}', request.principal.cluster_id = '${KUBERNETES_CLUSTER_ID}' }",Allow any-user to manage buckets in compartment id ${COMPARTMENT_ID} where all { target.bucket.name = '${BUCKET_NAME}', request.permission = 'PAR_MANAGE', request.principal.type = 'workload', request.principal.namespace = '${NAMESPACE_NAME}', request.principal.service_account = '${SERVICE_ACCOUNT_NAME}', request.principal.cluster_id = '${KUBERNETES_CLUSTER_ID}' }",Allow any-user to manage objects in compartment id ${COMPARTMENT_ID} where all { target.bucket.name = '${BUCKET_NAME}', request.principal.type = 'workload', request.principal.namespace = '${NAMESPACE_NAME}', request.principal.service_account = '${SERVICE_ACCOUNT_NAME}', request.principal.cluster_id = '${KUBERNETES_CLUSTER_ID}' }"

In the statements above, replace the following variables with each service account and bucket combination:

• COMPARTMENT_ID - The Oracle Cloud ID (OCID) for the compartment that contains the Kubernetes cluster
• BUCKET_NAME - The Object Storage bucket name
• NAMESPACE_NAME - The Kubernetes namespace where the services are running
• SERVICE_ACCOUNT_NAME - The Kubernetes service account name used by the pod needing access
• KUBERNETES_CLUSTER_ID - The Oracle Cloud ID (OCID) for the Kubernetes cluster running the pods

Use the table below to create rules for each service account and bucket combination. Replace the bucket name with your actual bucket name.

Kubernetes Service Account	Policy
aisvc	kafka
datasourcesvc	kafka, datasourcesvc
filesvc-migrations	kafka, filesvc
filesvc	kafka, filesvc
itemsvc-migrations	kafka, filesvc
itemsvc-rdbms-migrations	kafka, filesvc
itemsvc-telemetry-worker	kafka, filesvc
itemsvc-worker	kafka, filesvc
itemsvc	kafka, filesvc
objectmodelsvc	kafka
passportsvc-migrations	kafka, filesvc
passportsvc	kafka, filesvc
platform-notificationsvc-api	kafka
platform-notificationsvc-worker	kafka
scriptmanager	kafka, scriptmanager
workflowsvc-api	kafka, workflowsvc
workflowsvc-backend	kafka, workflowsvc
workflowwkr-backend	kafka, workflowsvc

Set up Kubernetes network policy

The Script Worker pods execute JavaScript code created by application developers on the Platform. You should isolate the Script Worker pods from the rest of Kubernetes and from the private subnets. The Script Worker must, however, still be able to communicate with the Script Manager.

Refer to the example network policy below, which uses Calico as the implementation.

apiVersion: crd.projectcalico.org/v1kind: NetworkPolicymetadata:  name: scriptworker-policyspec:  selector: app.kubernetes.io/name == 'dtplatform-scriptworker'  types:    - Ingress    - Egress  order: 1000  egress:    - action: Allow      protocol: UDP      source: {}      destination:        ports:          - 53    - action: Allow      protocol: TCP      source: {}      destination:        ports:          - 53    - action: Allow      source: {}      destination:        selector: app.kubernetes.io/name == 'dtplatform-scriptmanager'    - action: Deny      source: {}      destination:        nets:          - 10.0.0.0/8          - 169.254.0.0/16          - 172.16.0.0/12          - 192.168.0.0/16

Add configurations to Prometheus

To ensure that the Platform’s Script Worker scales properly, add two configurations to Prometheus.

Do the following:

1. If using the kube-prometheus-stack with Custom Resource Definitions (CRDs), add a ServiceMonitor for the scriptmanager.

Example:

apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata:  name: scriptmanager-${var.env_name}  namespace: ${var.prometheus_namespace}  labels:    release: ${var.prometheus_release_label_value}spec:  selector:    matchLabels:      app.kubernetes.io/instance: scriptmanager-${var.env_name}  namespaceSelector:    matchNames:      - ${var.env_name}  endpoints:    - port: http-web      interval: 5s

2. Configure this data as an external metric rule in the Prometheus Adapter to combine the values into a single external metric, as shown below.

- seriesQuery: '{__name__=~"scriptmanager_job_queue_size",service!=""}'      metricsQuery: avg(<<.Series>>{<<.LabelMatchers>>}) by (service)      resources:        overrides: { namespace: {resource: "namespace"} }

Note: This step is required because the scriptworker horizontal pod autoscaler must use a value scraped from the scriptmanager pods. The values need to be combined into a single value so the HPA can consume it.

3. If using the prometheus-adapter Helm chart, add the value described above to your values.yaml file.

rules:  external:    - seriesQuery: '{__name__=~"scriptmanager_job_queue_size",service!=""}'      metricsQuery: avg(<<.Series>>{<<.LabelMatchers>>}) by (service)      resources:        overrides: { namespace: {resource: "namespace"} }

Create Kubernetes secrets

Next, create the Kubernetes secrets. Almost every service requires a Kubernetes secret in the same namespace, holding sensitive values for external systems or encryption keys, keypairs, and certificates needed by the Platform.

Note: Refer to Notes on Kubernetes secrets for useful information.

Create a new Kubernetes namespace for your installation (only one Digital Twin Platform per namespace), then create the secrets.

Some of these secrets are generated by AWS managed services and others are user-generated. In either case, we recommend generating them with a tool such as Terraform and storing them in AWS Secrets Manager for durable, versioned storage. The External Secrets Operator is a useful complement for this part of the installation.

Note: Many of the secrets are encryption keys. If lost, encrypted data is not recoverable.

Configure Helm Charts

The final step is to configure the Helm Charts. The main part of this process is preparing the values.yaml files that Helm uses for the installation.

Note: Configuration information for the Helm Charts can be found in Notes on the dtplatform Helm Chart, Notes on the dtconsole Helm Chart, Notes on the dtreference Helm Chart, and Notes on the dtplugins Helm Chart. Refer to those pages and configure your Helm Charts as required.

Install Order

A self-hosted installation involves four Helm Charts: dtplatform, dtconsole, dtreference, and dtplugins. The dtplatform chart must be installed first, into the same namespace as the others. The three web-application charts (dtconsole, dtreference, dtplugins) depend on dtplatform in one of two ways:

• When deployed in their default mode, each chart runs a migration job that copies its static files into the staticwebsvc PersistentVolumeClaim provisioned by dtplatform. The PVC must exist before the migration job runs, so dtplatform must be installed first.
• When deployed in standalone mode (each chart serving its own application from its own nginx deployment), the migration jobs are disabled, but the running applications still call the Platform's REST APIs and Identity Service. dtplatform must therefore be installed and reachable before users can use any of the web applications.

In both modes, the config.global.url.app, config.global.url.id, and config.global.url.api values in each web-application chart must match those configured in dtplatform.