Let's Encrypt Automatic Certificate Renewal in OCI Load Balancer

First we need to setup an acme instance to run the SSL certificate renewal against Let's encrypt nightly. To do that, the acme instance must be behind the load balancer with the SSL certificate associated with it. On top of that, we only want Let's Encrypt traffic reaching this instance. To achive this, we use Path Route Sets.

Add Backend Set

Add Backends

Add Path Route Sets

Assign Path Route Sets to the Load Balancer SSL listener

At this point, only the Let's Encrypt traffic will be routed to the acme instance. Normal traffic will not be affected.

acme.sh is probably the easiest & smartest shell script to automatically issue & renew the free certificates from Let's Encrypt. On the acme instance, we will do the followings

Install acme.sh

curl https://get.acme.sh | sh -s email=pchiu@leavemealone.com

Issue SSL Certificate

~/.acme.sh/acme.sh --issue -d pws.leavemealone.com -w /opt/oracle/ords/config/ords/standalone/doc_root

Install SSL Certificate

~/.acme.sh/acme.sh --install-cert -d pws.leavemealone.com --key-file /opt/oracle/ords/config/ords/standalone/leavemealone.com.key --cert-file /opt/oracle/ords/config/ords/standalone/leavemealone.com.pem --ca-file /opt/oracle/ords/config/ords/standalone/ca.cer --reloadcmd "/home/oracle/renew_lb_certs.sh"

Create and Install pkcs8 private key

~/.acme.sh/acme.sh --renew -d pws.leavemealone.com --to-pkcs8
cp ~/.acme.sh/pws.leavemealone.com/pws.leavemealone.com.pkcs8 /opt/oracle/ords/config/ords/standalone/
chmod 600 /opt/oracle/ords/config/ords/standalone/pws.leavemealone.com.pkcs8

Set notifications

export MAIL_TO="pchiu@leavemealone.com"
export MAIL_FROM="no-reply@leavemealone.com"
acme.sh --set-notify --notify-hook mail

At this point, we have setup acme.sh to renewal the cetificate automatically for us and we will put the new certifcate in /opt/oracle/ords/config/ords/standalone

The following bash script will take the new certificate in /opt/oracle/ords/config/ords/standalone and add it to the load balancer and update the listener to use it.

renew_lb_certs.sh

#!/usr/bin/bash

LB_OCID="ocid1.loadbalancer.oc1.ca-toronto-1.aaaaaaaaxgcpd4izvacefithsabg3l4dhvl7y6gt2mcgitdlwt3ez45vzp5q"
LISTENER_NAME="listener_lb_323"
BACKEND_SET_NAME="bs_lb_2021-0122-2326"
CERT_PATH="/opt/oracle/ords/config/ords/standalone"

CERT_NAME="pws.leavemealone.com_"`date +"%Y-%m%d-%H%M"`

~/.acme.sh/acme.sh --renew -d pws.leavemealone.com --to-pkcs8
cp ~/.acme.sh/pws.leavemealone.com/pws.leavemealone.com.pkcs8 /opt/oracle/ords/config/ords/standalone/
chmod 600 /opt/oracle/ords/config/ords/standalone/pws.leavemealone.com.pkcs8

oci lb certificate create --load-balancer-id $LB_OCID --wait-for-state SUCCEEDED --certificate-name $CERT_NAME --ca-certificate-file $CERT_PATH/ca.cer --private-key-file $CERT_PATH/pws.leavemealone.com.key --public-certificate-file $CERT_PATH/pws.leavemealone.com.pem
oci lb listener update --force --wait-for-state SUCCEEDED --listener-name $LISTENER_NAME --default-backend-set-name $BACKEND_SET_NAME --port 443 --protocol HTTP --load-balancer-id $LB_OCID --ssl-certificate-name $CERT_NAME --hostname-names \[\"pws.leavemealone.com\"\] --routing-policy-name acme --rule-set-names \[\"ADD_HSTS\"\] --cipher-suite-name oci-default-http2-ssl-cipher-suite-v1

#oci lb certificate list --all --load-balancer-id $LB_OCID

We still need to manually update the certificate between load balancer and instances in private subnet

Update Certificate in Backend Set

Secure OCI Load Balancer Setup

Encryption of data in Transit

Assumption: Load Balancer is listening on Port 80,443 on the public side and ORDS standalone is listening on Port 8080,8443 on the private side.

End to End SSL

Use SSL to communicate with ORDS standalone

Ensure Health Check is using Port 8443

Add HSTS response header

Rule Set for adding HSTS response header

Listener on Port 443 with the rule set

Setup another listener on Port 80, rediect traffic to 443

Rule Set for Redirection

Listener on Port 80 with the rule set

Run SSL Labs Test

Autoscaling a Load Balanced Compute Instance

Prepare the template compute instance

Create /usr/local/bin/warmup. Instance specific setting goes here.

cat << EOF > /usr/local/bin/warmup
#!/bin/sh
# warmup: sync displayName data into our index and mark as healthy for load balancer
DISPLAY_NAME=`curl -H "Authorization: Bearer Oracle" -Ls http://169.254.169.254/opc/v2/instance/displayName`
cat << EOF > /opt/oracle/ords/config/ords/standalone/doc_root/index.html
<h3>$DISPLAY_NAME</h3>
<img style="max-width: 100%; height: auto; width: auto;" src="oci.jpg">
EOF

Create /etc/systemd/system/warmup.service

cat << EOF > /etc/systemd/system/warmup.service
[Unit]
After=network.target
  
[Service]
ExecStart=/usr/local/bin/warmup
  
[Install]
WantedBy=default.target
EOF

Setup warmup as service

systemctl start warmup
systemctl enable warmup
systemctl status warmup

Reboot the instance and ensure everything comes up. If okay, then Create Custom Image from this instance, ords-as-v1

Create instance-config-ords-as-v1.json, fix compartmentId, ssh_authorized_keys, imageId

cat << EOF > instance-config-ords-as-v1.json
{
    "instanceType": "compute",
    "launchDetails": {
      "compartmentId": "ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a",
      "createVnicDetails": {
      },
      "metadata": {
        "ssh_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuPoZBbXjC/5ojt+ECoIj5KGmLHWPPreWcjkp/1metOBgRv8f6W7w615+kRcrrdtyB5Tk6MzIs6CmF8RZ1BkkSankhHG62aKkqXn7T9VDenvEHaJpJqQRkhkYzJKyYqL+04O942gSgv8Kpw1IpFWvznfelf30xaxQzcLa0tMjYvOmqTmeAndEM3E1ZVMcEq3r3OlTTjCyfBPsdRPV2hFClvQziueRrUF61lhLotPUkCKxc6Iie+OpqW5hhU8vypFT0MAB6hoTH7EO7BGmQWInQlO3Pt4m7q9dNSee731TzRceDFa5cC/uigeAFgjEY8lwM5CFrcgMW3n3B3BpfumQv PC@PCLAPPY"
      },
        "shape": "VM.Standard.E3.Flex",
        "shapeConfig": {
          "memoryInGBs": 16.0,
          "ocpus": 1.0
        },
        "sourceDetails":   {
          "bootVolumeSizeInGBs": null,
          "imageId": "ocid1.image.oc1.ca-toronto-1.aaaaaaaabpzh6hlkg7jrdmm4eghhzhb3pwbk7pevvkn2cxqm3pk7s46fjn5q",
          "sourceType": "image"
        }
    }
}
EOF

Instance Pool Setup

Create instance configuration via OCI

oci compute-management instance-configuration create --instance-details file://instance-config-ords-as-v1.json --compartment-id ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a --display-name instance-config-ords-as-v1

# Get instance configuration details sample in JSON
oci compute-management instance-configuration create --generate-param-json-input instance-details

# Get instance configuration details
oci compute-management instance-configuration get --instance-configuration-id ocid1.instanceconfiguration.oc1.ca-toronto-1.aaaaaaaaxljbut4sn5hwqybubuarqr2lb5omshiog27bta4dcyvtydldetaa

From Instance Configurations, create Instance Pool, assign Load Balancer and create Autoscaling Configuration

Stress Test

We can run stress-ng on one instance, bring the CPU to 100% and wait to see auto scaling bring up a new instance within the Instance Pool.