Bash script for Instance Pool Lifecycle Management

This script will create a new image, a new instance configuration based on the new image and update the current instance pool with the new instance configuration.

After that, we can run recycle_instances.sh to recycle all the instances. See my previous post for details.

instance-config-ords-as-template.json

{
    "instanceType": "compute",
    "launchDetails": {
      "compartmentId": "{{compartmentId}}",
      "createVnicDetails": {
        "assignPublicIp": false
      },
      "metadata": {
        "ssh_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuPoZBbXjC/5ojt+ECoIj5KGmLHWPPreWcjkp/1metOBgRv8f6W7w615+kRcrrdtyB5Tk6MzIs6CmF8RZ1BkkSankhHG62aKkqXn7T9VDenvEHaJpJqQRkhkYzJKyYqL+04O942gSgv8Kpw1IpFWvznfelf30xaxQzcLa0tMjYvOmqTmeAndEM3E1ZVMcEq3r3OlTTjCyfBPsdRPV2hFClvQziueRrUF61lhLotPUkCKxc6Iie+OpqW5hhU8vypFT0MAB6hoTH7EO7BGmQWInQlO3Pt4m7q9dNSee731TzRceDFa5cC/uigeAFgjEY8lwM5CFrcgMW3n3B3BpfumQv PC@PCLAPPY"
      },
        "shape": "VM.Standard.E3.Flex",
        "shapeConfig": {
          "memoryInGBs": 16.0,
          "ocpus": 1.0
        },
        "sourceDetails":   {
          "bootVolumeSizeInGBs": null,
          "imageId": "{{imageId}}",
          "sourceType": "image"
        }
    }
}

update_instance_pool_image_part2_template.txt

#!/usr/bin/bash

COMPARTMENT_OCID="{{compartmentId}}"
INSTANCE_POOL_OCID="{{instancepoolId}}"
INSTANCE_CONFIG_JSON="{{INSTANCE_CONFIG_JSON}}"
INSTANCE_CONFIG_NAME="{{INSTANCE_CONFIG_NAME}}"

INSTANCE_CONFIG_OCID=`oci compute-management instance-configuration create --compartment-id $COMPARTMENT_OCID --instance-details file://$INSTANCE_CONFIG_JSON --display-name $INSTANCE_CONFIG_NAME | jq -r '.data.id'`
sleep 30
oci compute-management instance-pool update --wait-for-state RUNNING --instance-pool-id $INSTANCE_POOL_OCID --instance-configuration-id $INSTANCE_CONFIG_OCID

update_instance_pool_image_part1.sh

#!/usr/bin/bash

COMPARTMENT_OCID="ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a"
INSTANCE_POOL_OCID="ocid1.instancepool.oc1.ca-toronto-1.aaaaaaaaswx6wld7z77u32shwivedvgn5usofurtjzfd3kdnfopbi56wqlfa"
INSTANCE_OCID="ocid1.instance.oc1.ca-toronto-1.an2g6ljrmpjzp2icj3zh7m5ndqp365ahhxv5j2b2u4t7omixuugdqwojymsq"

DATETIME=`date +"%Y-%m%d-%H%M"`
IMAGE_NAME="ords-as_"$DATETIME
INSTANCE_CONFIG_JSON="instance-config-ords-as_"$DATETIME".json"
INSTANCE_CONFIG_NAME="instance-config-ords-as_"$DATETIME
UPDATE_INSTANCE_POOL_SCRIPT="update_instance_pool_image_part2.sh"

sed "s/{{INSTANCE_CONFIG_JSON}}/$INSTANCE_CONFIG_JSON/g;s/{{INSTANCE_CONFIG_NAME}}/$INSTANCE_CONFIG_NAME/g;s/{{compartmentId}}/$COMPARTMENT_OCID/g;s/{{instancepoolId}}/$INSTANCE_POOL_OCID/g" update_instance_pool_image_part2_template.txt > $UPDATE_INSTANCE_POOL_SCRIPT
chmod 755 $UPDATE_INSTANCE_POOL_SCRIPT

IMAGE_OCID=`oci compute image create --compartment-id $COMPARTMENT_OCID --instance-id $INSTANCE_OCID --display-name $IMAGE_NAME | jq -r '.data.id'`

sed "s/{{imageId}}/$IMAGE_OCID/g;s/{{compartmentId}}/$COMPARTMENT_OCID/g" instance-config-ords-as-template.json > $INSTANCE_CONFIG_JSON

echo "Custom Image $IMAGE_NAME created. This instance will be taken offline for several minutes during the imaging process. Run update_pool_instance_image_part2.sh when instance come back online."

update_instance_pool_image_part2.sh

#!/usr/bin/bash

COMPARTMENT_OCID="ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a"
INSTANCE_POOL_OCID="ocid1.instancepool.oc1.ca-toronto-1.aaaaaaaaswx6wld7z77u32shwivedvgn5usofurtjzfd3kdnfopbi56wqlfa"
INSTANCE_CONFIG_JSON="instance-config-ords-as_2021-0308-0731.json"
INSTANCE_CONFIG_NAME="instance-config-ords-as_2021-0308-0731"

INSTANCE_CONFIG_OCID=`oci compute-management instance-configuration create  --compartment-id $COMPARTMENT_OCID --instance-details file://$INSTANCE_CONFIG_JSON --display-name $INSTANCE_CONFIG_NAME | jq -r '.data.id'`
sleep 30
oci compute-management instance-pool update --wait-for-state RUNNING --instance-pool-id $INSTANCE_POOL_OCID --instance-configuration-id $INSTANCE_CONFIG_OCID

Bash script to recycle all running instances in an instance pool

We have an instance pool, auto scaling with minimum 2 instances. After we update the instance configuration for the instance pool, we need to bring all running instances up to date.

The process is terminating one instance at a time, wait till the load balancer get back to "OK" and repeat. Eventually, we will terminated all instances with the old instance configuration with zero downtime.

recycle_instances.sh

#!/usr/bin/bash

COMPARTMENT_OCID="ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a"
LOAD_BALANCER_OCID="ocid1.loadbalancer.oc1.ca-toronto-1.aaaaaaaaxgcpd4izvacefithsabg3l4dhvl7y6gt2mcgitdlwt3ez45vzp5q"
INSTANCE_POOL_OCID="ocid1.instancepool.oc1.ca-toronto-1.aaaaaaaaswx6wld7z77u32shwivedvgn5usofurtjzfd3kdnfopbi56wqlfa"


# Wait till Load Balancer status is OK
lb_status_check () {
while [ "$LB_STATUS" != "OK" ]
do
  sleep 1m
  LB_STATUS=`oci lb load-balancer-health get --load-balancer-id $LOAD_BALANCER_OCID | jq -r '.data.status'`
  date
  echo "Load Balance Status: $LB_STATUS"
done
}

# Ensure Load Balancer is in OK status before we start
LB_STATUS=`oci lb load-balancer-health get --load-balancer-id $LOAD_BALANCER_OCID | jq -r '.data.status'`
lb_status_check

# Terminate all running instances in this instance pool
for INSTANCE_OCID in $(oci compute instance list --compartment-id $COMPARTMENT_OCID --lifecycle-state RUNNING | jq -r '.data[]| select(."freeform-tags"."oci:compute:instancepool" | contains("'$INSTANCE_POOL_OCID'"))? | .id') 
do
  date
  echo "Terminate instance OCID: ${INSTANCE_OCID}"
  oci compute instance terminate --force --instance-id ${INSTANCE_OCID}
  LB_STATUS="UNKNOWN"
  sleep 5m
  lb_status_check
done

date
echo "Recycle all running instances completed"

Instance Pool Lifecycle Management

List all custom images

oci compute image list --all --compartment-id ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a | jq '.data[]|select(."compartment-id" | contains("ocid"))? | ."display-name"'

Create a custom image based on the master instance

oci compute image create --compartment-id ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a --instance-id ocid1.instance.oc1.ca-toronto-1.an2g6ljrmpjzp2icj3zh7m5ndqp365ahhxv5j2b2u4t7omixuugdqwojymsq --display-name ords-as-v3

Create a new instance configuration

oci compute-management instance-configuration create  --compartment-id ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a --instance-details file://instance-config-ords-as-v4.json --display-name instance-config-ords-as-v4

Update the instance pool with the new instance configuration

oci compute-management instance-pool update --instance-pool-id ocid1.instancepool.oc1.ca-toronto-1.aaaaaaaaswx6wld7z77u32shwivedvgn5usofurtjzfd3kdnfopbi56wqlfa --instance-configuration-id ocid1.instanceconfiguration.oc1.ca-toronto-1.aaaaaaaasggwcn4pa3vejt5t3wthrze46fctsfw7hmnmbetxj4qav6mykrrq

List all running instances using the old instance configuration

oci compute instance list --all --compartment-id ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a --lifecycle-state RUNNING | jq '.data[]| select(."freeform-tags"."oci:compute:instanceconfiguration" | contains("ocid"))? | {"display-name": ."display-name", id: .id, ic: ."freeform-tags"."oci:compute:instanceconfiguration"}'

Teminate all running instances using the old instance configuration

oci compute instance terminate --instance-id ocid1.instance.oc1.ca-toronto-1.an2g6ljrmpjzp2icr34kwlkjldkmohstbs7bfwgtm3e5k2plcc2lox6ow5qa  --force

oci lb load-balancer-health get --load-balancer-id ocid1.loadbalancer.oc1.ca-toronto-1.aaaaaaaaxgcpd4izvacefithsabg3l4dhvl7y6gt2mcgitdlwt3ez45vzp5q

oci compute instance terminate --instance-id ocid1.instance.oc1.ca-toronto-1.an2g6ljrmpjzp2icfnl33iqrbcy7rbpn7ixg46s6dgb4tyrbdqtm2cfiv5iq --force

oci lb load-balancer-health get --load-balancer-id ocid1.loadbalancer.oc1.ca-toronto-1.aaaaaaaaxgcpd4izvacefithsabg3l4dhvl7y6gt2mcgitdlwt3ez45vzp5q

Autoscaling a Load Balanced Compute Instance

Prepare the template compute instance

Create /usr/local/bin/warmup. Instance specific setting goes here.

cat << EOF > /usr/local/bin/warmup
#!/bin/sh
# warmup: sync displayName data into our index and mark as healthy for load balancer
DISPLAY_NAME=`curl -H "Authorization: Bearer Oracle" -Ls http://169.254.169.254/opc/v2/instance/displayName`
cat << EOF > /opt/oracle/ords/config/ords/standalone/doc_root/index.html
<h3>$DISPLAY_NAME</h3>
<img style="max-width: 100%; height: auto; width: auto;" src="oci.jpg">
EOF

Create /etc/systemd/system/warmup.service

cat << EOF > /etc/systemd/system/warmup.service
[Unit]
After=network.target
  
[Service]
ExecStart=/usr/local/bin/warmup
  
[Install]
WantedBy=default.target
EOF

Setup warmup as service

systemctl start warmup
systemctl enable warmup
systemctl status warmup

Reboot the instance and ensure everything comes up. If okay, then Create Custom Image from this instance, ords-as-v1

Create instance-config-ords-as-v1.json, fix compartmentId, ssh_authorized_keys, imageId

cat << EOF > instance-config-ords-as-v1.json
{
    "instanceType": "compute",
    "launchDetails": {
      "compartmentId": "ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a",
      "createVnicDetails": {
      },
      "metadata": {
        "ssh_authorized_keys": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDuPoZBbXjC/5ojt+ECoIj5KGmLHWPPreWcjkp/1metOBgRv8f6W7w615+kRcrrdtyB5Tk6MzIs6CmF8RZ1BkkSankhHG62aKkqXn7T9VDenvEHaJpJqQRkhkYzJKyYqL+04O942gSgv8Kpw1IpFWvznfelf30xaxQzcLa0tMjYvOmqTmeAndEM3E1ZVMcEq3r3OlTTjCyfBPsdRPV2hFClvQziueRrUF61lhLotPUkCKxc6Iie+OpqW5hhU8vypFT0MAB6hoTH7EO7BGmQWInQlO3Pt4m7q9dNSee731TzRceDFa5cC/uigeAFgjEY8lwM5CFrcgMW3n3B3BpfumQv PC@PCLAPPY"
      },
        "shape": "VM.Standard.E3.Flex",
        "shapeConfig": {
          "memoryInGBs": 16.0,
          "ocpus": 1.0
        },
        "sourceDetails":   {
          "bootVolumeSizeInGBs": null,
          "imageId": "ocid1.image.oc1.ca-toronto-1.aaaaaaaabpzh6hlkg7jrdmm4eghhzhb3pwbk7pevvkn2cxqm3pk7s46fjn5q",
          "sourceType": "image"
        }
    }
}
EOF

Instance Pool Setup

Create instance configuration via OCI

oci compute-management instance-configuration create --instance-details file://instance-config-ords-as-v1.json --compartment-id ocid1.compartment.oc1..aaaaaaaarocn3npultgruh5iwghhvor6s3kairokq4mil5bp52va6qkk7x6a --display-name instance-config-ords-as-v1

# Get instance configuration details sample in JSON
oci compute-management instance-configuration create --generate-param-json-input instance-details

# Get instance configuration details
oci compute-management instance-configuration get --instance-configuration-id ocid1.instanceconfiguration.oc1.ca-toronto-1.aaaaaaaaxljbut4sn5hwqybubuarqr2lb5omshiog27bta4dcyvtydldetaa

From Instance Configurations, create Instance Pool, assign Load Balancer and create Autoscaling Configuration

Stress Test

We can run stress-ng on one instance, bring the CPU to 100% and wait to see auto scaling bring up a new instance within the Instance Pool.