Oracle forms and reports OHS service stop working

We are running Oracle forms and reports server since 2018 and all of a sudden the OHS service crashed on one server and then next one and next one and next one... Not good.

After some debugging execises, the problem is due to

1. Self-signed cwallet.sso expired and
2. Weblogic nodemanger default keystore SSL certficate expired

Let's start from cwallet.sso

File location: E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\config\fmwconfig\components\OHS\instances\ohs1\keystores\default\cwallet.sso
Let's check the expired date

orapki wallet display -wallet cwallet.sso
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:
User Certificates:
Subject:        CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY
Trusted Certificates:
Subject:        CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY
 
orapki wallet export -wallet cwallet.sso -dn "CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY" -cert cwallet.cer

Goto Certificate Decoder and paste the cwallet.cer content into there

The current cwallet.sso is expired. We need to create a new one

orapki wallet create -wallet ./ -pwd WalletPasswd123 -auto_login
orapki wallet add -wallet ./ -pwd WalletPasswd123 -dn "CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY" -keysize 1024 -self_signed -validity 3650

Replace E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\config\fmwconfig\components\OHS\instances\ohs1\keystores\default\cwallet.sso with the new one.

Next, let's fix nodemanager SSL certificate

cd E:\oracle\penfax_app\Oracle_Home\oracle_common\common\bin\

E:\oracle\penfax_app\Oracle_Home\oracle_common\common\bin>wlst.cmd

Initializing WebLogic Scripting Tool (WLST) ...

Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline> nmConnect(domainName='base_domain', username='weblogic', password='leave.me.alone')
Connecting to Node Manager ...
<Jan 17, 2023 11:23:07 AM EST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify
 -Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>
<Jan 17, 2023 11:23:07 AM EST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to HMACDRBG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>
<Jan 17, 2023 11:23:07 AM EST> <Info> <Security> <BEA-090909> <Using the configured custom SSL Hostname Verifier implementation: weblogic.security.utils.SSLWLSHostnameVerifier$Null
HostnameVerifier.>
Traceback (innermost last):
  File "<console>", line 1, in ?
  File "<iostream>", line 111, in nmConnect
  File "<iostream>", line 553, in raiseWLSTException
WLSTException: Error occurred while performing nmConnect : Cannot connect to Node Manager. : General SSLEngine problem
Use dumpStack() to view the full stacktrace :
wls:/offline>

This doesn't tell much.

set WLST_PROPERTIES=-Djavax.net.debug=all -Dssl.debug=true
E:\oracle\penfax_app\Oracle_Home\oracle_common\common\bin>wlst.cmd

Initializing WebLogic Scripting Tool (WLST) ...

Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.

Welcome to WebLogic Server Administration Scripting Shell

Type help() for help on available commands

wls:/offline> nmConnect(domainName='base_domain', username='weblogic', password='leave.me.alone')


...


***
main, fatal error: 46: General SSLEngine problem
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed
%% Invalidated:  [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256]
main, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
main, WRITE: TLSv1.2 Alert, length = 2
main, fatal: engine already closed.  Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
main, called closeOutbound()
main, closeOutboundInternal()
[Raw write]: length = 7
0000: 15 03 03 00 02 02 2E                               .......
wls:/offline>

Check the output close to the bottom, it tells us there is a "timestamp check failed" problem.

Let's generate new Custom Identity and Custom Trust jks file

E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -genkey -alias base_domain -keyalg RSA -keysize 2048 -dname "CN=base_domain, OU=Digital, O=Leave Me Alone, L=Toronto, ST=Ontario, C=CA" -keypass WalletPasswd123 -keystore identity.jks -storepass WalletPasswd123
E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -selfcert -v -alias base_domain -keypass WalletPasswd123 -keystore identity.jks -storepass WalletPasswd123 -storetype jks -validity 3650
E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -export -v -alias base_domain -file rootCA.der -keystore identity.jks -storepass WalletPasswd123
E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -import -v -trustcacerts -alias base_domain -file rootCA.der -keystore trust.jks -storepass WalletPasswd123

Copy identity.jks and trust.jks to E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\security

In Weblogic Admin Console

Custom Identity Keystore: E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\security\identity.jks
Custom Trust Keystore: E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\security\trust.jks

Edit E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\nodemanager\nodemanager.properties, add the following to the bottom.

StartScriptEnabled=true
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreFileName=E\:\\oracle\\penfax_app\\Oracle_Home\\user_projects\\domains\\base_domain\\security\\identity.jks
CustomIdentityAlias=base_domain
CustomIdentityPrivateKeyPassPhrase=WalletPasswd123
CustomTrustKeyStoreFileName=E\:\\oracle\\penfax_app\\Oracle_Home\\user_projects\\domains\\base_domain\\security\\trust.jks

We need to update the java Trust certificate as well

E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -import -alias base_domain -trustcacerts -file rootCA.der -keystore E:\oracle\temp\java\jdk1.8.0_112\jre\lib\security\cacerts -storepass changeit

Finally, everything is done. Reboot the server and OHS should start automatically.