We are running Oracle forms and reports server since 2018 and all of a sudden the OHS service crashed on one server and then next one and next one and next one... Not good.
After some debugging execises, the problem is due to
- Self-signed cwallet.sso expired and
- Weblogic nodemanger default keystore SSL certficate expired
Let's start from cwallet.sso
File location: E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\config\fmwconfig\components\OHS\instances\ohs1\keystores\default\cwallet.sso
Let's check the expired date
orapki wallet display -wallet cwallet.sso
Oracle PKI Tool Release 19.0.0.0.0 - Production
Version 19.4.0.0.0
Copyright (c) 2004, 2021, Oracle and/or its affiliates. All rights reserved.
Requested Certificates:
User Certificates:
Subject: CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY
Trusted Certificates:
Subject: CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY
orapki wallet export -wallet cwallet.sso -dn "CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY" -cert cwallet.cer
Goto Certificate Decoder and paste the cwallet.cer content into there
The current cwallet.sso is expired. We need to create a new one
orapki wallet create -wallet ./ -pwd WalletPasswd123 -auto_login
orapki wallet add -wallet ./ -pwd WalletPasswd123 -dn "CN=localhost,OU=FOR TESTING ONLY,O=FOR TESTING ONLY" -keysize 1024 -self_signed -validity 3650
Replace E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\config\fmwconfig\components\OHS\instances\ohs1\keystores\default\cwallet.sso with the new one.
Next, let's fix nodemanager SSL certificate
cd E:\oracle\penfax_app\Oracle_Home\oracle_common\common\bin\
E:\oracle\penfax_app\Oracle_Home\oracle_common\common\bin>wlst.cmd
Initializing WebLogic Scripting Tool (WLST) ...
Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline> nmConnect(domainName='base_domain', username='weblogic', password='leave.me.alone')
Connecting to Node Manager ...
<Jan 17, 2023 11:23:07 AM EST> <Info> <Security> <BEA-090905> <Disabling the CryptoJ JCE Provider self-integrity check for better startup performance. To enable this check, specify
-Dweblogic.security.allowCryptoJDefaultJCEVerification=true.>
<Jan 17, 2023 11:23:07 AM EST> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to HMACDRBG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>
<Jan 17, 2023 11:23:07 AM EST> <Info> <Security> <BEA-090909> <Using the configured custom SSL Hostname Verifier implementation: weblogic.security.utils.SSLWLSHostnameVerifier$Null
HostnameVerifier.>
Traceback (innermost last):
File "<console>", line 1, in ?
File "<iostream>", line 111, in nmConnect
File "<iostream>", line 553, in raiseWLSTException
WLSTException: Error occurred while performing nmConnect : Cannot connect to Node Manager. : General SSLEngine problem
Use dumpStack() to view the full stacktrace :
wls:/offline>
This doesn't tell much.
set WLST_PROPERTIES=-Djavax.net.debug=all -Dssl.debug=true
E:\oracle\penfax_app\Oracle_Home\oracle_common\common\bin>wlst.cmd
Initializing WebLogic Scripting Tool (WLST) ...
Jython scans all the jar files it can find at first startup. Depending on the system, this process may take a few minutes to complete, and WLST may not return a prompt right away.
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
wls:/offline> nmConnect(domainName='base_domain', username='weblogic', password='leave.me.alone')
...
***
main, fatal error: 46: General SSLEngine problem
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed
%% Invalidated: [Session-1, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256]
main, SEND TLSv1.2 ALERT: fatal, description = certificate_unknown
main, WRITE: TLSv1.2 Alert, length = 2
main, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: General SSLEngine problem
main, called closeOutbound()
main, closeOutboundInternal()
[Raw write]: length = 7
0000: 15 03 03 00 02 02 2E .......
wls:/offline>
Check the output close to the bottom, it tells us there is a "timestamp check failed" problem.
Let's generate new Custom Identity and Custom Trust jks file
E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -genkey -alias base_domain -keyalg RSA -keysize 2048 -dname "CN=base_domain, OU=Digital, O=Leave Me Alone, L=Toronto, ST=Ontario, C=CA" -keypass WalletPasswd123 -keystore identity.jks -storepass WalletPasswd123
E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -selfcert -v -alias base_domain -keypass WalletPasswd123 -keystore identity.jks -storepass WalletPasswd123 -storetype jks -validity 3650
E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -export -v -alias base_domain -file rootCA.der -keystore identity.jks -storepass WalletPasswd123
E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -import -v -trustcacerts -alias base_domain -file rootCA.der -keystore trust.jks -storepass WalletPasswd123
Copy identity.jks and trust.jks to E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\security
In Weblogic Admin Console
Custom Identity Keystore: E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\security\identity.jks
Custom Trust Keystore: E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\security\trust.jks
Edit E:\oracle\penfax_app\Oracle_Home\user_projects\domains\base_domain\nodemanager\nodemanager.properties, add the following to the bottom.
StartScriptEnabled=true
KeyStores=CustomIdentityAndCustomTrust
CustomIdentityKeyStoreFileName=E\:\\oracle\\penfax_app\\Oracle_Home\\user_projects\\domains\\base_domain\\security\\identity.jks
CustomIdentityAlias=base_domain
CustomIdentityPrivateKeyPassPhrase=WalletPasswd123
CustomTrustKeyStoreFileName=E\:\\oracle\\penfax_app\\Oracle_Home\\user_projects\\domains\\base_domain\\security\\trust.jks
We need to update the java Trust certificate as well
E:\oracle\penfax_app\Oracle_Home\oracle_common\jdk\bin\keytool.exe -import -alias base_domain -trustcacerts -file rootCA.der -keystore E:\oracle\temp\java\jdk1.8.0_112\jre\lib\security\cacerts -storepass changeit
Finally, everything is done. Reboot the server and OHS should start automatically.
