Archive for the ‘Patching’ Category

Good afternoon, Oracle Database Enthusiasts! Nothing gets me in the holiday spirit like quarterly patching… Not!! When it comes to Oracle Patching, it will fail for just about any reason it feels like. Here’s another fun one. If you haven’t experienced it yet, read on…

Still here? So…

Two Fridays ago, when patching Grid on Node 1, the patch apply hung for about an hour and then quit. The log sat there quiet for the entire hour. Once the patch finally failed, it spit out all of the errors it was experiencing.

Root cause was this…

Unable to lock Central Inventory.  OPatch will attempt to re-lock.Do you want to proceed? [y|n]

Y (auto-answered by -silent)

User Responded with: Y

OPatch will sleep for few seconds, before re-trying to get the lock…

OUI-10022:The target area /u01/app/oraInventory cannot be used because it is in an invalid state.

OPatchSession cannot load inventory for the given Oracle Home /u01/app/11.2.0.4/grid. Possible causes are:

No read or write permission to ORACLE_HOME/.patch_storage

Central Inventory is locked by another OUI instance

No read permission to Central Inventory

The lock file exists in ORACLE_HOME/.patch_storage

The Oracle Home does not exist in Central Inventory

 

Had the log actually spit out the warning during the patch attempt, I might have been able to fix it on the fly. No such luck. Instead it spit 100 copies of the above error AFTER the patch failed. LOVELY. I ran out of time and had to rollback. 

When there is a lock in /u01/app/oraInventory, it creates a locks directory and a file underneath it like /u01/app/oraInventory/locks/inventory.lock. The solution was to remove the file and the locks directory. I tested this by creating this directory and file manually in our sandbox. I Love Oracle Sandboxes!! If you don’t have one, you really should get one! As I attempted to patch grid on node 1 of our sandbox, it created the exact same scenario. It shut down CRS and just hung for an hour, trying over and over again to create a lock on oraInventory. Once it errored out, I removed the directory and file. Patching of Grid took then less than ten minutes. I am now adding this step to my prerequisites:

Check the /u01/app/oraInventory/locks directory and subsequent files.  If they exist, remove them.

I hope this helps those of you who have not yet applied October’s CPU. Or, if anything, it applies a little comic relief for all you sick techies like me who laugh at other’s misfortunes.

Thanks for reading!

Jason

Update!! I attempted the patch again tonight, armed with my former findings. I patched the Grid Home and Oracle Home and seven Catbundle scripts in less than 2 hours!! Much better this time around!!

 

 

Advertisements

OUI-67073:UtilSession failed: Prerequisite check “CheckActiveFilesAndExecutables” failed.

Good afternoon/evening/morning to Oracle Enthusiasts Across our Great Globe!

One of my biggest pet peeves in network television is when a season ends with too many questions and won’t pick up again until next year. For those of you who read my post earlier today, I am sparing you the agony of waiting all weekend to see how my patching turned out!

Earlier today, I published an article on Upgrading to the Latest Patch (OPatch) Utility. This was the primary error I received while trying to apply the bugfix patch for Class of Secure Transport (COST). Since the post ended in an error, I needed to update you with the solution. In truth, I almost backed out and turned the databases back over to the DEV and QA users. However, since these users reside in Europe, they won’t be on shift for several hours. So I kept digging. As usual, the solution was very close.

The logs stated…

OUI-67073:UtilSession failed: Prerequisite check “CheckActiveFilesAndExecutables” failed.

OPatch failed with error code 73

There are two steps to Applying COST to Restrict Instance Registration(of course you need a subscription to get here) The steps are abbreviated on purpose. I am not Oracle Support and patching should be done by strictly following the procedures documented on that page. However, as life turns out, we don’t always get the results that the documentation and the Read Me page presents.

  1. Obtain and Apply bugfix patch 12880299.
    • Download and unzip the patch
    • Upgrade to the latest version of OPatch for your database. Mine was 11.2.0.1. Yes it is an old version.
    • Apply the patch.
      • In the Read Me file, the instructions are to stop all of the Oracle database instances. They are all single instance databases so I shut all of them down.
      • Next, I was to apply the patch. The log stated the above errors.
      • I took to Google and found an interesting article. by an anonymous blogger who calls himself acehints. Thank you, Mr. Acehints. In his situation, he found Enterprise Manager still running and subsequently killed the process. That got me to thinking. This patch is to restrict the listener. Wouldn’t it make sense to stop the listener before applying the patch?
      • Even though the Read Me omitted this step, I stopped the listener.
        • $ lsnrctl stop listener
      • $ opatch apply
      • Start the database instances.
    • Hooray! OPatch was applied successfully!!
  2. Add the COST TCP protocol restriction to the listener.ora to restrict usage to TCP Protocol only.
    • SECURE_REGISTER_LISTENER = (TCP)
  3. Start the listener.
  4. Connect to the databases using the listener instead of straight up SQL Plus.
    • $ sqlplus / as sysdba@db_name;
  5. Inform Database users to start their applications.

All appears to be well!

Thanks for reading!

Jason

Happy Friday, Oracle bloggers and blog followers alike!

Today’s installment comes on the heels of testing the bugfix for BUG:12880299. I will refer to Doc ID 1453883.1 on Oracle Support and maybe a couple other Doc ID’s. I recommend you have these documents open as you read. I ran into a couple issues while following the document. The purpose of this article is to show how I solved the issues I ran into.

One of the first issues I noticed was covered in the Read Me for the patch. Oracle recommends you have the latest version of OPatch installed in your ORACLE HOME. You verify this by running the lsinventory command as oracle in the ORACLE HOME.

$ opatch lsinventory

Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oracle/product/oraInventory
from : /var/opt/oracle/oraInst.loc
OPatch version : 11.1.0.6.6
OUI version : 11.2.0.1.0

I looked past this step, stopped the database instances, and attempted to apply the patch

$ opatch apply

ApplySession failed during prerequisite checks: Prerequisite check “CheckActiveFilesAndExecutables” failed.
System intact, OPatch will not attempt to restore the system

OPatch failed with error code 74

After going back to the read me again, I read Oracle’s “recommendation” to have the latest version of OPatch. It even supplies the document to use.  latest version of OPatch In this document there is a link to a video that shows you how to install the latest version of OPatch. Let me clear something up…

You need to download the OPatch version that matches up to your version of Oracle. For example, don’t use OPatch for 12c if you are using Oracle11g.

Once you have transferred Oracle’s download to your patches directory, follow the document’s easy steps.

cd $ORACLE_HOME
mv OPatch OPatch.bkp
unzip <download directory>/p6880880_<version>_<platform>.zip
cd OPatch
./opatch version

–Include OPatch in PATH

$ export PATH=$ORACLE_HOME/OPatch:$PATH

Now, my version of OPatch is up to date with my database version.

$ opatch lsinventory

OPatch version : 11.2.0.3.12
OUI version : 11.2.0.1.0

Now I can run the patch without fail!!

$ opatch apply

OUI-67073:UtilSession failed: Prerequisite check “CheckActiveFilesAndExecutables” failed.

Ugh. Another stinking issue. The solution will be my next blog topic.

To be continued…