OEM12C: After changing DBSNMP password on RAC database, target monitoring is no longer working

Posted: September 2, 2015 in Change DSNMP on RAC
Tags: ,

I have an update to this post that should make life much easier when updating the password for DBSNMP…

Before you go dropping any targets or the DBSNMP user, try these steps first…

  1. Log into OEM 12C and navigate to your cluster database.
  2. Hit the dropdown for Oracle Database –> Target Configuration –> Test Connection.  This will tell you the current status of DBSNMP.
    1. If the account is expired, OEM will guide you through a password change. Lucky for us, OEM will not ask you for the old password. 
    2. Once you change the password, please store it somewhere safe. Trust me, you may need it some day!!
  3. Let’s say the DBSNMP account is set to expire soon… from the Target Configuration page, you can change the monitor password (DBSNMP).
    1. Under the Test Connection button, change the Monitor Password. 
    2. Click Next and Submit.
  4. This will change the password at the Cluster Database Instance and in OEM. Voila!! No more walking on broken glass, hoping you don’t get passwords out of SYNC!! 
  5. Log into SQL Plus as DBSNMP just to ensure it worked. If it did, please ignore the rest of this post!!

Here’s a little lunch time special for Oracle fans and foes alike.

The moral of my story is use EMCLI whenever possible

Problem: I created an issue the other day when one of my OEM 12C database targets were stuck in a “pending state after a blackout”. It seems every time I conduct a blackout, some of my targets get stuck in a pending state. This appears to be a bug that I am hoping to introduce with Oracle Support. How did I create this issue? Read on and please learn from my mistake as I most certainly have.

Solution: I found the solution but I did it in an unconventional way.

Attempts to solve this problem made it worse until steps 4&5: Step 3 might have worked had I skipped step 2 and known the old password.

Warning: Don’t ever perform step two unless you want to “unexpire” the password with its same previous password.

  1. Dropped and rediscovered the target in OEM. OEM asked for the DBSNMP password and I did not know the DBSNMP password.
  2. Changed the password manually using sqlplus. This got the password out of sync between the database and OEM.
  3. emcli update_db_password. This job failed. My dbsnmp password had a special character in it. This could have been fixed with the following syntax…

    emcli update_db_password -target_name=orcl -target_type=rac_database -user_name=dbsnmp -change_at_target=yes -old_password=Pa\$\$word1 -new_password=Pa\$\$word2 -retype_new_password=Pa\$\$word2

  4. drop and recreate DBSNMP user.
    1. Drop Cluster database target from OEM12C console.
    2. Ensure there are no sessions running as DBSNMP in the database before you run catsnmp.sql.
    3. Run catnsnmp.sql and catsnmp.sql. (located at $ORACLE_HOME/rdbms/admin/) The first script drops the user and the second script creates the user.
    4. Unlock the DBSNMP user and store the password somewhere safe.
    5. Rediscover database target. I got “slow_connection/firewall error” OEM discovered the target and showed it as being up but I could not connect.
  5. Navigated to Cluster Database–>Target Setup–>Monitoring Configuration
    1. Discovered the Listener Machine Name did not match other database targets on the host target.
    2. Edited the Listener Machine Name to match the other targets on the host.
    3. Tested the connection and connected to the database target.

I hope this is helpful to you. As I said earlier, to change the DBSNMP password, use emcli’s change password command. I hope to publish another article once I figure out while targets get stuck in a pending state following a blackout. I have read many articles on this and have yet to get to the bottom of it. The one thing I haven’t tried is using EMCLI to conduct the blackout. Looks like we will never get completely away from command line. That’s not necessarily a bad thing…

Thanks for reading,

Jason Brown

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s