An RMAN Rant

Posted: March 13, 2014 in Backup and Recovery
Tags: ,

Welcome back, Oracle fans!!

A Little OEM History Lesson

Before I jump in with my rant, let me be clear. The introduction of OEM12C was a huge step forward for Oracle. Those of us who have used OEM 10g and even OEM 9i will testify that there really is no comparison. When I used OEM 9i, 90% of my DBA work was performed with the UNIX command line and SQL Plus. About the only thing I used the OEM for was “Advanced Replication” administration.

At my next job, the company used OEM10g. This was a much better tool. It was great for performance management, user accounts and schema comparisons. It still lacked the basic needs a GUI should have in order to surpass the functionality of the Unix command line. At this point 40% of my DBA work was performed by the OEM. The remaining 60% was split between SQL Developer, PL/SQL Developer and the command line.

About three months ago, I installed OEM12C. I am still learning all of its capabilities but what I have learned and employed so far has been a pleasant surprise. I set up security, performance management, ASH Analytics, Dataguard, incident management and notifications, and the Oracle Scheduler including RMAN.


This brings me to my rant. About two months ago, I converted all of my RMAN jobs to the OEM12C for scheduling and management. I ran a configuration script on all my databases and I was ready to go. I will post a separate thread to provide details on what I did. Prior to OEM 12C, RMAN was set up with the RMAN command line console and managed in three places including our company job scheduler, CRON and OEM10g. It was a mess!! Now I can say with confidence that all of my RMAN jobs are managed by the OEM. My job just got a lot easier.

About a month ago, we decided to test the Dataguard capabilities in preparation for a server move to an offsite data center. Again, OEM made setting up Dataguard a snap. We decided to use the RMAN recovery database as our test case. We set up a standby database on our test server. This was complete in less than an hour. All went well until we performed a manual switchover. When we included scheduled jobs, all of our RMAN jobs started failing. OEM did not recognize the standby database as the recovery catalog when we made it the primary database.

In order to fix the problem, I went into OEM and added the new primary database as the recovery catalog. This solution is problematic if a fail-over occurs. Another proposed solution was to connect using the control file instead of the recovery catalog. This solution presents a problem when we attempt to resync the catalog. So I opened up an SR with Oracle Support. After about a week of ping pong, this was their final answer…

  You can not add the standby database directly in OEM12c as recovery catalog database. Catalog should be created manually before configuring the recovery catalog through EM.

Kindly refer the note Add RMAN Catalog To OEM 12c (Doc ID 1632006.1) for your reference.

So in your case, after switchover of catalog database completes, you need to add the new catalog primary database as new recovery catalog database in EM12c.

I have left out names to protect the guilty. I have responded to their solution and will update this thread if I get a better response.

I am also reaching out to you, the Oracle Enthusiast. Have you found a viable solution to this dilemma? If so, kindly reply to my post or send me a tweet.


Jason Brown

Follow me on Twitter!!  @dba_jay

  1. Reblogged this on @dba_jay and commented:

    I really should change the title from “An RMAN Rant” to “An OEM12C Rant”.

  2. Oracle Support added “additional” information…


    I have already suggested you, the way OEM 12c tool works with the recovery catalog. I hope you can understand, regarding the concern you have raised, is out of my scope of work. If you have any other concern, let me know I will be happy in resolving the same.

    We responded with this…
    We are not concerned with the “scope of work” of any individual support analyst. These are questions which must be answered regardless of your capability to answer them. If you need to pass this SR onto someone who is capable of answering these questions, then please do so immediately. If we need to escalate to do so, then please let us know?

    The question that we believe you may have answered is: EM12c is apparently not capable of handling the failover or switchover of a recovery catalog database to a standby.

    Since failover or switchover of a standby is normal Oracle and RMAN functionality, and since RMAN can do this easily outside of EM12c, then this is clearly a product deficiency or “bug” in EM12c.

    So, when can we expect this deficiency or “bug” to be corrected in EM12c?
    We look forward to your helpful response.

    After much ado about nonsense, Oracle support asked…

    I am raising an enhancement request. but for that I will require some information regarding following point. Kindly brief about it, so that it will be helpful for me.

    1) Describe why the current product functionality does not meet your needs. Explain in detail the enhancement you would like implemented.
    2) Describe your business expectations. Include key milestone dates and justifications as to why this request is important and the benefits your organization stands to gain should this request be accepted.

    We answered with…

    1) Describe why the current product functionality does not meet your needs. Explain in detail the enhancement you would like implemented.
    When the RMAN recovery catalog database is configured as a physical or logical standby database, the use of standard TNS names allows RMAN to automatically connect to the primary role database correctly, without need for manual reconfiguration. This failover functionality has been a part of Oracle Net since Oracle7. Unfortunately, EM12c does not seem to use the TNS failover functionality when connecting EM12c jobs to the recovery catalog. Instead, when the primary database role of the recovery catalog changes to another database instance due to Data Guard switchover or failover, EM12c requires that the recovery catalog be manually redefined, allowing EM12c backup-related jobs to fail until that is done.

    2) Describe your business expectations. Include key milestone dates and justifications as to why this request is important and the benefits your organization stands to gain should this request be accepted.
    Automated failover/switchover from database redundancy using Data Guard or RAC is important and expensive functionality in Oracle Database and Oracle Net which has been available since the 1990s. Oracle customers have been using Data Guard and/or RAC to protect the RMAN recovery catalog for many years, and the ability of Oracle Net to find an available instance automatically is an important part of that functionality. For EM12c not to support this automated functionality after so many years means that it is not a viable alternative for managing the backups in a high-availability environment. For that reason, our business expectation is that this functionality, standard to Oracle Database and Oracle Net for over a decade, be restored within EM12c immediately, so that EM12c can become a viable solution for managing backup jobs.

    I am hoping for a solution in the mean time but we may have to wait until the next version of OEM12C.

    Any suggestions??

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s