Upgrading RAC DB to 12.2.0.1: ORA-00443: background process “IPC0” did not start.

Posted: October 10, 2017 in Database Upgrades to 12.2.0.1
Tags: , ,

The day after I published an abbreviated list for upgrading to 12.2.0.1, my partner and I were upgrading two QA Databases which happened to be 2 node RAC.

As DBUA was doing its final restart, it spit out this error stack…

Error ignored: PRCR-1013 : Failed to start resource ora.*****.dbPRCR-1064 : Failed to start resource ora.q*****.db on node xdt**dbadm02CRS-5017:
The resource action “ora.q*****.db start” encountered the following error: ORA-00443: background process “IPC0” did not start.
For details refer to “(:CLSN00107:)” in “/u01/app/grid/diag/crs/xdt**dbadm02/crs/trace/crsd_oraagent_oracle.trc”.CRS-2674: Start of ‘ora.q****.db’
on ‘xdt**dbadm02’ failedCould not complete the RAC specific configuration changes.

As any good DBA would do, I Googled ORA-00443: background process “IPCO” did not start. I got nothing. Not a single suggestion. My partner had an open SR with Oracle on this same issue but it has been open for 30+ days. This was my first experience with this where I wasn’t able to start one of the nodes. Previously, I removed the srvctl entry, re-created it, and restarted the database with success. No such luck this time.

As it turns out, there are a few documents including a bug id for this issue.

1) Exadata: Connect After Upgrade Returns ORA-600 [ipc_create_que_1], [2], [8] ( Doc ID 2134464.1 )
2) ORA-00600 [ipc_create_que_1] on Instance Startup ( Doc ID 2110743.1 )

3) Bug 25498930 – Startup fails due to IPC0 process getting ORA-600 [ipc_create_que_1] with underlying rdma_bind_addr failed – OS errno 98 ( Doc ID 25498930.8 )

Rather than update the pfile, we did this:

On the node that opens,

SQL> alter system set “_ksxp_xrc_cm_port”=1026 scope=both sid=’*’;

From srvctl start the node that didn’t start the first time.

srvctl start database -d db_name

That did it. Note that support recommends using a port between 1026 and 2048. Be creative! Choose your port with style!

Since I have had trouble with Oracle Support in the past, it’s nice to see something work in our favor!

Thanks for reading! Also check out my new blog at <free-indeed.blog>! I talk about my past (not so great) experience in a really bad church.

Regards,

Jason

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s