Recreate GI Management Repository database 12.1.0.1.5 fails with PLS-801

Key Facts:

  • Recreate the GI Management Repository database – fails with 12.1.0.1.5
  • Erorr: PLS-00801: internal error [*** ASSERT at file pdw1.c, line 3491; Failed To Get”
  • Reference:  BUG 20125473 – PLS-00801 ERROR OCCURRED WHEN APPLYING PATCH 19392372

Delete old  MGMTDB database

If needed start MGMTDB with a pfile
SQL> create pfile='/tmp/m.ora' from spfile='+DATA/_MGMTDB/spfile-MGMTDB.ora';
SQL> startup pfile='/tmp/m.ora'

On each node, as root user:
# $GRID_HOME/bin/crsctl stop res ora.crf -init
# $GRID_HOME/bin/crsctl modify res ora.crf -attr ENABLED=0 -init

As Grid User on the node where ora.mgmtdb resource is running execute:
$ $GRID_HOME/bin/dbca -silent -deleteDatabase -sourceDB -MGMTDB
If the above dbca -silent -deleteDatabase not work you may need to use asmcmd rm -rf command

Recreate the 12.1.0.1 MGMTDB database

Note that  there are different commands for 12.1.0.1  and  12.1.0.2  (see Doc ID 1589394.1)
[grid@gract1 ~]$ $GRID_HOME/bin/dbca -silent -createDatabase -templateName MGMTSeed_Database.dbc -sid -MGMTDB -gdbName _mgmtdb -storageType ASM -diskGroupName +DATA -datafileJarLocation $GRID_HOME//assistants/dbca/templates
     -characterset AL32UTF8 -autoGeneratePasswords -oui_internal
Copying database files
1% complete
3% complete
10% complete
17% complete
24% complete
31% complete
35% complete
Creating and starting Oracle instance
37% complete
42% complete
47% complete
52% complete
53% complete
56% complete
58% complete
Registering database with Oracle Grid Infrastructure
64% complete
Completing Database Creation
68% complete
79% complete
90% complete
100% complete
Look at the log file "/u01/app/grid/cfgtoollogs/dbca/_mgmtdb/_mgmtdb1.log" for further details.

--> Your really should check that log file. Even dbca reports no error there can be an error
    Review your database creation log file very carefully ! 

Reported error:
PRCD-1000 : Database _mgmtdb already exists
PRCR-1086 : resource ora.mgmtdb is already registered
DBCA_PROGRESS : 64%
Completing Database Creation
DBCA_PROGRESS : 68%
ORA-06550: line 1, column 7:
PLS-00801: internal error [*** ASSERT at file pdw1.c, line 3491; Failed To Get

DBCA_PROGRESS : 79%
DBCA_PROGRESS : 90%
DBCA_PROGRESS : 100%
Database creation complete. For details check the logfiles at:
 /u01/app/grid/cfgtoollogs/dbca/_mgmtdb.
Database Information:
Global Database Name:_mgmtdb

Delete and recreate the  MGMT resources

[root@gract1 _mgmtdb]# srvctl stop mgmtlsnr
[root@gract1 _mgmtdb]# srvctl remove mgmtdb
Remove the database _mgmtdb? (y/[n]) y
[root@gract1 _mgmtdb]# srvctl remove mgmtlsnr
[root@gract1 gract1]#  srvctl add mgmtlsnr
[root@gract1 gract1]#  srvctl add mgmtdb
[root@gract1 gract1]#  srvctl modify mgmtdb  -spfile '+DATA/_MGMTDB/spfile-MGMTDB.ora'
[root@gract1 gract1]# srvctl stop mgmtdb
PRCC-1016 : _mgmtdb was already stopped
[root@gract1 gract1]# srvctl start mgmtdb
*****  Cluster Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.MGMTLSNR                   1   ONLINE       ONLINE       gract1          169.254.235.72 192.1 68.2.111,STABLE
ora.mgmtdb                     1   ONLINE       ONLINE       gract1          Open,STABLE
Enable and start ora.crf resource.
On each node, as root user:
# $GRID_HOME/bin/crsctl modify res ora.crf -attr ENABLED=1 -init
# $GRID_HOME/bin/crsctl start res ora.crf -init
*****  Local Resources: *****
Resource NAME               INST   TARGET       STATE        SERVER          STATE_DETAILS
--------------------------- ----   ------------ ------------ --------------- -----------------------------------------
ora.crf                        1   ONLINE       ONLINE       gract1          STABLE

 

Verifying  oclumon functionality

[grid@gract2 ~]$   oclumon dumpnodeview -allnodes
CRS-9118-Grid Infrastructure Management Repository connection error 
 ORA-12514: TNS:listener does not currently know of service requested in connect descriptor

[grid@gract1 ~]$ lsnrctl status MGMTLSNR

Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=MGMTLSNR)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.2.111)(PORT=1531)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=169.254.235.72)(PORT=1531)))
The listener supports no services
The command completed successfully
-->  Service _mgmtdb not registered by MGMTLSNR

[grid@gract1 ~]$  lsnrctl status
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
--> Service _mgmtdb is configured with default listener 

Local listener not defined 
SQL> show parameter local_listener
NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
local_listener                 string

Change LISTENER_NETWORKS ( from  Bug 19313202 )
SQL>  ALTER  SYSTEM SET LISTENER_NETWORKS = '((NAME=private_network)
 (LOCAL_LISTENER="(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)
        (HOST=169.254.235.72)(PORT=1531))(ADDRESS=(PROTOCOL=TCP)
        (HOST=192.168.2.111)(PORT=1531)))"))' 
SCOPE=MEMORY SID='-MGMTDB'; 

SQL> show parameter listener
NAME                     TYPE     VALUE
------------------------------------ ----------- ------------------------------
listener_networks             string     ((NAME=private_network) (LOCAL_LISTENER="(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST
                                           169.254.235.72)(PORT=1531))(ADDRESS=(PROTOCOL=TCP)(HOST=192.
                                           168.2.111)(PORT=1531)))"))
local_listener                string
remote_listener               string

[grid@gract1 ~]$ lsnrctl status MGMTLSNR
..
Services Summary...
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service.

[grid@gract1 ~]$    lsnrctl status
..
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...

--> Service  "_mgmtdb" is not registered for both Listeners   MGMTLSNR and default LISTENER                                 

[grid@gract1 ~]$ oclumon dumpnodeview -allnodes
----------------------------------------
Node: gract1 Clock: '14-10-28 08.02.01' SerialNo:8758 
----------------------------------------
SYSTEM:
#pcpus: 1 #vcpus: 1 cpuht: N chipname: Intel(R) cpu: 17.76 cpuq: 16 physmemfree: 218160 physmemtotal: 4354416 mcache: 2374288 swapfree: 5053196 swaptotal: 5210108 hugepagetotal: 0 hugepagefree: 0 hugepagesize: 2048 ior: 95 iow: 207 ios: 30 swpin: 0 swpout: 0 pgin: 95 pgout: 196 netr: 35.711 netw: 32.879 procs: 340 rtprocs: 10 #fds: 12832 #sysfdlimit: 6815744 #disks: 18 #nics: 5 nicErrors: 0

TOP CONSUMERS:
topcpu: 'mdb_vktm_-mgmtd(13626) 3.40' topprivmem: 'ocssd.bin(11228) 98072' topshm: 'ora_mman_erp_1(15523) 229804' topfd: 'ocssd.bin(11228) 305' topthread: 'console-kit-dae(3598) 64' 

....

[grid@gract3 ~]$ oclumon dumpnodeview -allnodes
--> works for all 3 nodes 
[grid@gract2 ~]$  oclumon dumpnodeview -allnodes
--> works for all 3 nodes 
[grid@gract1 ~]$ oclumon dumpnodeview -allnodes
----------------------------------------
Node: gract1 Clock: '14-10-28 08.19.56' SerialNo:8973 
----------------------------------------
--> Works only on local node gract1 - need to investigate 

Checking patch status of -MGMTDB:

[grid@gract1 ~]$ env | grep SID
ORACLE_SID=-MGMTDB

SQL> select * from dba_registry_history;
ACTION_TIME               ACTION                  NAMESPACE              VERSION
------------------------------ ------------------------------ ------------------------------ ------------------------------
    ID BUNDLE_SERIES          COMMENTS
---------- ------------------------------ ----------------------------------------
24-MAY-13 12.21.12.617091 PM   APPLY                  SERVER                 12.1.0.1
     0 PSU                  Patchset 12.1.0.0.0

27-OCT-14 07.34.33.119553 PM   APPLY                  SERVER                 12.1.0.1
     5 PSU                  PSU 12.1.0.1.5

SQL> select * from  dba_registry_sqlpatch ;
no rows selected

--> Datapatch not yet installed - Installing 

[grid@gract1 ~]$ $GRID_HOME/OPatch/datapatch -verbose
SQL Patching tool version 12.1.0.1.0 on Tue Oct 28 06:56:35 2014
Copyright (c) 2014, Oracle.  All rights reserved.

Connecting to database...OK 
Determining current state...
Currently installed SQL Patches: 
Currently installed C Patches: 19121550
Adding patches to installation queue and performing prereq checks...
Installation queue:
  Nothing to roll back
  The following patches will be applied: 19121550
Installing patches...
Patch installation complete.  Total patches installed: 1
Validating logfiles...
Patch 19121550 apply: WITH ERRORS
  logfile: /u01/app/121/grid/sqlpatch/19121550/19121550_apply__MGMTDB_2014Oct28_06_57_09.log (no errors)
  catbundle generate logfile: /u01/app/grid/cfgtoollogs/catbundle/catbundle_PSU__MGMTDB__mgmtdb_GENERATE_2014Oct28_06_57_11.log (no errors)
  catbundle apply logfile: /u01/app/grid/cfgtoollogs/catbundle/catbundle_PSU__MGMTDB__mgmtdb_APPLY_2014Oct28_06_57_14.log (errors)
    Error at line 102: ORA-06550: line 1, column 7:
    Error at line 103: PLS-00801: internal error [*** ASSERT at file pdw1.c, line 3491; Failed To Get

See support note 1609718.1 for information on how to resolve the above errors

--> The above problems seems to be related to  
BUG 20125473 - PLS-00801 ERROR OCCURRED WHEN APPLYING PATCH 19392372 

Please open a SR with Oracle Support to get details about a potential WA.

Reference

  • BUG 20125473 – PLS-00801 ERROR OCCURRED WHEN APPLYING PATCH 19392372
  • DBCA is not failing if Management Database Resource Creation Fails (Doc ID 1631311.1)
  • Bug 19313202 : MGMTDB HAS ALL LISTENERS IN LOCAL_LISTENER PARAMETER

3 thoughts on “Recreate GI Management Repository database 12.1.0.1.5 fails with PLS-801”

  1. Hi Helmut

    I had exactly the same error applying PSU 12.1.0.1.5

    ” Error at line 103: PLS-00801: internal error [*** ASSERT at file pdw1.c, line 3491; Failed To Get”

    How did you resolve it?

    Thanks in advance.
    regards,
    Lawrence

    1. Hi Lawrence,

      please ask Oracle Support for the WA explained in
      BUG 20125473 – PLS-00801 ERROR OCCURRED WHEN APPLYING PATCH 19392372

      br
      Helmut

  2. Thanks Helmut, much appreciated. When I applied this patch on 17th October, it was only 3 days old having been released on 14th so when we reported the error Oracle didn’t have an answer. Thanks for a very helpful blog.

    regards
    Lawrence

Leave a Reply

Your email address will not be published. Required fields are marked *