ORA-15040: diskgroup is incomplete

<<Back to Oracle ASM Main Page

ORA-15040: diskgroup is incomplete

SQL> startup
ORA-00099: warning: no parameter file specified for ASM instance
ASM instance started

Total System Global Area 1140850688 bytes
Fixed Size                  8629704 bytes
Variable Size            1107055160 bytes
ASM Cache                  25165824 bytes
ORA-15110: no diskgroups mounted

Reason:

The reason of this error is simply the ASM is not able to find the some or all the disks.

Solution:

Investigate and make all the disks available to ASM to mount the disk group.
Make sure the disks has proper permissions.
If you are using AFD check following services are online
oracleacfs
oracleadvm
oracleoks 
oracleafd  

Source of Problem: Issue started after restart of the server

After restarting the server when I tried to start the ASM instance its started throwing error. ORA-15110: no diskgroups mounted

Investigation in my Case

Step1> ASM Logfile Scanning 

Looked in the ASM log file and noticed following errors 

WARNING: using default parameter settings without any parameter file
NOTE: remote asm mode is local (mode 0x301; from cluster type)
ERROR: Unable to get logical block size for spfile '+DG_TST_DATA/ASM/ASMPARAMETERFILE/registry.253.985607873'.

So it was very clear that ASM was not able to find the parameter file ( and yes that's what I expected because the parameter file reside in DG which was not mounted). So I decided to grab the parameters from last restart of ASM Instance and create the pfile and start the ASM with pfile.

Step2> Constructing of PFILE and Attempting to Start ASM with PFILE

From ASM Alert Log

Using parameter settings in server-side spfile +DG_TST_DATA/ASM/ASMPARAMETERFILE/registry.253.985607873
System parameters with non-default values:
  large_pool_size          = 12M
  remote_login_passwordfile= "EXCLUSIVE"
  asm_diskstring           = "AFD:*"
  asm_diskgroups           = "DG_TST_FRA"
  asm_diskgroups           = "DG_TST2_FRA"
  asm_diskgroups           = "DG_TST2_DATA"
  asm_diskgroups           = "DG_TEST"
  asm_power_limit          = 1

I created a pfile with highlighted parameters in bold above and tried to start the ASM instance again.

SQL> startup pfile='/u01/oragrid/12.2.0.1/grid/dbs/init+ASM.ora';

ASM instance started

Total System Global Area 1140850688 bytes
Fixed Size                  8629704 bytes
Variable Size            1107055160 bytes
ASM Cache                  25165824 bytes
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DG_TST_FRA" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15017: diskgroup "DG_TST2_FRA" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15017: diskgroup "DG_TST2_DATA" cannot be mounted
ORA-15040: diskgroup is incomplete
ORA-15017: diskgroup "DG_TEST" cannot be mounted
ORA-15040: diskgroup is incomplete

The error is still present

AS ROOT USER

Step3>Cross Check DISK at OS Level and 

#ls -lrt /dev/xvd*

Everything was fine

Step4> Validate if the DISKS are visible to ASM AFD

Logged in as root user and cross checked the disk. Everything seems fine. Even disks are visible to ASM

# export ORACLE_HOME=/u01/oragrid/12.2.0.1/grid
# export ORACLE_SID=+ASM
# export PATH=$PATH:$HOME/.local/bin:$HOME/bin:$ORACLE_HOME/bin
# asmcmd afd_lslbl

--------------------------------------------------------------------------------
Label                     Duplicate  Path
================================================================================
DISK011                               /dev/xvdd11
DISK012                               /dev/xvdd12
DISK013                               /dev/xvdd13
DISK014                               /dev/xvdd14
DISK015                               /dev/xvdd15
DISK10                                /dev/xvdd10
DISK11                                /dev/xvdc11
DISK12                                /dev/xvdc12
DISK13                                /dev/xvdc13
DISK14                                /dev/xvdc14
DISK15                                /dev/xvdc15

Step5> Validate if the DISKS are listed under  /dev/oracleafd/disks.

#ls -lrt /dev/oracleafd/disks

Nothing returned 

Note:- There was no disks in this directory and that's the issue

Step6> Check if the ACFS related modules are loaded

#lsmod | grep acfs

Nothing returned 

 #lsmod | grep oracle
oracleafd             212992  1

That's it. Now I can see there is no acfsload  service running and that's the issue.

Step6> Start the ACFSLOAD module 

 #acfsload start 

# lsmod | grep oracle
oracleacfs           4616192  0
oracleadvm            782336  1
oracleoks             655360  2 oracleacfs,oracleadvm
oracleafd             212992  1

Login as ASM owner and start the ASM instance.

$sqlplus "/as sysasm"

SQL> shut abort

SQL> startup

ASM instance started successfully. All the disks are now visible to ASM and ASM was able to mount the diskgroups successfully