Skip to main content

Important Tools to Troubleshoot ASM Issues


<<Back to Oracle ASM Main Page

Utilities:Tools Used to Troubleshoot ASM Issues

In today's post I will discuss 3 most import utilities used to troubleshoot ASM issues so lets start without any further delay. These are
KFOD (Kernel Files OSM Disk)
KFED (Kernel File Metadata Editor)
AMDU (ASM Metadata Dump Utility)
KFOD: KFOD utility is very useful to troubleshoot the ASM disk issues. The utility is also used by oracle internal tools like DBCA, DBUA and ASMCA to perform disk discovery.
NOTE: Other utilities are available for determining the status of a disk's header which should fulfill the business need.V$ASM_DISK from an ASM instance;asmcmd lsdsk;kfod. kfed is a utility that should only be used under the guidance of Oracle support
$ kfod help=y
 Usage: kfod op=<op> asm_diskstring=... | pfile=...

If you type just kfod it will display ASM instance and Grid home 
Valid options types with KFOD are
DISKS/CANDIDATES/MISSING/GROUPS/INSTS/VERSION/PATCHES/PATCHLVL/CLIENTS/RM/RMVERS/DFLTDSTR/GPNPDSTR/METADATA/CREDCRECLUS/GETCLSTYPE/CREDEXPORT/GETASMGUID/CREDDELCLUS/CREDVERIFY/UPGRADEVERIFY/LSCC/CREDLIST/CELLCONFIG/ALL
$kfod
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /u01/oragrid/12.2.0.1/grid
Find version
$kfod op=version
--------------------------------------------------------------------------------
ORACLE_SID RAC VERSION
================================================================================
      +ASM  NO 12.2.0.1.0
List ASM DG known to an ASM instance
$kfod _asmsid=+ASM
--------------------------------------------------------------------------------
Group          Size          Free Redundancy Name
================================================================================
   1:      30720 MB      30666 MB     EXTERN DG_TEST_REDO
   2:      30720 MB      30658 MB     EXTERN DG_TEST_FRA
   3:      30720 MB      30657 MB     EXTERN DG_TEST_DATA
List ASM disks
$kfod disks=all op=disks
--------------------------------------------------------------------------------
 Disk          Size Path                                     User     Group
================================================================================
   1:      30720 MB AFD:DATA01
   2:      30720 MB AFD:FRA01
   3:      30720 MB AFD:REDO01
List diskgroups with asm compatibility
$kfod asmcompatibility=true op=groups
--------------------------------------------------------------------------------
Group          Size          Free Redundancy Name           ASM Compatibility
================================================================================
   1:      30720 MB      30666 MB     EXTERN DG_TEST_REDO          12.2.0.1.0
   2:      30720 MB      30658 MB     EXTERN DG_TEST_FRA           12.2.0.1.0
   3:      30720 MB      30657 MB     EXTERN DG_TEST_DATA          12.2.0.1.0
Metadata size
kfod disks=all op=METADATA
Approximate Metadata Size: 635
kfod disks=all shallow=TRUE
Shallow discovery returned 3 devices
--------------------------------------------------------------------------------
ORACLE_SID ORACLE_HOME
================================================================================
      +ASM /u01/oragrid/12.2.0.1/grid
Note: You can run KFOD in verbose (verbose=none,errors,warnings,functions,all)

$kfod verbose=all disks-all

KFED: the utility allows us to read and analyze and ASM diskgroup header. sometime this information is needed to investigate the DG issue if the DG can not be mounted. The utility can even update the DG header and therefore be careful while using it.
Permitted operations are READ/WRITE/MERGE/REPAIR/NEW/FORM/FIND/STRUCT/STAMP
type kfed help to find the supported options and available switches. To see the detail of a command type help kfed command name eg help kfed read I will walk you through here with some important one.

Reading ASM Disk Header Using KFED

$ kfed read AFD:DATA01 |egrep 'size|name|type|spfile'
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD <=ASM disk header
kfdhdb.dskname:                  DATA01 ; 0x028: length=6 <=ASM disk name
kfdhdb.grpname:            DG_TEST_DATA ; 0x048: length=12 <= ASM DGname
kfdhdb.fgname:                   DATA01 ; 0x068: length=6 <=ASM Failgroup name
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200 <=Disk sector size
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000 <=ASM block size
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000 <= ASM AU size
kfdhdb.dsksize:                   30720 ; 0x0c4: 0x00007800 <=ASM disk size
kfdhdb.spfile:                       60 ; 0x0f4: 0x0000003c  <= ASM Spfile location

KFED returns a readable output only when a disk device is/was the part of an ASM diskgroup else it will through below exception
KFED-00322: invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]
AMDU:AMDU is an utility to extract all the available metadata from one or more ASM disks and/or generate formatted printouts of individual blocks
$asmcmd lsdg
State    Type    Rebal  Sector  Logical_Sector  Block       AU  Total_MB  Free_MB  Req_mir_free_MB  Usable_file_MB  Offline_disks  Voting_files  Name
MOUNTED  EXTERN  N         512             512   4096  1048576     30720    30666                0           30666              0             N  DG_TEST/
MOUNTED  EXTERN  N         512             512   4096  1048576     30720    30657                0           30657              0             N  DG_TEST_DATA/
MOUNTED  EXTERN  N         512             512   4096  1048576     30720    30658                0           30658              0             N  DG_TEST_FRA/
MOUNTED  EXTERN  N         512             512   4096  1048576     30720    30666                0           30666              0             N  DG_TEST_REDO/
$amdu diskstring='AFD:*' -dump 'DG_TEST'amdu_2018_06_11_11_10_49/
$cd amdu_2018_06_11_11_10_49/
$ls -lrt-rw-r--r-- 1 oragrid oinstall     4320 Jun 11 11:10 DG_TEST.map
-rw-r--r-- 1 oragrid oinstall 54534144 Jun 11 11:10 DG_TEST_0001.img
-rw-r--r-- 1 oragrid oinstall     6322 Jun 11 11:10 report.txt
$cat report.txt


********************************* DISCOVERY **********************************
----------------------------- DISK REPORT N0001 ------------------------------
                Disk Path: AFD:DATA01
           Unique Disk ID:
               Disk Label: DATA01
     Physical Sector Size: 512 bytes
                Disk Size: 30720 megabytes
               Group Name: DG_TEST_DATA
                Disk Name: DATA01
       Failure Group Name: DATA01
              Disk Number: 0
            Header Status: 3
       Disk Creation Time: 2018/06/05 16:32:35.361000
          Last Mount Time: 2018/06/08 15:05:05.496000
    Compatibility Version: 0x0c200100(12020001)
         Disk Sector Size: 512 bytes
         Disk size in AUs: 30720 AUs
         Group Redundancy: 1
      Metadata Block Size: 4096 bytes
                  AU Size: 1048576 bytes
                   Stride: 113792 AUs
      Group Creation Time: 2018/06/05 16:32:35.045000
  File 1 Block 1 location: AU 10
     Voting File Start AU: 0
       Voting File End AU: 0
              OCR Present: NO
             Exadata Disk: NO
    kfed Write IP address: 192.168.1.6
          kfed Write Time: 1528463105
----------------------------- DISK REPORT N0002 ------------------------------
                Disk Path: AFD:FRA01
           Unique Disk ID:
               Disk Label: FRA01
     Physical Sector Size: 512 bytes
                Disk Size: 30720 megabytes
               Group Name: DG_TEST_FRA
                Disk Name: FRA01
       Failure Group Name: FRA01
              Disk Number: 0
            Header Status: 3
       Disk Creation Time: 2018/06/06 10:54:52.140000
          Last Mount Time: 2018/06/08 15:05:05.539000
    Compatibility Version: 0x0c200100(12020001)
         Disk Sector Size: 512 bytes
         Disk size in AUs: 30720 AUs
         Group Redundancy: 1
      Metadata Block Size: 4096 bytes
                  AU Size: 1048576 bytes
                   Stride: 113792 AUs
      Group Creation Time: 2018/06/06 10:54:51.916000
  File 1 Block 1 location: AU 10
     Voting File Start AU: 0
       Voting File End AU: 0
              OCR Present: NO
             Exadata Disk: NO
    kfed Write IP address: 192.168.1.6
          kfed Write Time: 1528463105
----------------------------- DISK REPORT N0003 ------------------------------
                Disk Path: AFD:REDO01
           Unique Disk ID:
               Disk Label: REDO01
     Physical Sector Size: 512 bytes
                Disk Size: 30720 megabytes
               Group Name: DG_TEST_REDO
                Disk Name: REDO01
       Failure Group Name: REDO01
              Disk Number: 0
            Header Status: 3
       Disk Creation Time: 2018/06/06 11:09:34.614000
          Last Mount Time: 2018/06/08 15:05:05.568000
    Compatibility Version: 0x0c200100(12020001)
         Disk Sector Size: 512 bytes
         Disk size in AUs: 30720 AUs
         Group Redundancy: 1
      Metadata Block Size: 4096 bytes
                  AU Size: 1048576 bytes
                   Stride: 113792 AUs
      Group Creation Time: 2018/06/06 11:09:34.230000
  File 1 Block 1 location: AU 10
     Voting File Start AU: 0
       Voting File End AU: 0
              OCR Present: NO
             Exadata Disk: NO
    kfed Write IP address: 192.168.1.6
          kfed Write Time: 1528463105
----------------------------- DISK REPORT N0004 ------------------------------
                Disk Path: AFD:TEST01
           Unique Disk ID:
               Disk Label: TEST01
     Physical Sector Size: 512 bytes
                Disk Size: 30720 megabytes
               Group Name: DG_TEST
                Disk Name: TEST01
       Failure Group Name: TEST01
              Disk Number: 0
            Header Status: 3
       Disk Creation Time: 2018/06/11 10:46:28.905000
          Last Mount Time: 2018/06/11 10:46:38.577000
    Compatibility Version: 0x0c200100(12020001)
         Disk Sector Size: 512 bytes
         Disk size in AUs: 30720 AUs
         Group Redundancy: 1
      Metadata Block Size: 4096 bytes
                  AU Size: 1048576 bytes
                   Stride: 113792 AUs
      Group Creation Time: 2018/06/11 10:46:28.607000
  File 1 Block 1 location: AU 10
     Voting File Start AU: 0
       Voting File End AU: 0
              OCR Present: NO
             Exadata Disk: NO
    kfed Write IP address: 192.168.1.6
          kfed Write Time: 1528706798
***************** Slept for 6 seconds waiting for heartbeats *****************
************************* SCANNING DISKGROUP DG_TEST *************************
            Creation Time: 2018/06/11 10:46:28.607000
         Disks Discovered: 1
               Redundancy: 1
                  AU Size: 1048576 bytes
      Metadata Block Size: 4096 bytes
     Physical Sector Size: 512 bytes
          Metadata Stride: 113792 AU
   Duplicate Disk Numbers: 0
                  Mounted: No

---------------------------- SCANNING DISK N0004 -----------------------------
Disk N0004: 'AFD:TEST01'
           Allocated AU's: 54
                Free AU's: 30666
       AU's read for dump: 54
       Block images saved: 13314
        Map lines written: 54
          Heartbeats seen: 0
  Corrupt metadata blocks: 0
        Corrupt AT blocks: 0

----------------------- SUMMARY FOR DISKGROUP DG_TEST ------------------------
           Allocated AU's: 54
                Free AU's: 30666
       AU's read for dump: 54
       Block images saved: 13314
        Map lines written: 54
          Heartbeats seen: 0
  Corrupt metadata blocks: 0
        Corrupt AT blocks: 0

******************************* END OF REPORT ********************************

Comments

Post a Comment

Popular posts from this blog

How to Power On/off Oracle Exadata Machine

<<Back to Exadata Main Page How to Power On/off Oracle Exadata Machine Oracle Exadata machines can be powered on/off either by pressing the power button on front of the server or by logging in to the ILOM interface. Powering on servers using  button on front of the server The power on sequence is as follows. 1. Start Rack, including switches  Note:- Ensure the switches have had power applied for a few minutes to complete power on  configuration before starting Exadata Storage Servers 2.Start Exadata Storage Servers  Note:- Ensure all Exadata Storage Servers complete the boot process before starting the   database servers 3. Start Database Servers Powering On Servers Remotely using ILOM The ILOM can be accessed using the Web console, the command-line interface (CLI), IPMI, or SNMP. For example, to apply power to server dm01cel01 using IPMI, where dm01cel01-ilom is the host name of the ILOM for the server to be powered on, run the
Post a Comment
Read more

How to Find VIP of an Oracle RAC Cluster

<<Back to Oracle RAC Main Page How to Find Out VIP of an Oracle RAC Cluster Login clusterware owner (oracle) and execute the below command to find out the VIP hostname used in Oracle RAC $ olsnodes -i node1     node1-vip node2     node2-vip OR $ srvctl config nodeapps -viponly Network 1 exists Subnet IPv4: 10.0.0.0/255.255.0.0/bondeth0, static Subnet IPv6: Ping Targets: Network is enabled Network is individually enabled on nodes: Network is individually disabled on nodes: VIP exists: network number 1, hosting node node1 VIP Name: node1-vip VIP IPv4 Address: 10.0.0.1 VIP IPv6 Address: VIP is enabled. VIP is individually enabled on nodes: VIP is individually disabled on nodes: VIP exists: network number 1, hosting node node2 VIP Name: node2-vip VIP IPv4 Address: 10.0.0.2 VIP IPv6 Address: VIP is enabled. VIP is individually enabled on nodes: VIP is individually disabled on nodes:
1 comment
Read more

ORA-28374: typed master key not found in wallet

<<Back to Oracle DB Security Main Page ORA-46665: master keys not activated for all PDBs during REKEY SQL> ADMINISTER KEY MANAGEMENT SET KEY FORCE KEYSTORE IDENTIFIED BY xxxx WITH BACKUP CONTAINER = ALL ; ADMINISTER KEY MANAGEMENT SET KEY FORCE KEYSTORE IDENTIFIED BY xxxx WITH BACKUP CONTAINER = ALL * ERROR at line 1: ORA-46665: master keys not activated for all PDBs during REKEY I found following in the trace file REKEY: Create Key in PDB 3 resulted in error 46658 *** 2019-02-06T15:27:04.667485+01:00 (CDB$ROOT(1)) REKEY: Activation of Key AdnU5OzNP08Qv1mIyXhP/64AAAAAAAAAAAAAAAAAAAAAAAAAAAAA in PDB 3 resulted in error 28374 REKEY: Keystore needs to be restored from the REKEY backup.Aborting REKEY! Cause: All this hassle started because I accidently deleted the wallet and all wallet backup files too and also forgot the keystore password. There was no way to restore the wallet back. Fortunately in my case the PDB which had encrypted data was supposed to be deco
7 comments
Read more

ORA-16905: The member was not enabled yet

<<Back to Oracle DataGuard Main Page ORA-16905 Physical Standby Database is disabled DGMGRL> show configuration; Configuration - DG_ORCL1P   Protection Mode: MaxPerformance   Members:   ORCL1PP - Primary database     ORCL1PS - Physical standby database (disabled)       ORA-16905: The member was not enabled yet. Fast-Start Failover:  Disabled Configuration Status: SUCCESS   (status updated 58 seconds ago) DGMGRL> DGMGRL> enable database 'ORCL1PS'; Enabled. DGMGRL>  show configuration; Configuration - DG_ORCL1P   Protection Mode: MaxPerformance   Members:   ORCL1PP - Primary database     ORCL1PS - Physical standby database Fast-Start Failover:  Disabled Configuration Status: SUCCESS   (status updated 38 seconds ago)
Post a Comment
Read more

How to Switch Log File from All Instances in RAC

<<Back to Oracle RAC Main Page Switch The Log File of All Instances in Oracle RAC. In many cases you need to switch the logfile of the database. You can switch logfile using alter system switch logfile command but if you want to switch the logfile from all the instances you need to execute the command on all the instances individually and therefore you must login on all the instances. You can avoid this and switch logfile of all instances by just running the below command from any of the instance in RAC database SQL> ALTER SYSTEM SWITCH ALL LOGFILE;   System altered.
Post a Comment
Read more

ORA-65104: operation not allowed on an inactive pluggable database alter pluggable database open

<<Back to DB Administration Main Page ORA-65104: operation not allowed on an inactive pluggable database SQL> alter pluggable database TEST_CLON open; alter pluggable database TEST_CLON open * ERROR at line 1: ORA-65104: operation not allowed on an inactive pluggable database Cause The pluggable database status was UNUSABLE. It was still being created or there was an error during the create operation. A PDB can only be opened if it is successfully created and its status is marked as NEW in cdb_pdbs.status column SQL> select PDB_NAME,STATUS from cdb_pdbs; PDB_NAME             STATUS -------------------- --------------------------- PDB$SEED             NORMAL TEST_CLON            UNUSABLE Solution:  Drop the PDB and create it again. Related Posts How to Clone Oracle PDB (Pluggable Database) with in the Same Container
Post a Comment
Read more

ORA-46630: keystore cannot be created at the specified location

<<Back to DB Administration Main Page ORA-46630: keystore cannot be created at the specified location CDB011> ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '+DATAC4/CDB01/wallet/' IDENTIFIED BY "xxxxxxx"; ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '+DATAC4/CDB01/wallet/' IDENTIFIED BY "EncTest123" * ERROR at line 1: ORA-46630: keystore cannot be created at the specified location Cause  Creating a keystore at a location where there is already a keystore exists Solution To solve the problem, use a different location to create a keystore (use ENCRYPTION_WALLET_LOCATION in sqlnet.ora file to specify the keystore location), or move this ewallet.p12 file to some other location. Note: Oracle does not recommend deleting keystore file (ewallet.p12) that belongs to a database. If you have multiple keystores, you can choose to merge them rather than deleting either of them.
Post a Comment
Read more

Starting RMAN and connecting to Database

  <<Back to Oracle Backup & Recovery Main Page Starting RMAN and connecting to Database Starting RMAN and connecting to Database To start RMAN you need to set the environment and type rman and press enter. You can connect to database either using connect command or using command line option. using command line option localhost:$ export ORACLE_HOME=/ora_app/product/18c/dbd2 localhost:$ export PATH=$ORACLE_HOME/bin:$PATH localhost:$ export ORACLE_SID=ORCL1P localhost:$ rman target / Recovery Manager: Release 18.0.0.0.0 - Production on Sun Apr 4 08:11:01 2021 Version 18.11.0.0.0 Copyright (c) 1982, 2018, Oracle and/or its affiliates.  All rights reserved. connected to target database: ORCL1P (DBID=4215484517) RMAN> using connect option localhost:$ rman RMAN> connect target sys@ORCL1P  target database Password:******** connected to target database: ORCL1P (DBID=4215484517) NOTE: To use connect command you need to ensure that  you have proper TNS sentry for database (ORCL
Post a Comment
Read more

How to Attach to a Datapump Job and Check Status of Export or Import

<<Back to Oracle DATAPUMP Main Page How to check the progress of  export or import Jobs You can attach to the export/import  job using ATTACH parameter of oracle datapump utility. Once you are attached to the job you check its status by typing STATUS command. Let us see how Step1>  Find the Export/Import Job Name You can find the datapump job information from  DBA_DATAPUMP_JOBS or  USER_DATAPUMP_JOBS view. SQL> SELECT OWNER_NAME,JOB_NAME,OPERATION,JOB_MODE,STATE from DBA_DATAPUMP_JOBS; OWNER_NAME JOB_NAME                       OPERATION            JOB_MODE   STATE ---------- ------------------------------ -------------------- ---------- ---------- SYSTEM     SYS_EXPORT_FULL_02             EXPORT               FULL       EXECUTING OR You can also find the job name for export/import in logfile in beginning itself. Step2>Attach to the Job and check status One you get the Export/Import Job Name attach the job and check its status. You can attach or det
7 comments
Read more

ORA-15040: diskgroup is incomplete

<<Back to Oracle ASM Main Page ORA-15040: diskgroup is incomplete SQL> startup ORA-00099: warning: no parameter file specified for ASM instance ASM instance started Total System Global Area 1140850688 bytes Fixed Size                  8629704 bytes Variable Size            1107055160 bytes ASM Cache                  25165824 bytes ORA-15110: no diskgroups mounted Reason: The reason of this error is simply the ASM is not able to find the some or all the disks. Solution: Investigate and make all the disks available to ASM to mount the disk group. Make sure the disks has proper permissions. If you are using AFD check following services are online oracleacfs oracleadvm oracleoks  oracleafd   Source of Problem : Issue started after restart of the server After restarting the server when I tried to start the ASM instance its started throwing error.  ORA-15110: no diskgroups mounted Investigation in my Case Step1> ASM Logfile Scanning  Looked i
2 comments
Read more