askMLabs

This post talks about an Exadata RAC ASM issue. This issue is observed after completing Exadata patching.  It is observed that the ACFS mount point is missing. The following steps explains how we identified the issue and How can we troubleshoot this issue.

This is a 8 node Exadata full rack. The ACFS mount point is missing on first node , but all other 7 nodes have this ASM mount point.

First try to get the missing mount point …

Use “df –h” on any node where you can see the ACFS mount point. In our case getting the ACFS mount point details from node2 exaaskmdb02

[root@exaaskmdb02 ~]# df -h /u01/app/askmacfs
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/askmdata-282  512G   64G  449G  13% /u01/app/askmacfs
[root@exaaskmdb01 ~]#

Check1 :

Using these details, try to manually mount the ACFS on node1 exaaskmdb01

[root@exaaskmdb01 ~]# mount -t acfs /dev/asm/askmdata-282  /u01/app/askmacfs
mount.acfs: CLSU-00100: operating system function: open64 failed with error data: 2
mount.acfs: CLSU-00101: operating system error message: No such file or directory
mount.acfs: CLSU-00103: error location: OOF_1
mount.acfs: CLSU-00104: additional error information: open64 (/dev/asm/askmdata-282)
mount.acfs: ACFS-02017: Failed to open volume /dev/asm/askmdata-282. Verify the volume exists.
[root@exaaskmdb01 ~]#

We are not able to mount the ACFS mountpoint. The first approach for any troubleshooting is to analyze the error messages. Cetain times, there could be useful error messages, which can directly help to point to the issue. But sometimes, the error messages may not help to focus on the real issue , rather informational messages. Your experience will count a lot in these scenarios where you should be able to relate the error message to the issue.

Check2:

The error message which caught to our attention is “No such file or directory”. Now our next check will be to check the directories…

On exaaskmdb01 check the directories  /dev/asm/askmdata-282 and /u01/app/askmacfs

[root@exaaskmdb01 ]# ls -lrt /dev/asm/askmdata-282
ls: cannot access /dev/asm/askmdata-282: No such file or directory

[root@exaaskmdb01 ]# ls -ld /u01/app/askmacfs
drwxrwxr-x 11 grid oinstall 4096 May  9 12:00 /u01/app/askmacfs

Same thing, verify on any other node ( 2-8)

[root@exaaskmdb02 ]# ls -lrt /dev/asm/askmdata-282
brwxrwx--- 1 root asmadmin 248, 144385 Sep 12 15:33 /dev/asm/askmdata-282

[root@exaaskmdb02 ]# ls -ld /u01/app/askmacfs
drwxrwxr-x 11 grid oinstall 4096 May  9 12:00 /u01/app/askmacfs

From the above check, it is clear that node1 exaaskmdb01 is missing a directory. Now lets check further how to bring this directory. As these are ACFS mounts, there could be possibility that these are missing due to ACFS drivers not loading properly during node reboot. Let’s concentrate on the ACFS drivers….

Check3:

Check acfs modules to be loaded …

[root@exaaskmdb01 ~]# lsmod | grep oracle
oracleacfs           4825088  0
oracleadvm            806912  3
oracleoks             659456  2 oracleacfs,oracleadvm

This check looks good, the acfs drivers are loaded properly. The next check would be to check the ACFS volume informaiton.

Check4:

ASMCMD> volinfo --all
Diskgroup Name: ASKM_DG

         Volume Name: ASKMDATA
          Volume Device: /dev/asm/askmdata-282
          State: DISABLED                               <<<< Check this. This volume is disabled.
          Size (MB): 524288
          Resize Unit (MB): 64
          Redundancy: MIRROR
          Stripe Columns: 8
          Stripe Width (K): 1024
          Usage: ACFS
          Mountpath: /u01/app/askmacfs

From the above output, we get that the volume is disabled. Now let’s check the same thing on the other nodes exaaskmdb02-08

[grid@exaaskmdb02 ~]$ asmcmd
ASMCMD> volinfo --all
Diskgroup Name: ASKM_DG

         Volume Name: ASKMDATA
          Volume Device: /dev/asm/askmdata-282
          State: ENABLED                     <<< It is enabled on node2
          Size (MB): 524288
          Resize Unit (MB): 64
          Redundancy: MIRROR
          Stripe Columns: 8
          Stripe Width (K): 1024
          Usage: ACFS
          Mountpath: /u01/app/askmacfs

From the above check4, we understood that this is due to the reason that the volume is not enabled by default after patching. Now we need to enable this volume and mount this ACFS manually.

Solution

ASMCMD> volenable -G ASKM_DG -a
ASMCMD> volinfo --all
Diskgroup Name: ASKM_DG

         Volume Name: ASKMDATA
          Volume Device: /dev/asm/askmdata-282
          State: ENABLED
          Size (MB): 524288
          Resize Unit (MB): 64
          Redundancy: MIRROR
          Stripe Columns: 8
          Stripe Width (K): 1024
          Usage: ACFS
          Mountpath: /u01/app/askmacfs

ASMCMD> exit
[grid@exaaskmdb01 ~]$ ls -lrt /dev/asm/askmdata-282
brwxrwx--- 1 root asmadmin 248, 144385 Sep 12 15:56 /dev/asm/askmdata-282

[root@exaaskmdb01 ~]# ls -ld /u01/app/askmacfs
drwxrwxr-x 2 grid oinstall 4096 Mar 18  2016 /u01/app/askmacfs

[root@exaaskmdb01 ~]# mount -t acfs /dev/asm/askmdata-282 /u01/app/askmacfs

[root@exaaskmdb01 ~]# df -h /u01/app/askmacfs
Filesystem            Size  Used Avail Use% Mounted on
/dev/asm/askmdata-282  512G   64G  449G  13% /u01/app/askmacfs
[root@exaaskmdb01 ~]#

That’s it. We fixed the issue.

Questions to the readers

Q1) Where do you find the ASM related modules/drivers on Linux OS ?

Q2) What to do if you find any missing ASM modules/drivers on Linux OS ?

Q3) How to load the asm drivers if they were not loaded by default during node reboot ?

Refer to my other post <Place Holder> which is similar to this issue , but happened during our exa grid upgrade from 12.1.0.2 to 12.2.0.1.

Hope this information helps….