Once we deliver the pNFS server, you really, really would not want to do what I am about to
show you. But, I’ve got orphaned data on my DSes and I need to clean them up. Before my
latest server changes enabled on disk layouts, you only had to make changes on the DS, but
now you will also have to nuke some state on the MDS.

First we want to disable any traffic from the clients. I could do that by unmounting
all shares or rebooting them.

Second, we want to reboot the MDS - that will cause all state to either be flushed to
disk or nuke it out of memory.

Third, on each DS, I want to do:

1. Determine the datasets:

   [root@pnfs-17-22 pnfs-17-22]> dservadm liststor
   storage:
       pnfs1/ds1
       pnfs2/ds2
       pnfs1/ds8
   

2. Disable the datasets:

   [root@pnfs-17-22 pnfs-17-22]> dservadm dropstor pnfs1/ds1
   [root@pnfs-17-22 pnfs-17-22]> dservadm dropstor pnfs2/ds2
   [root@pnfs-17-22 pnfs-17-22]> dservadm dropstor pnfs1/ds8
   

3. Get rid of the dangling crap beneath each:

   [root@pnfs-17-22 pnfs-17-22]> zfs list
   NAME                         USED  AVAIL  REFER  MOUNTPOINT
   pnfs1                       15.6G      0    23K  /pnfs1
   pnfs1/ds1                   7.89G      0    15K  -
   pnfs1/ds1/DB41524C083AB6A2  3.49G      0  3.49G  -
   pnfs1/ds1/EF25FBCF08FA1BD7  4.39G      0  4.39G  -
   pnfs1/ds8                   7.74G      0    15K  -
   pnfs1/ds8/DB41524C083AB6A2  3.48G      0  3.48G  -
   pnfs1/ds8/EF25FBCF08FA1BD7  4.26G      0  4.26G  -
   pnfs1/scratch                 21K      0    21K  /pnfs1/scratch
   pnfs2                       7.89G  12.0G    22K  /pnfs2
   pnfs2/ds2                   7.89G  12.0G    15K  -
   pnfs2/ds2/DB41524C083AB6A2  3.59G  12.0G  3.59G  -
   pnfs2/ds2/EF25FBCF08FA1BD7  4.30G  12.0G  4.30G  -
   pnfs2/scratch                 21K  12.0G    21K  /pnfs2/scratch
   

   We are looking for unamed filesystems under our datasets.

   [root@pnfs-17-22 pnfs-17-22]> zfs destroy pnfs1/ds1/DB41524C083AB6A2
   ...
   [root@pnfs-17-22 pnfs-17-22]> zfs destory pnfs2/ds2/EF25FBCF08FA1BD7
   

   Note, if you get the following:

   [root@pnfs-17-23 ~]> zfs destroy pnfs1/ds3/DB41524C083AB6A2
   cannot destroy 'pnfs1/ds3/DB41524C083AB6A2': dataset is busy
   

   Then you need to first reboot the DS!

4. Do not re-enable the storage yet or reboot. This should be done
   on all dataservers first!

Once all DSes have been scrubbed, it is time to scrub the MDS. We
need to remove all of the layouts which refer to files which are
no longer there!

[root@pnfs-17-24 layouts]> cd /var/nfs/v4_state/layouts/
[root@pnfs-17-24 layouts]> ls -la
total 16
drwxr-xr-x   5 daemon   daemon       512 Jul 29 03:17 .
drwxr-xr-x   5 daemon   daemon       512 Jul 23 16:30 ..
drwxr-xr-x   2 daemon   daemon      2560 Jul 31 01:27 2d90003
drwxr-xr-x   2 daemon   daemon      1536 Jul 28 20:43 2d90005
drwxr-xr-x   2 daemon   daemon      1024 Jul 31 01:27 2d90006
[root@pnfs-17-24 layouts]> rm -rf 2d*

At this point, you want to reboot the MDS. Why? Why not!

And then you want to re-enable the datasets on the DSes:

[root@pnfs-17-22 pnfs-17-22]> dservadm addstor pnfs1/ds1
[root@pnfs-17-22 pnfs-17-22]> dservadm addstor pnfs2/ds2
[root@pnfs-17-22 pnfs-17-22]> dservadm addstor pnfs1/ds8

And then, you guessed it, reboot the DSes. Actually this
one serves a purpose - I don’t think the DS understands to
send a DS_REPORTAVAIL when new storage is added. So this
will force a clean state!

Source/Kaynak : http://blogs.sun.com/tdh/entry/cleaning_up_stale_ds_data