Saturday, June 26, 2010

NetApp - Theory of Aggregate

NetApp's Aggregate is a DataONTAP feature that combines one or more Raid Groups(RG)  into a pool of disk space that can be used to create multiple volumes in different flavors. Newly added disks are assigned to the spare pool and a new aggregate or an existing aggregate that requires more space will then be pulled from the spare pool.


Unlike other Storage arrays where you create LUN from a specific RG, DataONTAP creates volumes within the available space in the aggregate across multiple RG. DataONTAP dynamically stripes the volumes across the aggregates within the Raid Groups. When disks are added to the aggregate they will either go into an existing RG that is not yet full or into a new RG if all the existing groups are full. By default RG are created as RAID-DP(dual parity drives) with stripes of 14+2 for FC/SAS disks and 12+2 for SATA disks. If you want to separate your "Vol0" in a separate aggregate, usually its aggr0 with a minimum of 3 disks for RAID-DP.

Aggregates have snapshots, snap reserve & snap schedule just like any other volumes. Aggregates may either be mirrored or unmirrored. A plex is a physical copy of the WAFL storage within the aggregate. A mirrored aggregate consists of two plexes;(A plex may be online or offline unmirrored aggregates contain a single plex. In order to create a mirrored aggregate, you must have a filer configuration that supports RAID-level mirroring. When mirroring is enabled on the filer, the spare disks are divided into two disk pools. When an aggregate is created, all of the disks in a single plex must come from the same disk pool, and the two plexes of a mirrored aggregate must consist of disks from separate pools, as this maximizes fault isolation.

An aggregate may be online, restricted, or offline. When an aggregate is offline, no read or write access is allowed. When an aggregate is restricted, certain operations are allowed (such as aggregate copy, parity recomputation or RAID reconstruction) but data access is not allowed. 

Creating an aggregate

Aggregate has a finite size of 16TB raw disk with 32 bit DataONTAP. By default, the filer fills up one RAID group with disks before starting another RAID group. Suppose an aggregate currently has one RAID group of 12 disks and its RAID group size is 14. If you add 5 disks to this aggregate, it will have one RAID group with 14 disks and another RAID group with 3 disks. The filer does not evenly distribute disks among RAID groups.

To create an aggregate called aggr1 with 4 disks 0a.10 0a.11 0a.12 0a.13 from a loop on fiber channel port 0a use the following command:

netapp> aggr create -d  0a.10 0a.11 0a.12 0a.13

Expanding an aggregate

To expand an aggregate called aggr1 with 4 new disks use the following command:

netapp> aggr add aggr1 -d  0a.11 0a.12 0a.13 0a.14

Destroying an aggregate

Aggregates can be destroyed but there are restrictions if volumes are bounded to the aggregate. Trying to destroy an aggr with volumes will throw an error but this can be overridden with the -f flag. It is recommended to go over each and every volume and destroying it before using the -f option to avoid any potential data loss. Aggregates can be destroyed much like any volumes but it needs to be taken offline.


To destroy an aggregate aggr1 use the following command:

netapp> aggr offline aggr1
netapp> aggr destroy aggr1

Friday, June 25, 2010

DataONTAP Logical View - LUNs, Deduplication, and Thin Provisioning

 This is one of the coolest storage model for NetApp that explains the placing of disks, Raid Groups, Aggregates, Flex Volumes, NAS and SAN. Thanks to Dr. Dedupe for sharing this piece of information. 

Sunday, June 20, 2010

NetApp Fractional Reserve

When we snapshot the filesystem, we take a copy of the file allocation tables and this locks the data blocks in place. Any new or changed data is written to a new location, and the changed blocks are preserved on disk.

This means that any new/changed blocks (D1 ) in the active file-system are written to a different location - this overwrite space is called as fractional reserve.

Fractional reserve ensures that there is space available for overwrites and the default fractional reserve space is 100% but this can be changed and generally set to "0" value.

So the reason a LUN will be switched offline if the fractional reservation area is set to 0, is that the filer needs to protect the existing data that is locked between the active file system and the most recent snapshot, plus any additional changes that happen to the active file system. If the volume / LUN / frac-res and snap reserve are full, then this space is not available and the filer needs to take action to prevent these writes from failing. The filers guarantee no data loss, but with no space free and no-where to write the new data, it has to offline the lun to prevent the writes from failing.

So fractional reservation is in constant use by the filer as an over-write area for the LUN. Without it, you need to make sure that sufficient space is free to allow for the maximum rate of change you would expect. Defaults are good, but trimming down on these you need to monitor the rate of change and make sure the worst case scenario is within a buffer of free space that you allow.

Pocket Survival Guide NetApp

Setting Up a Filer:
1. Check version and licenses
(telnet) license to list
(telnet) license add (key) to add a license
(FilerView) Filer -> Manage Licenses
2. Setup Network Interfaces (set up single-mode and multi-mode vifs or ifgrps)
NOTE: You can't modiy a vif once it is created. I usually have to delete the vif and start over to modify it. This also means network service will go down on that port group!
(telent) ifconfig -a to list the interfaces
(telnet) ifconfig command to manage the interfaces
(telent) To manage vifs in ONTAP 7: vif command, in ONTAP 8: ifgrp
(telnet) You can also run setup, will probably require a reboot!
(FilerView) Network -> Manage Interfaces -> i.e. vif1a or vif1b
3. Enable SSH on the filer
(telnet) secureadmin setup ssh
(FilerView) Secure Admin ->SSH -> Configure -> Generate Keys -> OK -> Apply
4. Set Snap reserve on Aggr0 if not MetroCluster
The default for this is 5%. I have seen some set this to 0% if not using MetroCluster. I set it to 3% because I have seen issues if set to 0% and NetApp support likes at least a little reserve at this level for disaster recovery
(telnet) snap reserve -A aggr0 3
(FilerView) I don't think this is possible from FilerView
5. Resize Vol0
(telnet) vol size vol0 15g for 2000 series (20gb on 3000 series)
(telnet) vol size vol0 to check
(FilerView) Volumes -> Manage -> Click on vol0
Click Resize Storage
Click Next -> Click Next -> Enter New Volume Size -> Next -> Commit -> Close
6. Iscsi & FCP status
(telnet) iscsi status to check
(telnet) iscsi start to enable
(telnet) same commands for fcp
(FilerView) LUNs -> Enable/Disable ->Check Enable Box
7. NTP Setup
(telnet) to list all time options: options timed
(telnet) options timed.servers 0.us.pool.ntp.org,1.us.pool.ntp.org,2.us.pool.ntp.org (or enter your time servers for your site)
(telnet) options timed.proto ntp (if not already set to ntp)
(telent) options timed.log on (if you would like the updates to go to console and log)
(telnet) options timed.enable on
(FilerView) Filer -> Set Date/Time -> Modify Date/Time
Choose Time Zone -> Click Next
Choose Automatic -> Click Next
Change Protocol to ntp -> Click Next
Enter ntp servers 0.us.pool.ntp.org (1&2) - or any other ntp server you have
Click Commit -> Click Close
8. Enable and Test Autosupport
(telnet) to list all options: options autosupport
(telnet) options autosupport.from (userid) - sets the userid autosupport is sent from
(telnet) options autosupport.mailhost (host ip or name)
(telnet) options autosupport.to (user1,user1)
(telnet) options autosupport.enable on
(telnet) options.autosupport.doit test - generates a test autosupport
(FilerView) Filer -> Configure Autosupport
Change Autosupport enabled to Yes
Enter mailhosts
Enter To: fields -> Click Apply
Click Test Autosupport on the left -> Click Send Mail
9. Check Cluster Failover
(telnet) cf status – Check Cluster Failover Status
(telnet) cf partner - Lists the partner head
(telent) cf monitor – more details on failover
(telnet) cf takeover - This will reboot the host you are taking over!
(telnet) cf giveback - when partner is ready to receive back
(FilerView) Cluster - Click the buttons! :)
10. Disk Commands (Assign Drives to Controller)
Typically I would assign the odd disks to the first controller and the evens to the second controller. I have also assigned the majority of drives to one controller and a minimum (3-4 drives) to the other controller to maximize capacity
(telnet) disk show –n - show disks NOT owned
(telnet) disk show –o (controller name) - show disks owned by a controller
(telnet) disk show –v - show all disks owned and not owned
(telnet) disk assign (drive#)
(telnet) disk remove_ownership (drive#) (This is a priv set advanced command!)
(telent) disk zero spares
(FilerView) - I don't think this is possible from FilerView
11. Aggregate Commands (Assign Drives to Aggregate)

In RAID-DP environments, if there are 2 hot spares, I often add one back in. You already have 2 drive failure with RAID-DP and this way you have one more drive capacity
(telnet) aggr status -v - shows status with volume information
(telnet) aggr status –r - shows which drives are in the aggr already including raid groups
(telnet) aggr options (aggr) - shows all options for the aggr (raid groups, raid type, etc.)
(telnet) aggr show_space -m OR -g
(telnet) aggr add (aggr) –d (drive#)
(FilerView) Aggregates -> Click on the aggregate
Click Show Raid to see existing disk configuration
Click Add Disks to add new disks and follow the wizard
12. Volume Commands (Create Volumes From Aggregates)

(telnet) vol status - shows all volumes and options
(telnet) vol create (volumename) -s none (aggr) (size) - create volume with no guarantee
(telnet) vol create (volumename) -s volume (aggr) (size) - create volume with volume guarantee
(telnet) vol size (volume) - check the volume size
(telnet) vol size (volume) (size) - set volume to new size
(telnet) vol autosize (volume) on - turn on volume autogrow, the volume will grow by default in 5% increments until 20% growth achieved by default
(telnet) vol autosize volume -m (new max size) -i (new increment amount) - notice that even though the defaults are 20% max and 5% increments, if you change the value the new values have to be in space (MB or GB)!
(telnet) vol autosize (volume) - reports the current setting
(telnet) vol options (volume) - list the volume options
(telnet) vol options (volume) guarantee volume OR none - sets the volume guarantee to volume or none
(telnet) vol options (volume) fractional_reserve (value) - sets fractional reserve to (value) but only works if volume guarantee is volume.
(telnet) vol options (volume) try_first snap_delete OR volume_grow - sets the value to which method a volume will use when it runs out of space. It will either grow the volume or delete snap shots. The defaults is to grow the volume.
(FilerView) Too much here to list. Just click on volumes and poke around. The following aren't possible in FilerView that I can tell: volume autosize, volume try_first option, changing fractional reserve
13. Check and Modify Snapshot Settings as Needed per Volume

NOTE: Disable the snapshot schedule on the volume if SnapManager is protecting the volume
(telnet) vol options (volume) nosnap on - Turn off the SnapShot Schedule but leave the schedule in place. I do this to disable snapshots in favor of the following command
(telnet) snap sched (volume) 0 0 0 - Disable SnapShots by modifying the schedule. I like the first option better.
(telnet) snap reserve (volume) (#) - Sets the snap reserve to #% of the volume
(telnet) snap autodelete (volume) - shows the snapshot autodelete settings
(telnet) snap autodelete (volume) on - turns on snapshot autodelete (Check the settings! Too many default settings to list here!)
(FilerView) Too much here to list. Just click on Volumes -> Snapshots and poke around. The snap autodelete option is not possible in FilerView that I can tell.
14. Set DeDupe Settings per Volume
DELETE EXISTING SNAPSHOTS FIRST ON THE TARGET VOLUME!
Note: For the sis command you need the full volume path, /vol/vol0 for instance
(telnet) sis on (full volume path) - enables dedupe on that volume
(telnet) sis start (full volume path) - runs dedupe on that volume now
(telnet) df –sh - To check space saving on DeDupe
(telnet) sis config and sis status to check settings
(telnet) sis config -s sun,mon-fri,sat@0,3-9,12 (full volume path) - The -s option allows you to change the schedule. You can list the days separated by commas or a range by dash, same with hours.
(FilerView) This is not possible in FilerView
15. Cifs setup (set up ntp first!)
(telnet) cifs setup
OR (Filer View) CIFS -> Configure -> Setup Wizard
16. To Set-up a CIFS Share (Add Volume First!)
(Filer View) CIFS -> Shares -> Add
(Filer View) CIFS -> Shares -> Manage -> Change Access
17. NFS Setup (Add Volume First!)
Need to Add NFS Export Permissions
18. LUN Setup (Add Volume First!)
Need more information here
(telnet) lun setup - wizard to create the lun and map the igroup all in one
(telnet) lun set reservation (full path to lun) enable OR disable - disable space reservation to thin provision the LUN
(Filer View) LUNS -> Add (Name Lun with aggr.LUN file convention)
Add an igroup and assign initiators
LUNS -> Manage -> No Maps
Other Common Information Commands That I Still Need to Add and Document:
Sysconfig

snap reclaimable
snap delta
aggr show_space (aggr) -g
shelfchk
Wrfile /etc/hosts
Rdfile /etc/hosts
Rdfile /etc/rc
rdfile /etc/messages

Priv set advanced
Led_on (drive#) (get nmber from disk show) (need priv set advanced first)
(priv set) disk shm_stats
(need to add deswizzle status command)
sysstat
statit (disk utilization)
stats (performance data)
reallocate measure or
reallocate start
to see raid groups: sysconfig -r OR aggr status -R
fcstat device_map - command to show shelf layout
sysconfig -d
interface alias process

storage show
vif favor