NetApp volume has the fixed filesystem size option set

The following error is reported when trying to alter the size of a destination flex vol that is in a SnapMirror relation, or that was in a SnapMirror relation.

I most often run into this on volumes that were migrated off older hardware and no longer in a SnapMirror relation, but the fs_size_fixed option is not automatically changed after relationship is broken.

fs_size_fixed

This option causes the file system to remain the same size and not grow or shrink when a SnapMirrored volume relationship is broken, or when a volume add is performed on it. It is automatically set to true when a volume becomes a SnapMirrored volume. It stays set to true after the snapmirror break command is issued for the volume. This allows a volume to be SnapMirrored back to the source without needing to add disks to the source volume. If the volume is a traditional volume and the size is larger than the file system size, setting this option to false forces the file system to grow to the size of the volume. If the volume is a flexible volume and the volume size is larger than the file system size, setting this option to false forces the volume size to equal the file system size.

To change the option in Data ONTAP 7-Mode:
vol options volumename fs_size_fixed off  
To change the option in Clustered Data ONTAP:
volume modify -filesys-size-fixed false -volume volumename  
To change the option using Data ONTAP PowerShell Toolkit, 7-mode:
Get-NaVol volumename | Set-NaVolOption -key fs_size_fixed -value off  
To change the option using Data ONTAP PowerShell Toolkit, Clustered Data ONTAP:
Get-NcVol volumename | Set-NcVolOption -key fs_size_fixed -value off  

Repair corrupt 3PAR System Reporter MySQL DB

My 3PAR System Reporter server stopped sampling. I checked the sampleloop.log and saw MySQL error:

MySQL DB error executing show full columns from statvlun_hires_4: mysqlsel/db server: Table './inservstats/statvlun_hires_4' is marked as crashed and should be repaired  

Here’s what I did to fix:

$ mysqlcheck --all-databases -uroot -p

That identified 2 tables with issues:

inservstats.statpd_hires_4  
warning : 1 client is using or hasn't closed the table properly  
error : Record at pos: 119879784 is not remove-marked  
error : record delete-link-chain corrupted  
error : Corrupt

inservstats.statvlun_hires_4  
warning : Table is marked as crashed  
warning : 1 client is using or hasn't closed the table properly  
error : Record at pos: 702342512 is not remove-marked  
error : record delete-link-chain corrupted  
error : Corrupt  

The others came back OK.

Log in to DB (in my case “inservstats”):

$ mysql -uroot -p inservstats

And then repair table(s):

mysql> REPAIR TABLE statpd_hires_4;  
+----------------------------+--------+----------+----------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+----------------------------+--------+----------+----------------------------------------------+
| inservstats.statpd_hires_4 | repair | warning | Number of rows changed from 242760 to 243404 |
| inservstats.statpd_hires_4 | repair | status | OK |
+----------------------------+--------+----------+----------------------------------------------+
2 rows in set (4 min 50.02 sec)  
mysql>

mysql> REPAIR TABLE statvlun_hires_4;  
+------------------------------+--------+----------+------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+------------------------------+--------+----------+------------------------------------------------+
| inservstats.statvlun_hires_4 | repair | warning | Number of rows changed from 2679897 to 2686814 |
| inservstats.statvlun_hires_4 | repair | status | OK |
+------------------------------+--------+----------+------------------------------------------------+
2 rows in set (19 min 13.59 sec)  
mysql>  

After that I was able to get 3Par System Reporter sampling again. (Confirm by doing a “cat /var/log/sampleloop/sampleloop.log”)

Install 3PAR System Reporter on CentOS 6 64-bit

My weekend project was rebuilding our HP 3PAR System Reporter installation. I used the HP 3PAR System Reporter 3. 1 MU1 Software User’s Guide to work off of. Section 2 contains the build instructions and I do encourage you to read it, but it’s a little unclear in parts so I made some notes of my own build which I’ll share below; and between these two you should have success.

For new installation I used a CentOS 6.4 x86-64 VM. It’s running the System Reporter sampler and Web server 3.1 components, 3PAR CLI 3.1.2, Apache 2.2.15, and MySQL 5.1.

HP notes “when running System Reporter on a VM it’s best to have the database server running on a separate VM, especially for Microsoft SQL, MySQL or Oracle.” I haven’t particularly found performance to be an issue myself but I assume that would depend on the number of storage arrays we were reporting against.

I started with a minimal CentOS 6.4 64-bit base install. After install, the first thing I needed to do was modify “/etc/sysconfig/network-scripts/ifcfg-eth0” which had onboot=no, so I changed to yes.

Then once you have a valid IP:

yum -y update  
yum -y upgrade  

Since I am running this in a VMware environment. I installed vmware tools. Note: you need to install vmware tools in the vmware console. If you install via ssh remoting, vmware tools does not automatically start at boot.

yum install make gcc kernel-devel kernel-headers glibc-headers perl  
mkdir /mnt/cdrom  
mount /dev/cdrom /mnt/cdrom  
cp /mnt/cdrom/VMwareTools*.tar.gz /tmp/  
cd /tmp/  
tar xvfz VMwareTools*.tar.gz  
cd /tmp/vmware-tools-distrib  
./vmware-install.pl

To use the vmxnet driver:

/etc/init.d/network stop
rmmod pcnet32  
rmmod vmxnet  
modprobe vmxnet  
/etc/init.d/network start

Next allow ssh, http, and https through firewall:

vi /etc/sysconfig/iptables  

Add the following entries before “-A INPUT -j REJECT –reject-with icmp-host-prohibited”:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 25 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT

Add the following entries after “-A INPUT -j REJECT –reject-with icmp-host-prohibited”:

-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 25 -j ACCEPT
-A INPUT -m tcp -p tcp --dport 80 -j ACCEPT
-A INPUT -m tcp -p tcp --dport 443 -j ACCEPT

Restart iptables:

service iptables restart  

Now you can continue via ssh session. Login as root and install apache:

yum install httpd  
chkconfig httpd on  
service httpd start  

Install 3PAR CLI. This is included in the e-media kit download or System Reporter installation CD. I used WinSCP to copy it over to my server. From the location where you’ve copied the Linux contents:

chmod +x setup.bin  
./setup.bin

Note: make sure you document the location of the 3PAR CLI install. In my case, 3.1.2 defaulted to “/opt/3PAR/inform_cli_3.1.2/bin/cli”

Install MySQL:

yum install mysql-server  
service mysqld start  
chkconfig --level 345 mysqld on  
/usr/bin/mysql_secure_installation

You’ll be prompted for your current root password. Since you just installed MySQL, leave it blank and hit [enter]. Then the prompt will ask you if you want to set up a root password. Enter a root password at this time and continue with prompts accepting all defaults.

When installing on a CentOS server where MySQL server is running locally on the same server, you must complete the following procedures:

  • Create a symbolic link using the following command (I added this to “/etc/init.d/mysqld” to have it execute at boot):
ln -s /var/lib/mysql/mysql.sock /var/run/mysqld/mysqld.sock  
  • Change the “max_allowed_packet” MySQL parameter by editing its value in the MySQL initialization file located in “/etc/my.cnf”:
vi /etc/my.cnf  

Add the following to “/etc/my.cnf”:

max_allowed_packet=32M  

Restart MySQL:

service mysqld restart  

Create MySQL Users and Schema:

mysql -u root -p  
create database inservstats;  
create user cliuser identified by 'cliuserpassword';  
create user webuser identified by 'webuserpassword';  
use inservstats grant all on * to cliuser;  
grant select on * to webuser;  
exit;  

Install 3PAR System Reporter Tools for x86_64. System Reporter tools on Linux x86_64 consists of two RPM packages available in the RHEL6_U1-U2x86_64 folder in the e-media kit or CD:

  • sampleloop (sampleloop-3.1-3.x86_64.rpm) consists of the files needed to install the sampleloop daemon.
  • sysrptwebsrv (sysrptwebsrv-3.1-3.x86_64.rpm) consists of the files needed to install the System Reporter web server code.

NOTE: The sysrptwebsrv requires the 32-bit version of the gd (version 2.0 or greater) package to be installed. The 32-bit version available in CentOS 6.4 is: gd.i686 2.0.35-11.el6.

Install gd.i686:

yum install gd.i686  

Using WinSCP or other method, copy sampleloop-3.1-3.x86_64.rpm to server, then install from that location:

yum install sampleloop-3.1-3.x86_64.rpm  

Edit /etc/sampleloop.conf:

vi /etc/sampleloop.conf  

Set the full path to cli, (this was “/opt/3PAR/inform_cli_3.1.2/bin/cli” from earlier):

set Sysdb::cli "/opt/3PAR/inform_cli_3.1.2/bin/cli"  

Set the IP address or name of the smtp server

set Sysdb::smtpserver "yoursmtpserver.com"  

Set the originator’s email address

set Sysdb::smtporig "sysreporter@yourdomain.com"  

Set the username and password for the smtp server (if you do not need a username or password, these can remarked out as I have below):

#set Sysdb::smtpuser "username"
#set Sysdb::smtppasswd "password"

For mysql: Set the IP address or name of the database server

set Sysdb::dbhost nameofyourserver  

For MySQL databases, create the “/etc/sampleloop_dbpwfile” file to include a single line that has the database user name and password separated by a space.

vi /etc/sampleloop_dbpwfile  

This should look like the example below:

cliuser cliuserpassword  

Start the sampleloop daemon by running:

/etc/init.d/sampleloop start

Add sampleloop to the execute at startup:

chkconfig -add sampleloop  

Using WinSCP or other method, copy sysrptwebsrv-3.1-3.x86_64.rpm to server, then install from copy location:

yum install sysrptwebsrv-3.1-3.x86_64.rpm  

Edit “/var/www/cgi-bin/3par-rpts/config.tcl”:

vi /var/www/cgi-bin/3par-rpts/config.tcl  

Set the full path to cli, (this was “/opt/3PAR/inform_cli_3.1.2/bin/cli” from earlier):

set ::cli "/opt/3PAR/inform_cli_3.1.2/bin/cli"  

For mysql: Set the IP address or name of the database server

set Sysdb::dbhost nameofyourserver  

Set the database user name and password.
The user need only have SELECT privileges for the schema

set Sysdb::dbuser webuser  
set Sysdb::dbpasswd webuserpassword  

Set the IP address or name of the smtp server

set Sysdb::smtpserver "yoursmtpserver.com"  

Set the originator’s email address

set Sysdb::smtporig "sysreporter@yourdomain.com"  

Set the username and password for the smtp server (if you do not need a username or password, these can remarked out as I have below):

#set Sysdb::smtpuser "username"
#set Sysdb::smtppasswd "password"

Edit “/var/www/cgi-bin/3par-policy/config.tcl”:

vi /var/www/cgi-bin/3par-policy/config.tcl  

Set the full path to cli, (this was “/opt/3PAR/inform_cli_3.1.2/bin/cli” from earlier):

set ::cli "/opt/3PAR/inform_cli_3.1.2/bin/cli"  

For mysql: Set the IP address or name of the database server

set Sysdb::dbhost nameofyourserver  

Set the database user name and password
The user should be the same as used for the the sampleloop daemon which has all the privileges for the schema.

set Sysdb::dbuser cliuser  
set Sysdb::dbpasswd cliuserpassword  

Disable SELinux:

sestatus | grep -i mode  
setenforce 0  

Edit “/etc/sysconfig/selinux” to disable upon boot:

vi /etc/sysconfig/selinux  

Change SELINUX to “disabled”:

SELINUX=disabled  

If the configuration files are correct , you should be able to open the webpage: http://hostname/3par.

At this point you are functionally finished but an additional step you should consider is configuring Apache HTTP Server to require a username and password before allowing a user to:

  • read the HP 3PAR System Reporter database or view the database sampling policies.
  • make changes to database sampling policies.

This is outlined in the System Reporter User’s Guide in the section “Apache HTTP Server Authorization and Access Control”.

I chose to only password protect the modifications to the policy settings. To do that you must first create a password file.

mkdir /etc/htpasswd/  
htpasswd -c /etc/htpasswd/.htpasswd yourusername  
chmod ug+rw /etc/htpasswd/.htpasswd  
chown apache:apache /etc/htpasswd/.htpasswd  

At this point you can add Directory directives to the httpd.conf file:

vi /etc/httpd/conf/httpd.conf  

Add the following:

<Directory "/var/www/cgi-bin/3par-policy">
AuthType Basic  
AuthName "3PAR System Reporter Policy Update"  
AuthUserFile /etc/htpasswd/.htpasswd  
Require user yourusername  
</Directory>

Then restart httpd and the next time you try to make policy changes, you’ll be prompted for the username and password you’ve specified. (Consult the User’s Guide if you’re interested in password protecting the reading of the database and the display of policy settings).

service httpd restart  

Lastly, you should observe best practices & create a user account for yourself and disable root ssh login.

Replace a Cache Controller Battery on an IBM DS4300

I’ve had to replace many of these and honestly – if I could pass along any single tip it would be this one – please check the firmware of your Hitachi FC drives first.

If you read enough IBM release notes, this text will be familiar to you:

Customer with IBM branded Hitachi FC drives should verify that these drives are at drive firmware FQ8, or later, before doing a controller firmware or ESM firmware download. It has been observed by IBM Support that customer’s with older versions of the Hitachi FC drive firmware have experienced failure of these Hitachi drives during controller firmware and ESM firmware downloads. Upgrading the drive firmware to FQ8 will prevent this problem.

The way this bug was described to me was – if you have Hitachi drives with older firmware than FQ8 – as soon as you reseat your controller – all of your Hitachi drives will show up as failed. I would not wish this on anyone. So. Check your firmware. And upgrade if necessary. Sometimes IBM will check this for you, but ultimately you as the customer are responsible.

That out of the way, you should have the replacement battery on-site. And we’re ready to begin.

First you’ll need to manually fail the controller. This is done through the Storage Manager Client. You’ll select the controller. Then select Advanced > Recovery > Place Controller > Offline.

In this example, failing Controller B will fail over all of its associated array groups to Controller A. You’re counting on MPIO/RDAC here to be successful on the host side, so definitely something to check and consider in a non-disruptive maintenance. Once Controller B is failed. It can be removed.

When removing the fiber cables, keep them identified so that you have some way of reseating them in the correct order later on. Whichever is your preference. I usually label them prior to removal.


To remove, push down the black latch and pull on the levers. Controller slides right out.


To open the Controller, place it lever-side-down and use a Philips head screwdriver to remove the screws holding the bottom panel to the sides of the Controller. Three on each side. The bottom panel then slides out and can be tilted up and away.


The final screw on the battery access panel will allow access to the failed battery.


You must connect the new battery harness to the two-pin connector on the Controller board. The connector is keyed so that there is only one way to attach the harness to the connector. Reattach the battery access panel.


Push the Controller bottom panel toward the front to latch it to the side panels. Reinsert the six screws.


Insert the Controller back into its slot. The levers will catch. Push to lock them back in place.


Controller B will start charging its new battery as soon as its inserted. Reconnect the fiber and network exactly as it was before. I usually wait a minute before bringing the Controller back online. And that is done by going through the Storage Manager Client. You’ll select the controller. Then select Advanced > Recovery > Place Controller > Online.


The last thing you need to do is reset the battery age back to zero. That’s done by clicking on the “View” button and selecting “Batteries”. Click “Reset” and answer “Yes” when prompted.

The array should now be in an optimal state.

Unlock a shared file on NetApp CIFS

We had a ticket come in regarding an Excel file which was complaining that it was “locked for editing” by a user whenever anyone (including that user) tried to open it. There were no temp Excel files in the directory container and user had already rebooted their workstation and cleared out temp windows cache.

First thing I did was open Computer Management and connect to the filer. I looked at the list of open files and didn’t see the file listed as being open.

Next thing I did was ssh into the filer and issue a lock status command:

lock status -f "/vol/Share/folder/file.xlsx" -p cifs  

This produced results back. And indeed, it was indicating the user had a lock on the file.

To unlock:

Data ONTAP 7-mode:
lock break -f "/vol/Share/folder/file.xlsx" -p cifs  
Clustered Data ONTAP:
::> vserver locks show

Notice: Using this command can impact system performance. It is recommended that you specify both the vserver and the volume when issuing this command to minimize the scope of the command’s operation. To abort the command, press Ctrl-C.

Change to advanced mode:

::> set advanced

Perform a break for that client:

::*> vserver locks break -vserver vservername -volume volname -lif lifname -path pathtofile

Warning: Breaking file locks can cause applications to become unsynchronized and might lead to data corruption. If you are breaking a file lock on a volume that is being accessed by a FlexCache you must take the volume offline on the FlexCache to reestablish proper delegation synchronization between the origin and the cache.