Disable SSLv2 and SSLv3 in Data ONTAP 7-mode for CVE-2016-0800 and CVE-2014-3566

NetApp KB1015015 provides information and procedures for disabling SSLv2 and SSLv3 in Data ONTAP operating in 7-Mode and clustered Data ONTAP versions 8.1 though 8.3 for CVE-2016-0800 and CVE-2014-3566.

The procedure is 2-steps: (1) enable tls (disabled by default and must be enabled prior to disabling SSL) and (2) disable SSLv2 and v3.

The following simple PowerShell script will automate performing this procedure on multiple number of 7-mode systems.

It relies on either either specifying filername or providing a .csv list of filernames it can authenticate against.

.CSV file should be formatted as:

#requires -Version 2 -Modules dataontap
<#  
        .SYNOPSIS           
        Simple script which automates disabling SSLv2 and SSLv3 in Data ONTAP 7-Mode for CVE-2016-0800 and CVE-2014-3566.

        .DESCRIPTION
        Uses Set-NaOption to enable tls and disable SSLv2 and v3.

        .PARAMETER filer
        Specifies the name of the NetApp filer. Optional.

        .NOTES
        (1) Script will prompt for credentials. Uses same cred for multiple filers.
        (2) If no parameter is specified it will prompt for .csv list of filers. 
        .CSV should be formatted as:
        name
        filer1
        filer2

        .EXAMPLE
        C:PS> netapp-disable-ssl-7mode.pst 

        .EXAMPLE
        C:PS> netapp-disable-ssl-7mode.pst filer1

        Author: David Maldonado
        Date: 09/01/2016
        Version: 1.0 - Initial Script - for 7mode
#>

param( [string[]] $filerinput)  
If ($filerinput -eq $NULL)  
{ 
    function Get-FileName($initialDirectory)
    {   
        $NULL = [System.Reflection.Assembly]::LoadWithPartialName('System.windows.forms')

        $OpenFileDialog = New-Object -TypeName System.Windows.Forms.OpenFileDialog
        $OpenFileDialog.initialDirectory = $initialDirectory
        $OpenFileDialog.filter = 'All files (*.*)| *.*'
        $NULL = $OpenFileDialog.ShowDialog()
        $OpenFileDialog.filename
    } 
    Write-Host -Object 'No controller specified, please provide source .csv file.' -BackgroundColor Yellow -ForegroundColor Blue 
    $filers = Import-Csv (Get-FileName -initialDirectory 'c:') 
}
Else  
{
    $filers = $filerinput 

    $filerresults = @() 
    $filerhash = foreach ($filer in $filers)
    {
        $filerresult  = New-Object -TypeName PSObject
        $filerresult  | Add-Member -MemberType NoteProperty -Name 'name' -Value $filer
        $filerresults += $filerresult
    }

    $filers = $filerresults | Select-Object -Property *
}

Import-Module -Name DataONTAP  
$mycreds = (Get-Credential)
function Disable-7MSSL  
{
    Connect-NaController -Name $filer.name -Credential $mycreds

    Set-NaOption -OptionName tls.enable -OptionValue on
    if (((Get-NaOption -OptionNames tls.enable).value) -eq 'on') 
    {
        Set-NaOption -OptionName ssl.v2.enable -OptionValue off
        Set-NaOption -OptionName ssl.v3.enable -OptionValue off
    }
}

Foreach ($filer in $filers)  
{
    Disable-7MSSL
}

The mount request was denied by the NFS server | VMware | Troubleshooting netgroups

Call “HostDatastoreSystem.CreateNasDatastore” for object “datastoreSystem-663108” on vCenter Server failed.
NFS mount failed: The mount request was denied by the NFS server. Check that the export exists and that the client is permitted to mount it.
An unknown error has occurred.

VMware message encountered while trying to mount volume from newly added nodes 13 and 14 in a 14 node cluster running 8.3.2P2.

Troubleshooting steps:

  • Checked namespace export policy – looked good; identical to dozens of other volumes backed by node1 thru node12 already mounted as VMware datastores.
  • Tested mounting on different vSphere hosts – same message.
  • As the export policy uses local netgroups, I checked the local netgroups definitions file:
vserver services name-service netgroup file show  
  • File looked fine.
  • Checked the status of netgroups definitions across all nodes in the cluster – found the culprit:
::> set -privilege advanced

Warning: These advanced commands are potentially dangerous; use them only when directed to do so by NetApp personnel.  
Do you want to continue? {y|n}: y

::*> vserver services name-service netgroup status
Vserver   Node            Load Time           Hash Value  
--------- --------------- ------------------- -------------------
vs01  
          node13 
                          -                   -

          node14 
                          -                   -

vserver services name-service netgroup status enables you to verify that netgroup definitions are consistent across all nodes that back a SVM into which netgroup definitions have been loaded.

The command displays the following information:

  • SVM name
  • Node name
  • Load time for netgroup definitions
  • Hash value of the netgroup definitions

Node13 and node14 contained no load time or hash value, as this is a manual step, and doesn’t happen automatically when a node is added. No worries. To resolve, I reloaded the netgroup file onto the SVMs:

::> vserver services name-service netgroup load -vserver vservername -source http://ipaddress/netgroup.txt

This loaded the netgroup file onto node13 and node14, which took about 5 minutes to build the netgroup.byhost map. (I tried mounting on vSphere host immediately after netgroup load, and got the same error message. Waited 5 minutes, and it mounted fine.)

More on displaying the local netgroup definitions status here:
https://library.netapp.com/ecmdocs/ECMP12452955/html/vserver/services/name-service/netgroup/status.html

System Alert from SP of (REBOOT (watchdog reset)) CRITICAL | NetApp

Ran into this recently. Was upgrading SP in older FAS3270 HA pair running 7.3.6. I did software install, then sp upgrade. That’s when Cluster Takeover occured:

Mon Aug 22 08:59:47 MDT [cf.fsm.firmwareStatus:info]: Cluster monitor: partner dumping core  
Mon Aug 22 08:59:49 MDT [cf.ic.xferTimedOut:error]: wafl interconnect transfer timed out  
Mon Aug 22 08:59:50 MDT [cf.ic.xferTimedOut:error]: ofw interconnect transfer timed out  
Mon Aug 22 08:59:50 MDT [netif.linkDown:info]: Ethernet c0a: Link down, check cable.  

I have seen this before, and after checking the SP I did confirm it was indeed over 500 days uptime.

SP> sp uptime  
 15:02:17 up 759 days, 23:14, load average: 1.26, 1.27, 1.12

To fix this, I did a giveback, rebooted both SPs, and then could perform SP Upgrade successfully. You are prompted to reboot the SP after a successful upgrade, and I know I’ve been able to do this in the past without issue, but lesson learned – it’s better to perform SP reboot prior, than risk a takeover.

More from the KB article:

Service Processor Can Trigger Watchdog Reset After 500+ Days of Uptime:
Summary

The Service Processor (SP) contains a memory resource leak affecting the IPMI software, eventually causing the SP to become unresponsive and possibly triggering a watchdog reset of Data ONTAP®. In order to trigger this reset, the SP must be running for approximately five hundred (500) days or longer.

Symptom

The SP reports the following errors regularly when it is close to failing on the Data ONTAP console:

[hostname: statd: sp.network.link.down:warning]: Service Processor (SP) network port link down due to cable or network errors.
[hostname: statd: sp.network.link.down:warning]: Service Processor (SP) network port link down due to cable or network errors.
[hostname: statd: sp.network.link.down:warning]: Service Processor (SP) network port link down due to cable or network errors.
[hostname: orftp_rcv_file_from: sp.orftp.failed:warning]: SP communication error, receiver could not locate file on SP.
[hostname: statd: spmgmt.driver.hourly.stats:warning]: The software driver for the Service Processor (SP) detected a problem: Configuration Error (1). 
The following messages are reported in the SP’s Event Log proximate to the watchdog reset:

[IPMI.notice]: 1705 | c0 | OEM: ffff70005000 | ManufId: 150300 | SP Reset Externally
[IPMI.notice]: 1805 | c0 | OEM: fcff70000000 | ManufId: 150300 | POS Register: Unexpected Reset
[IPMI.notice]: 1905 | c0 | OEM: ffff70005000 | ManufId: 150300 | SP Reset Externally
[IPMI.notice]: 1a05 | c0 | OEM: fcff70000000 | ManufId: 150300 | POS Register: Unexpected Reset
[IPMI.notice]: 1b05 | c0 | OEM: ffff70005000 | ManufId: 150300 | SP Reset Externally
[IPMI.notice]: 1c05 | c0 | OEM: fcff70000000 | ManufId: 150300 | POS Register: Unexpected Reset
[IPMI.notice]: 1d05 | c0 | OEM: ffff70005000 | ManufId: 150300 | SP Reset Externally
[IPMI.notice]: 1e05 | c0 | OEM: fcff70000000 | ManufId: 150300 | POS Register: Unexpected Reset
[IPMI.notice]: 1f05 | c0 | OEM: ffff70005000 | ManufId: 150300 | SP Reset Externally
[IPMI.notice]: 2005 | c0 | OEM: fcff70000000 | ManufId: 150300 | POS Register: Unexpected Reset
[IPMI.notice]: 2105 | 02 | EVT: 6f01ffff | Partner_IO_Pre | Assertion Event, "Absent"
[IPMI.notice]: 2205 | 02 | EVT: 0301ffff | System_Fault | Assertion Event, "State Asserted"
[IPMI.notice]: 2305 | 02 | EVT: 0301ffff | Controller_Fault | Assertion Event, "State Asserted"
[IPMI.notice]: 2405 | 02 | EVT: 0300ffff | System_Fault | Assertion Event, "State Deasserted"
[IPMI.notice]: 2505 | 02 | EVT: 0301ffff | PSU1_Input_Type | Assertion Event, "State Asserted"
[IPMI.notice]: 2605 | 02 | EVT: 0301ffff | PSU2_Input_Type | Assertion Event, "State Asserted"
[IPMI.notice]: 2705 | 02 | EVT: 0301ffff | System_Fault | Assertion Event, "State Asserted"
[IPMI Event.critical]: NMI
[IPMI Event.critical]: L2 watchdog timeout hard reset
The messages in the Event Log will occur close enough to the watchdog reset that they cannot be used to predict failure. If a system restarts due to a watchdog reset, this signature can be used to verify if this issue caused the reset.  
Workaround

Run the SP CLI command, sp uptime to determine how long the SP has been running since the last reset.

Reboot the SP:

A reboot of the SP will clear any existing memory resource issues. This action is nondisruptive to data operations on the storage system. Once restarted, the SP will be able to run undisturbed for approximately the next five hundred days. The rate and severity of the leak is well characterized, this approach is a reasonable workaround to address the problem until the SP is updated with a fixed firmware release.

Caution: If the SP Firmware of your system is < 1.3 (FAS32xx/62xx platforms) or < 2.1 (FAS22xx platforms), you should be aware of BUG ID: 546048: Service Processor (SP) fails to come up after “sp reboot” command. Plan for a potential maintenance window to power-cycle the system, if needed.

Solution

Install a SP firmware release that mitigates this issue, once they are available.

In my case, for platform FAS3270, running Data ONTAP Release 7.3.x; the SP FW Release would be 1.3.3P2; which is what I installed.

More on the KB here: https://kb.netapp.com/support/index?page=content&id=7010141&locale=en_US

NetApp volume has the fixed filesystem size option set

The following error is reported when trying to alter the size of a destination flex vol that is in a SnapMirror relation, or that was in a SnapMirror relation.

I most often run into this on volumes that were migrated off older hardware and no longer in a SnapMirror relation, but the fs_size_fixed option is not automatically changed after relationship is broken.

fs_size_fixed

This option causes the file system to remain the same size and not grow or shrink when a SnapMirrored volume relationship is broken, or when a volume add is performed on it. It is automatically set to true when a volume becomes a SnapMirrored volume. It stays set to true after the snapmirror break command is issued for the volume. This allows a volume to be SnapMirrored back to the source without needing to add disks to the source volume. If the volume is a traditional volume and the size is larger than the file system size, setting this option to false forces the file system to grow to the size of the volume. If the volume is a flexible volume and the volume size is larger than the file system size, setting this option to false forces the volume size to equal the file system size.

To change the option in Data ONTAP 7-Mode:
vol options volumename fs_size_fixed off  
To change the option in Clustered Data ONTAP:
volume modify -filesys-size-fixed false -volume volumename  
To change the option using Data ONTAP PowerShell Toolkit, 7-mode:
Get-NaVol volumename | Set-NaVolOption -key fs_size_fixed -value off  
To change the option using Data ONTAP PowerShell Toolkit, Clustered Data ONTAP:
Get-NcVol volumename | Set-NcVolOption -key fs_size_fixed -value off  

Unlock a shared file on NetApp CIFS

We had a ticket come in regarding an Excel file which was complaining that it was “locked for editing” by a user whenever anyone (including that user) tried to open it. There were no temp Excel files in the directory container and user had already rebooted their workstation and cleared out temp windows cache.

First thing I did was open Computer Management and connect to the filer. I looked at the list of open files and didn’t see the file listed as being open.

Next thing I did was ssh into the filer and issue a lock status command:

lock status -f "/vol/Share/folder/file.xlsx" -p cifs  

This produced results back. And indeed, it was indicating the user had a lock on the file.

To unlock:

Data ONTAP 7-mode:
lock break -f "/vol/Share/folder/file.xlsx" -p cifs  
Clustered Data ONTAP:
::> vserver locks show

Notice: Using this command can impact system performance. It is recommended that you specify both the vserver and the volume when issuing this command to minimize the scope of the command’s operation. To abort the command, press Ctrl-C.

Change to advanced mode:

::> set advanced

Perform a break for that client:

::*> vserver locks break -vserver vservername -volume volname -lif lifname -path pathtofile

Warning: Breaking file locks can cause applications to become unsynchronized and might lead to data corruption. If you are breaking a file lock on a volume that is being accessed by a FlexCache you must take the volume offline on the FlexCache to reestablish proper delegation synchronization between the origin and the cache.