Extremely Dangerous Cluster Setting on New Backup Agent.

DBA_Dave

I just installed the latest server components onto a cluster server. The service was added as a cluster resource, something I could have done without, and then it did the inexcusable: It set the resource to restart and affect the group.

The backup service then failed to start a couple of times for some unknown reason, triggering the unplanned failover of a production server in the middle of the work day.

The servers are not configured to fail over automatically. We have to reconfigure the memory settings first. This caused a rather large, public disaster, one that would have been much worse had it not occurred during the lunch hour.

For future reference, it is solely the prerogative of the cluster administrator to decide:

a. Whether to automatically restart a cluster resource or not
and if so,
b. Whether or not to allow it to affect the group

This decision is not a developer's to make. This ancillary service is not important enough to warrant a cluster failover, and as I said, it is my decision as to whether that happens, not yours.

It was totally irresponsible and reckless to install a cluster service with both those settings enabled.

I encourage you to correct this, and I hope this warning prevents this from happening to other DBA'a who depend on your products.

cluthy

I totally agree.

This problem caused unexpected failover on our cluster(s) when upgrading to 5.2, thank god I did the upgrade off hours.

This REALLY should not affect the group. If the server(s) have proper monitoring installed, simply having the service fail should be sufficient to set off any alarms.

HJoyce

An updated install of the Server components of SQL Backup may offer an option to failover (or not) the resource group upon a SQL Backup service startup failure.

The default at the moment is to failover if the service fails to start. We would be able to make the non-failover the default. Would this be a suitable solution for you were we to offer it?

In the case where the servers don't failover successfully by default it seems this would be the solution.

DBA_Dave wrote:

The servers are not configured to fail over automatically. We have to reconfigure the memory settings first.

In the case where clusters are configured to failover successfully (which I assume is normal) failing over if the SQL Backup services fails to start may be the most appropriate default setting, but it seems sensible for us to offer the choice at least.

thanks.
Helen

mdgraves

We saw this setting, and put it into our local installation document to immediately turn that option off. BACKUP svc stopping is not enough reason to fail over a cluster.

We're touching this value when we go in and also define the dependencies, which are not being set as well.

DBA_Dave

The service can be set to restart, but not to affect the group. It can't be put in as a dependency on a resource without taking that resource offline, so there should be no attempt to do so.

The default should be the safe option. The installer does not appear to be able to cleanly start the new service, and even if that were not the case, the only correct answer is to "first, do no harm".

mdgraves

We noticed the installation changes the DEFAULT BKUP PATH registry setting for SQL. During UPGRADES, this causes a problem, especially on Clusters, where we have multiple drives defined for SQL. RedGate install reconfigures the DEFAULT BKUP PATH registry setting to same as DATA path.

If DBA is not paying attention, he uses room on his DATA drive.

Have install ASK us, or not change at all.

petey

The installer for the next release of SQL Backup will not set the SQL Backup Agent service to cause a cluster failover if it fails to start, as the default option. You have a choice to change this option.

To mdgraves, SQL Backup does not change any SQL Server registry settngs during installation or upgrades.

DBA_Dave

petey wrote:

To mdgraves, SQL Backup does not change any SQL Server registry settngs during installation or upgrades.

It may not change any registry settings, but it sure changed the destination of several of my backups to the Windows default folder rather than the dedicated SAN disk I had been pointing at.

I found out when the drives filled... Now I make sure to check every job after an upgrade to make sure the drive stays the same.

petey

If SQL Backup did indeed change the settings, I would like to rectify it if I can reproduce the error.

When you first install SQL Backup, and don't configure any of its settings, a backup using the <AUTO> option will create SQL Backup backup files in the folder indicated by the BackupDirectory value found in the associated SQL Server instances' registry settings. If this value is empty, SQL Backup then retrieves the value for SQLDataRoot, and creates the backup files in the Backup subfolder indicated by that value. If this value is also emptry, SQL Backup then creates the backup files in the folder where the SQL Backup executables are located.

I have ran some tests today to confirm the above behaviour. SQL Server was first installed, followed by SQL Backup. The tests were performed on SQL Server 2000 and SQL Server 2005 (both 32-bit and 64-bit). At no point did SQL Backup change any of SQL Servers' registry settings.

I do have 2 questions:
- when you refer to the 'DEFAULT BKUP PATH' registry setting, are you referring to the 'BackupDirectory' value?
- when you mentioned that 'it sure changed the destination of several of my backups...', are you referring to backups created using SQL Server, or SQL Backup?

Thanks.

mdgraves

(1) Yes, the registry value for 'BackupDirectory' was changed. We had manually set these values on my 70 servers. As doing upgrade to v5.2, then found backups starting failing on space, discovered 'BackupDirectory' was now set back to same as "DATA" path. (2) We exclusively run via RedGate backup.

petey

I upgrade a clustered instance from SQL Backup version 5.0 to 5.1, followed by 5.2, from both the active node and passive node. The BackupDirectory value was never changed.

If you are upgrading to 5.2 on another server, could you please run something like Regmon, and trace the changes made to the registry? It would help us if we had a log of the changes. Thanks.