Extremely Dangerous Cluster Setting on New Backup Agent.
DBA_Dave
Posts: 31
I just installed the latest server components onto a cluster server. The service was added as a cluster resource, something I could have done without, and then it did the inexcusable: It set the resource to restart and affect the group.
The backup service then failed to start a couple of times for some unknown reason, triggering the unplanned failover of a production server in the middle of the work day.
The servers are not configured to fail over automatically. We have to reconfigure the memory settings first. This caused a rather large, public disaster, one that would have been much worse had it not occurred during the lunch hour.
For future reference, it is solely the prerogative of the cluster administrator to decide:
a. Whether to automatically restart a cluster resource or not
and if so,
b. Whether or not to allow it to affect the group
This decision is not a developer's to make. This ancillary service is not important enough to warrant a cluster failover, and as I said, it is my decision as to whether that happens, not yours.
It was totally irresponsible and reckless to install a cluster service with both those settings enabled.
I encourage you to correct this, and I hope this warning prevents this from happening to other DBA'a who depend on your products.
The backup service then failed to start a couple of times for some unknown reason, triggering the unplanned failover of a production server in the middle of the work day.
The servers are not configured to fail over automatically. We have to reconfigure the memory settings first. This caused a rather large, public disaster, one that would have been much worse had it not occurred during the lunch hour.
For future reference, it is solely the prerogative of the cluster administrator to decide:
a. Whether to automatically restart a cluster resource or not
and if so,
b. Whether or not to allow it to affect the group
This decision is not a developer's to make. This ancillary service is not important enough to warrant a cluster failover, and as I said, it is my decision as to whether that happens, not yours.
It was totally irresponsible and reckless to install a cluster service with both those settings enabled.
I encourage you to correct this, and I hope this warning prevents this from happening to other DBA'a who depend on your products.
Dave Rutledge
Database Administrator
SNL Financial LC
Database Administrator
SNL Financial LC
Comments
This problem caused unexpected failover on our cluster(s) when upgrading to 5.2, thank god I did the upgrade off hours.
This REALLY should not affect the group. If the server(s) have proper monitoring installed, simply having the service fail should be sufficient to set off any alarms.
The default at the moment is to failover if the service fails to start. We would be able to make the non-failover the default. Would this be a suitable solution for you were we to offer it?
In the case where the servers don't failover successfully by default it seems this would be the solution.
In the case where clusters are configured to failover successfully (which I assume is normal) failing over if the SQL Backup services fails to start may be the most appropriate default setting, but it seems sensible for us to offer the choice at least.
thanks.
Helen
SQL Backup Project Manager
Red Gate Software
We're touching this value when we go in and also define the dependencies, which are not being set as well.
The default should be the safe option. The installer does not appear to be able to cleanly start the new service, and even if that were not the case, the only correct answer is to "first, do no harm".
Database Administrator
SNL Financial LC
If DBA is not paying attention, he uses room on his DATA drive.
Have install ASK us, or not change at all.
To mdgraves, SQL Backup does not change any SQL Server registry settngs during installation or upgrades.
SQL Backup Consultant Developer
Associate, Yohz Software
Beyond compression - SQL Backup goodies under the hood, updated for version 8
It may not change any registry settings, but it sure changed the destination of several of my backups to the Windows default folder rather than the dedicated SAN disk I had been pointing at.
I found out when the drives filled... Now I make sure to check every job after an upgrade to make sure the drive stays the same.
Database Administrator
SNL Financial LC
When you first install SQL Backup, and don't configure any of its settings, a backup using the <AUTO> option will create SQL Backup backup files in the folder indicated by the BackupDirectory value found in the associated SQL Server instances' registry settings. If this value is empty, SQL Backup then retrieves the value for SQLDataRoot, and creates the backup files in the Backup subfolder indicated by that value. If this value is also emptry, SQL Backup then creates the backup files in the folder where the SQL Backup executables are located.
I have ran some tests today to confirm the above behaviour. SQL Server was first installed, followed by SQL Backup. The tests were performed on SQL Server 2000 and SQL Server 2005 (both 32-bit and 64-bit). At no point did SQL Backup change any of SQL Servers' registry settings.
I do have 2 questions:
- when you refer to the 'DEFAULT BKUP PATH' registry setting, are you referring to the 'BackupDirectory' value?
- when you mentioned that 'it sure changed the destination of several of my backups...', are you referring to backups created using SQL Server, or SQL Backup?
Thanks.
SQL Backup Consultant Developer
Associate, Yohz Software
Beyond compression - SQL Backup goodies under the hood, updated for version 8
If you are upgrading to 5.2 on another server, could you please run something like Regmon, and trace the changes made to the registry? It would help us if we had a log of the changes. Thanks.
SQL Backup Consultant Developer
Associate, Yohz Software
Beyond compression - SQL Backup goodies under the hood, updated for version 8