xp_readerrorlog usage overload results in 100% CPU load

ThailoThailo Posts: 4 Bronze 1
in Redgate Monitor
3 times in 9 days now, we experienced a build up in CPU load (to 100%) and a significant increase in connection count on an active cluster node, apparently caused by SQL Monitor calling xp_readerrorlog resulting in PREEMPTIVE_OS_GETPROCADDRESS waits (hundreds of them).

My setup consists of SQL Server 2016 Enterprise SP1 build 13.0.4466.4 with 2 cluster nodes, Windows 2012 R2/4-core/64 GB on ESX, everything running on a Compuverde metro storage cluster, with SQL Monitor base monitor service build 7.1.19.11592.

Redgate support is investigating too but does not have an answer yet. We have tried calling sp_cycle_errorlog several times with no visible effect. The only solution that worked for now was a hard reboot via the VMWare console because a logon to the Windows host was not possible due to the high CPU load (it caused RDP or other connection timeouts mainly), even Windows Failover clustering service did not work as expected.

Currently I've suspended cluster monitoring (obviously not what I want) to prevent another build up. So all is 'in the green' for now, except SQL Monitor for this cluster. I found https://forum.red-gate.com/discussion/81020/restore-process-hung-with-preemptive-os-getprocaddress-wait-type which might shed some light on my problem as well, however the most important link in that post suggesting a solution, is dead.

Any thoughts/experiences/solutions? Anyone?
Tagged:
· Share on Twitter

Comments

  • RichardLRichardL Posts: 417 Gold 4

    Hi @Thailo


    Thanks for your post. 

     

    This looks like a question that one of Support engineers will need to investigate for you.

     

    If you've a got support contract, please send us a ticket. Provide as much information as you can - screenshots of any errors, log files etc – so we can help you as fast as possible.

    If you're not covered by a Support contract at the moment, email our Sales team at sales@red-gate.com, and they'll be able to help.

    Customer Support
    Redgate Software
    · Share on Twitter
  • hcDBAhcDBA Posts: 4 New member
    We have the exact same problem with one of our servers.  It keeps hitting 100% CPU, and there are several processes running xp_readerrorlog that have wait types of PREEMPTIVE_OS_GETPROCADDRESS.  We don't have hundreds of them like the case above, but it would reach hundreds if we let it keep accumulating.  We end up having to reboot and the problem continues to arise.  In the meantime, we have had to suspend monitoring by SQL Monitor on that server.  I am going to put in a support ticket and hope the support staff can figure it out, but just wanted to give everyone a heads up that this is not an isolated case - we're getting it, too!
    · Share on Twitter
  • ThailoThailo Posts: 4 Bronze 1
    @hcDBA Which version of SQL Monitor are you using? It looks like the combination of clearing our error logs and upgrading SQL Monitor from build 7.1.19.11592 to 7.1.20.11693 solved our problems for now (knock on wood here): the cluster is behaving nicely for 48 hrs now and counting.
    · Share on Twitter
  • hcDBAhcDBA Posts: 4 New member
    We are also using 7.1.20.11693, so we are on the same version.  We have been good for about 7 hours now since the last reboot, and we are not going add it back to SQL Monitor, because we are trying to isolate the cause (for the last several reboots, it was being monitored by SQL Monitor, so we want to see if keeping it off will prevent the high CPU issue). 
    · Share on Twitter
  • Alex BAlex B Posts: 1,157 Diamond 4
    Hi @hcDBA

    Are you using SQL Server 2008 SP1 (or SQL Server 2005 and SQL Server 2008)?  If so it may be this issue, which we have had customers run into.  If this is the case you will need to patch your SQL Server.  If you are on SQL Server 2016 like @Thailo then you should also send in a support ticket as directed by @RichardL above.

    Kind regards,
    Alex


    Product Support Engineer | Redgate Software

    Have you visited our Help Center?
    · Share on Twitter
  • hcDBAhcDBA Posts: 4 New member
    Yes, that server is running SQL Server 2008 SP1 (it's scheduled to be upgraded later this year).  I will check it out.  Thank you!
    · Share on Twitter
  • hcDBAhcDBA Posts: 4 New member
    Just a follow-up: we installed SQL 2008 SP4 on that server (after testing on our dev servers) and that fixed the issue.  Thank you!
    · Share on Twitter
Sign In or Register to comment.