[BUG] SQL 2016 Passive Mirror Server - Still Happening

[jaywiseman1971]

Component

SQL collection scripts

Performance Monitor Version

2.1.0

SQL Server Version

SQL 2016

Windows Version

Windows 2016

Describe the Bug

Still happening with passive SQL on mirror server.

3 Dumps zipped up.

passive_mirror_server.zip

Steps to Reproduce

Still happening with passive SQL on mirror server.

Expected Behavior

Still happening with passive SQL on mirror server.

Actual Behavior

Still happening with passive SQL on mirror server.

Error Messages / Log Output

Still happening with passive SQL on mirror server.

Screenshots

Still happening with passive SQL on mirror server.

Additional Context

Still happening with passive SQL on mirror server.

[erikdarlingdata]

@jaywiseman1971 I did my best on this one without having a working setup to test against. If monitoring that server is important to you, you're going to have to let me know how to address it.

[jaywiseman1971]

@erikdarlingdata which I could help, not a developer.

Maybe an interface option to turn DB's off for monitoring? The DB is "customers" in a high availability failover setup for mirroring.

Best, Jay

[erikdarlingdata]

@jaywiseman1971 All you have to develop is the query to avoid those databases and pass that along. Otherwise, it's a consulting engagement where you give me access to the servers and I figure it out. I do not have that set up to test on. Thanks!

[jaywiseman1971]

Maybe I missed it, is there an uninstall script parameters to remove this from a server?

Best, Jay

[erikdarlingdata]

[erikdarlingdata]

Hey @jaywiseman1971 — we just merged an uninstall option into dev (PR #432). When the next release goes out, you'll have three ways to uninstall:

1. CLI: PerformanceMonitorInstaller.exe <server> <user> <pass> --uninstall
2. GUI: Connect to the server → click the Uninstall button
3. SSMS: Run install/00_uninstall.sql directly

This removes all Agent jobs, Extended Events sessions, server-side traces, and the database. Let us know if you have any questions!

[erikdarlingdata]

@jaywiseman1971 I analyzed all three dump files (including running them through WinDbg). Here's what we found:

Dump Analysis

All three dumps are identical — same spid 40, same error:

• ex_dump_if_requested: Exception raised, major=52, minor=42, state=9, severity=22
• SQL Server 13.0.7029.3 (SQL 2016 SP3 CU2)

The crashing session is a SQL Server internal background thread, not a PerformanceMonitor collector:

• Spid 40 with m_fBackground = 1, m_fUserProc = 0
• No login record (all zeros), no database context (m_fHasDbRef = 0)
• This is a system process like the lazy writer or QDS background task

The Customers database is in RESTORING state (mirroring passive). The error log shows hundreds of login failures:

Login failed for user 'WebCustomer'. Reason: Failed to open the explicitly specified database 'Customers'. [CLIENT: 192.168.213.26]

This is error 18456, state 38 — the WebCustomer login has Customers as its default database, which is inaccessible on the passive mirror.

Root Cause

This is a known SQL Server 2016 engine bug where internal background threads crash (severity 22) when they encounter metadata from a RESTORING database. The crash call chain (sqlmin+0x017EA6A7 → sqlmin+0x00DC3227 → sqlmin+0x00E8A4C1) is consistent with internal metadata access violations on inaccessible databases.

Recommendations

1. Update SQL Server — you're on SP3 CU2 (13.0.7029.3). The latest is SP3 CU17 (13.0.6455.2). Several CUs between CU2 and CU17 include fixes for severity 22 crashes on mirroring passive servers. This is the most impactful fix.

2. Change the WebCustomer login's default database — it's currently set to Customers which is RESTORING on the passive side. Change it to master or another database that's ONLINE on both partners:

   ALTER LOGIN [WebCustomer] WITH DEFAULT_DATABASE = [master];

   This will stop the flood of 18456 state 38 errors.

3. PerformanceMonitor hardening — while this crash isn't caused by our collectors, we're going to add a d.state = 0 (ONLINE-only) filter to the waiting_tasks collector as a defensive measure. The other collectors (query_stats, procedure_stats, query_store, file_io_stats) were already hardened in PR Fix SQL dumps on mirroring passive servers #385, but waiting_tasks was missed. We'll get that into the next build.

[jaywiseman1971]

ALTER LOGIN [WebCustomer] WITH DEFAULT_DATABASE = [master], did this on the primary and passive servers and it stopped the memory dumps now with your application.

I did re-install it after dropping the database/jobs from above.

Thank you! Glad you have another point to check on below. Keep up the great work!

we're going to add a d.state = 0 (ONLINE-only) filter to the waiting_tasks collector as a defensive measure.

[erikdarlingdata]

@jaywiseman1971 okay, cool, I'm going to close this one out for now. Thanks for letting me know about it. Let me know if anything kicks up on you down the line.