Category Archives: windows

WSUS Service Crashing

We had an odd thing happen the other day at work. The WSUS console kept crashing for not apparent reason.  It worked fine up until just now.  Digging into the error message, it seemed that the WSUS console was unable to reason the WSUS service and said to check and make sure IIS & SQL we’re running.  They both were.

As it turns out, in the IIS logs,  there was an error about about IIS exceeding its Private Memory limit and crashing.  Eventually I found this blog post:

That indicated that there was a problem with WSUS application pool.  As it turned out, that app pool was in fact stopped on our server!  Starting it enabled WSUS to function for a while until it crashed again, at which point the app pool was stopped again.

Per the above article, I changed the following entry to “0”:

IIS Manager->Server->Application Pools->WSUS Pool->Advanced Actions->Recycling

And changed Private Memory Limit to 0.  A reboot later, and the application pool was able to access much more than the 1.8 GB it is allowed by default.  However…..It was still crashing.  A second option in that same blog mentioned changing the application pools to default to being 64 bit instead of the default of 32 bit.

All that was required was going IIS->Manager->WSUS Website->ISAPI Filters-> right click – edit->and change the loading order of the 32 bit vs 64 bit entries so the 64 bit option loads first.

Another reboot, and ever since then WSUS has been 100% stable.  I don’t know for certain what caused this, other than we recently added some new servers to the domain for WSUS to manage as well as a new OS for it to manage patches for.  Those two things combined to be enough to push it over the line and crash out of memory crashes.

Exchange Free/Busy & Out of Office Issues

Once again, I’ve went many months between posts.  Sorry.  Life has just been crazy!  Since the last post we’ve moved to a hybrid config with Exchange 2010 and Office 365.

An issue I ran into a while back was with an on-prem user not able to see free/busy calendar info for anyone else, on-prem or 365.  At first I thought it was a problem with the hybrid configuration because it was initially reported that the on-prem user couldn’t see free/busy info for people already moved to O365 (and I had just setup the Hybrid configuration).  However after digging into the details, it turns out they couldn’t see free/busy info for anyone in the Org regardless of the location of their mailbox.  AND, they could access that info from OWA!  It was only broken in Outlook.  And the final head scratcher, the user hadn’t changed their password recently and wasn’t routinely getting locked out of AD.

It took some time to track down, but eventually it was found to be an old saved credential in the Window Credential Manager.  Deleted that, relaunched Outlook, and *poof*!  Free/busy info works!  I have no idea why it simply broke that part of Outlook and didn’t generate any failed login attempts in Exchange, nor did it lock out the AD account.  It simply broke Free/Busy info (and only free/busy).

Exchange & UPN Suffixes

I recently ran into a problem where we reactivated ActiveSync after having it disabled for years because we used a solution for mobile email that didn’t rely on ActiveSync (Good for Enterprise). We decided to re-evaluate ActiveSync, it’s improved a lot since Exchange 2003 and early versions of iOS. Re-enabling the proper settings for a couple users to test with and we were good to go (note, Outlook for iOS is awesome….check it out if you haven’t already). After a bit of testing we decided to increase the number of users using it. Enabling ActiveSync for those mailboxes is all we need to do, right?

Not so much.

I ran into a problem where none of them could setup thier Exchange account in either Outlook for iOS or It failed at the authenication setup. To make things more confusing my user account worked just fine, even on the same phones thier user account failed on.

A newly created test account failed in the same way, so off to I went. The test the website performed an Exchange ActiveSync test and said it worked fine. Connected, authenicated, logged in, etc It’s all good, except no one can connect using thier phone, outside of a few people it worked for orginally (and continues to work for). OWA and Outlook on a PC has always worked and continues to work. It seems to be limited just to ActiveSync. At this point I’m completely out of ideas.

Check the firewall, nothing seems wrong there. Trying a more complex password, no difference. Trying a simpler password, no difference. Finally I think to try a Mac OS X based mail app that talks to Exchange via ActiveSync. Boom, it works! What in the world…..? So it isn’t an ActiveSync problem after all, or least not 100% an ActiveSync problem.

Eventually I found an error when trying one of the Autodiscover tests at with the test account. In the past this same test had worked fine with my account, but I decided to try it with the test account. And it failed! Finally, something to dig into.

What I found was that the Auto Discover test reported:

An HTTP 401 Unauthorized response was received from the remote Unknown server. This is usually the result of an incorrect username or password. If you are attempting to log onto an Office 365 service, ensure you are using your full User Principal Name (UPN).

Digging into that error this was helpful, I found out the UPN suffix for my test account was set to domain.local instead of (which is what our email address is). Changed that and the test account was able to be setup on phones and devices as expected!


I ran into an issue the other day where a file on a network share ended up with its NTFS permissions being hosed in such a way that no one could edit, delete, or even take ownership of it.  I’m not sure how it happened, but it did and the ticket ended up with me to get it fixed.

Nothing I did in the GUI could fix the problem.  I could see the filesystem security attributes were hosed and nothing, not even taking ownership, would successfully complete.  After a quick visit to Google, I found the Technet page for takedown.exe.  It’s basically a tool for sysadmin’s to take ownership of a file with borked permissions.  Perfect!  That’s exactly what I need.

Unfortunately, it didn’t work and failed with a non-helpful generic error.  Turns out I was having a case of the stupids and the file was locked by a crashed application.  Killing the processes released the lock on the file and then I was able to delete the file and restore it from the previous days backup.  On the plus side, I found what looks to be a great tool to keep bookmarked for the future!  

Splunk Queries

I found a great little site (thanks the Splunk sub-Reddit) over at  As a mostly Windows shop running Splunk for log management, there are some real gems in there.  I restructured my Dashboard using some of the queries I found there.

Event IDs for Suspicious Behaviour

This can take a while to run if you have a large dataset, but it provides a look into a few different categories of Event IDs that I had previously overlooked.

source=WinEventLog:security User!=SYSTEM User!="LOCAL SERVICE" User!="NETWORK SERVICE" | eval Trigger=case(EventCode=516, "Audit Logs Modified",EventCode=517, "Audit Logs Modified",EventCode=612, "Audit Logs Modified",EventCode=623, "Audit Logs Modified",EventCode=806, "Audit Logs Modified",EventCode=807, "Audit Logs Modified",EventCode=1101, "Audit Logs Modified",EventCode=1102, "Audit Logs Modified",EventCode=4612, "Audit Logs Modified",EventCode=4621, "Audit Logs Modified",EventCode=4694, "Audit Logs Modified",EventCode=4695, "Audit Logs Modified",EventCode=4715, "Audit Logs Modified",EventCode=4719, "Audit Logs Modified",EventCode=4817, "Audit Logs Modified",EventCode=4885, "Audit Logs Modified",EventCode=4902, "Audit Logs Modified",EventCode=4906, "Audit Logs Modified",EventCode=4907, "Audit Logs Modified",EventCode=4912, "Audit Logs Modified", EventCode=642, "Account Modification",EventCode=646, "Account Modification",EventCode=685, "Account Modification",EventCode=4738, "Account Modification",EventCode=4742, "Account Modification",EventCode=4781, "Account Modification", EventCode=1102, "Audit Logs Cleared/Deleted",EventCode=517, "Audit Logs Cleared/Deleted", EventCode=628, "Passwords Changed",EventCode=627, "Passwords Changed",EventCode=4723, "Passwords Changed",EventCode=4724, "Passwords Changed", EventCode=528, "Successful Logons",EventCode=540, "Successful Logons",EventCode=4624, "Successful Logons", EventCode=4625, "Failed Logons",EventCode=529, "Failed Logons",EventCode=530, "Failed Logons",EventCode=531, "Failed Logons",EventCode=532, "Failed Logons",EventCode=533, "Failed Logons",EventCode=534, "Failed Logons",EventCode=535, "Failed Logons",EventCode=536, "Failed Logons",EventCode=537, "Failed Logons",EventCode=539, "Failed Logons", EventCode=576, "Escalation of Privileges",EventCode=4672, "Escalation of Privileges",EventCode=577, "Escalation of Privileges",EventCode=4673, "Escalation of Privileges",EventCode=578, "Escalation of Privileges",EventCode=4674, "Escalation of Privileges") | stats count by Trigger | sort - count

Windows Environment Logon Count is a great one that shows a stacked graph of network access to files/folders, Service Accounts, Local Console Access, Scheduled Tasks/Batch Files, Network Logins, and RDP Logins.

source="WinEventLog:security" | eval LogonType=case(Logon_Type="2", "Local Console Access", Logon_Type="3", "Accessing Network Folders or Files", Logon_Type="4", "Scheduled Task, Batch File, or Script", Logon_Type="5", "Service Account", Logon_Type="7", "Local Console Unlock", Logon_Type="8", "Network User Logon", Logon_Type="9", "Program launched with RunAs using /netonly switch", Logon_Type="10", "Remote Desktop via Terminal Services", Logon_Type="11", "Mobile Access or Network Domain Connection Resumed") | top limit=15 LogonType | eval percent = round(percent,2) . " %"

One of my favorite queries lists a table of geographical locations for IP addresses pulled from IIS access logs.  In particular, for Exchange OWA and Outlook Anywhere.

host="" sourcetype=iis NOT cs_method=POST | iplocation c_ip |stats count by City | sort count desc

I have 3 instances of this query on my dashboard, one each to show a different table for country, state, and city for our OWA users.  This pulls IIS logs form the Exchange server, filters out the “POST” entries and uses ‘iplocation’ to show what geographical location the GET requests for OWA and Outlook Anywhere are coming from.  It puts all of this in a table sorted by count, which makes it easy to see if we’re taking any hits from locations on the other side of the country or planet.

Spunk can be expensive, though a little less so since they introduced Splunk Light, however the alerting and reports that it enables is well worth it in my opinion.  Version 6.3 was recently released, it features quite a few new features!