Manually Updating Passwords in SharePoint

I had the pleasure recently of being tasked with updating application pool and other service accounts in a mixed version, multiple SharePoint farm SharePoint environment.  Since this is routinely performed annually, I was disappointed to learn that there was very little documentation and truly no telling exactly where each of these accounts may have been used throughout the farms.  Since these environments were in production, it was very important that we took our time and tested functionality each step of the way.  Previously we had an issue where we issued a command to force a single password change through SharePoint, but it automatically changed every single farm password causing a mismatch between what SharePoint had as a password verses what it was actually set as in Active Directory.  This was a disaster!  We had to manually update the passwords in SharePoint using powershell.  Needless to say, we disabled all managed accounts from automatically changing the password.

So we started with the obvious, managed accounts.  Each account had to be simultaneously changed in Active Directory and then updated in SharePoint.  Once we completed the the managed accounts, we restarted the services to make sure they were working properly, and checked each server to make sure they replicated properly.  Below I've listed a few lessons learned and perhaps even some ideas for proactive approaches that you can take now to make the next password update process much more pleasant with a proven and predictable outcome. 

These are in no particular order:

1. Make sure the timer service is started on all servers in the farm.  This will insure that the passwords are replicated from the managed accounts section in Central Administration to the services/applications across the farm.
2. If possible, do your password changes off hours.  Although hearing user complaints about something not working right would be an excellent indication that a change was not applied properly, it's best practice to reduce the impact as much as possible on your user population.
3. Take one step at a time.  Change the password in Active Directory, then change it in SharePoint, then test completly.  Only once you are positive the change was successful should you move to the next.
4. If you do not already have a SharePoint list of all the service accounts you have in your farm and where they are used, this is the perfect time to start one. It will save a ton of research time before your next password change and allow for the process to be shared by multiple teams.  I use a custom list with columns such as: service account name; account type; app pool or application service, details, last password change, etc.  I also recommend using a program like KeyPass or PasswordSafe to keep track of current account passwords and pass phrases.
5. Sometimes you have to manually change the password on the application pool itself in IIS Manager. Also be sure to update secure store service IDs.  This is where your list will pay off big time because once you go through this process once, you will know exactly where those accounts are being used so you can update all instances.

Having written proceedure and clearly documented details not only saves you time and effort the next time you need to manually change passwords, but it's also artifacts that can be passed down to newer team members as staff or responsibility changes.

KB2844286 Security Update on SharePoint 2010 - Causing List View Web Parts to Break

I received a call this week from an end-user reporting that a web part on his site was throwing the following error.  

Here is the text version of the error he was getting:
Unable to display this Web Part. To troubleshoot the problem, open this Web page in a Microsoft SharePoint Foundation-compatible HTML editor such as Microsoft SharePoint Designer. If the problem persists, contact your Web server administrator.

I logged into his site, but it initially rendered just fine for me so I figured it was on his end, but the next day, I too received the error.  So I looked into the ULS Logs on the WFEs to see if I could identify the problem.  As with most cases, the details were limited but there was one thing that stood out that I was able to search on:

Error while executing web part: System.NullReferenceException: Object reference not set to an instance of an object.

After a quick search, I was able to track this down to a Microsoft Security Update that was applied to our web front end servers over the course of the week.  We have two load-balanced Web Front End servers in our farm and it turns out that both did indeed receive the automatic update.  One server updated on the 13th of July (this was the server that the end-user was hitting) and the other server was updated on the 17th of July (which is the one that I was hitting and the reason why, a day later, I was getting the error).

At this point, I needed to check the OS to find the exact KB update that was associated with the issue.

Windows XP and Windows Server 2003:  KB2844285
Windows 7 Service Pack 1 and Windows Server 2008 R2 Service Pack: KB2844286 
Windows Vista Service Pack 2 and Windows Server 2008 Service Pack 2:  KB2844287

Here is the Microsoft Security Bulletin for July 2013:
Microsoft Security Bulletin MS13-052

As a temporary workaround, we found uninstalling the update (KB2844286) from the servers (all WFE’s and App servers), rebooting, and then performing an IISReset on the servers will resolve the issue. Several others have experienced this issue as I found here: http://social.technet.microsoft.com/Forums/sharepoint/en-US/cc9a557b-93cd-40d5-965c-e0a2f107624d/unable-to-display-this-web-part-error-message-after-patch-kb2844286 and here: http://social.msdn.microsoft.com/Forums/en-US/871a0661-05d0-4f3a-b66d-2504552142bd/error-while-executing-web-part-systemnullreferenceexception-object-reference-not-set-to-an

At this point, this is the best information I was able to find.

Also according to other blog/forum posts I've read, it appears this error "typically" appears when a list or web part view has had the XSLT customized.