Here is the scenario where this would be important:
- During the course of regular SharePoint administration duties a deployment process was hung up. From past experience and part of the standard procedures in such situation, a SharePoint 2010 timer service was restarted and a command was run to force deployment of pending jobs.
- When the jobs were forced to run, several pending jobs that were not scheduled to run were inadvertently triggered. The job that caused the issues was the automatic password change on SharePoint 2010 managed accounts. This job is typically scheduled for the 7th of every month, but is queued and marked as pending. As the password change job was queued as pending the force command kicked off the password changes. These jobs did not successfully complete and while the passwords were generated and updated in Active Directory, SharePoint was not updated.
- This resulted in a password mismatch between the password in Active Directory and the password that the Web and Service Application Identities were using to run the sites. Furthermore, once the cached credentials on the IIS servers were refreshed the applications were no longer available.
Once the source of the problem was identified Admins needed to manually change the passwords on the affected accounts and then update the passwords in SharePoint. Since CA was not accessible, passwords needed to be changed with PowerShell:
$P7 = ConvertTo-SecureString "password" -asplaintext -force
Set-SPManagedAccount -Identity <account name> -ExistingPassword $P7