Microsoft have done it again, with another security update, they’ve broken a lot of environments, I wish there was a better communication method than finding a small footnote on a blog post that a patch was going to potentially break environments. Many organisations have automated patch workflows that end up catching people out when this stuff happens, it’s a good reason to use technologies such as Veeam’s DataLabs to test patches before production, but that’s a story for another time…
Ok, what did Microsoft break this time?
Two things, and you’re likely here because either:
- You can’t access OWA/ECP
- You’re being bounced back to the main OWA page when trying to sign in and you’ve got a load balancer.
BUT wait there’s more! Microsoft have quietly updated the Microsoft Exchange schema in this security update, be sure to update your schema AFTER installed the July 2021 SU (Security Update).
Problem One, “protectionCertificates.Length<1”:
The full error code is “HMACProvider.GetCertificates:protectionCertificates.Length<1”
Good news: This one isn’t so bad to fix, Bad news: You’ve got to pay more attention to your environment from now on.
This error appears to be triggered when the OAuth certificate (Microsoft Exchange Server Auth Certificate) has expired, Microsoft have now published an article outlining exactly this, so I’d now expect OWA/ECP to stop working whenever your OAuth certificate has expired.
To solve this problem carry out the following:
- Open Exchange Management Shell as Administrator
- Run the following command. (Replace contoso.com with your SMTP domain)
New-ExchangeCertificate -KeySize 2048 -PrivateKeyExportable $true -SubjectName "cn=Microsoft Exchange Server Auth Certificate" -FriendlyName "Microsoft Exchange Server Auth Certificate" -DomainName "contoso.com"3. Take note of your thumbprint, you’ll need it for the next command. Now run the rest of the commands.
Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate (Get-Date)
Set-AuthConfig -PublishCertificate
Set-AuthConfig -ClearPreviousCertificate
Restart-Service MSExchangeServiceHost
Restart-WebAppPool MSExchangeOWAAppPool
Restart-WebAppPool MSExchangeECPAppPool4. If you have multiple Exchange servers, you’ll need to run the following commands on each of them, but wait for the new Exchange Auth Certificate to be replicated to them first.
Restart-Service MSExchangeServiceHost
Restart-WebAppPool MSExchangeOWAAppPool
Restart-WebAppPool MSExchangeECPAppPool5. Wait, this can take a few hours to replicate across (more than the one hour Microsoft state), but then everything will start working again. If you wish to confirm each server is aware of the new Auth configuration you can run “Get-AuthConfig” and validate the Thumbprint and effective date match your new certificate and the time you executed the first “Set-AuthConfig” command respectively. If you have a Hybrid Exchange environment you need to rerun the “Hybrid Configuration Wizard” again to update these changes to Azure Active Directory.
You may find that this isn’t enough, whilst Microsoft don’t reference it within their support document, I’ve also found expired Backend certificates causing this problem too, though based on the time delays mentioned above this could be coincidental. If your backend certificates have expired, this is also quite easy to replace, gather the Thumbprint of the certificate currently being used by the backend and then run the following command:
Get-ExchangeCertificate -thumbprint “Thumbprint” | New-ExchangeCertificateOnce this has been done, change the certificate binding on your Exchange Server by opening IIS, browsing to the “Exchange Back End “site, selecting “Bindings” and editing the HTTPS binding to use the new certificate. It will have the same name as the old certificate so ensure you click “View” to see the properties of your certificate, making sure the certificate has the new expiry.
Problem Two, Cannot log into OWA or ECP, login loop (Using load balancer)
If you’ve got a load balancer and you’re getting a sign-in loop after installing the SU, I also have good news for you, you’re doing things right in that you’re taking a phased rollout to minimize downtime (or so I hope!). But the bad news is that the July 2021 SU for Exchange 2013/2016/2019 does not support being partially deployed, once you’ve started the rollout you need to ensure you don’t maintain a “mixed pool” of patched and unpatched servers to resolve this issue.
Bonus Points: Update your AD Schema
Remember from earlier in the article I highlighted that the AD schema has changed, if you’re using Exchange Server 2013, you’re likely not used to updating your schema anymore as there aren’t any Cumulative Updates for Exchange that normally necessitated this. However if you didn’t update your schema for Exchange 2016/2019 during the June CU updates this applies to you too! To update your schema, open an administrative PowerShell, set yourself to the directory (cd command) of the Exchange Setup.exe. For example with Exchange Server 2013 the default is “C:\Program Files\Microsoft\Exchange Server\V15\Bin\”. Then run the following command.
Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTermsIMPORTANT: Remember when updating your schema, you require Schema Admin or Enterprise Admin privileges, Domain Admin isn’t sufficient.
Error Codes (To help others on their search to this article)
MSExchange Front End HTTP Proxy – Event ID 1003
“[Owa] An internal server error occurred. The unhandled exception was: Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1”
ASP.NET 4.0.30319.0 – Event ID 1309
“Event code: 3005
Event message: An unhandled exception has occurred.”
Michael Paul | @micoolpaul 
Leave a comment