Microsoft Exchange Server 2010/2013/2016/2019 – Unable to connect to OWA/ECP “protectionCertificates.Length<1"

Microsoft have done it again, with another security update, they’ve broken a lot of environments, I wish there was a better communication method than finding a small footnote on a blog post that a patch was going to potentially break environments. Many organisations have automated patch workflows that end up catching people out when this stuff happens, it’s a good reason to use technologies such as Veeam’s DataLabs to test patches before production, but that’s a story for another time…

Ok, what did Microsoft break this time?

Two things, and you’re likely here because either:

  1. You can’t access OWA/ECP
  2. You’re being bounced back to the main OWA page when trying to sign in and you’ve got a load balancer.

BUT wait there’s more! Microsoft have quietly updated the Microsoft Exchange schema in this security update, be sure to update your schema AFTER installed the July 2021 SU (Security Update).

Problem One, “HMACProvider.GetCertificates:protectionCertificates.Length<1”:

Good news: This one isn’t so bad to fix, Bad news: You’ve got to pay more attention to your environment from now on.

This error appears to be triggered when the OAuth certificate (Microsoft Exchange Server Auth Certificate) has expired, Microsoft have now published an article outlining exactly this, so I’d now expect OWA/ECP to stop working whenever your OAuth certificate has expired.

To solve this problem carry out the following:

  1. Open Exchange Management Shell as Administrator
  2. Run the following command. (Replace contoso.com with your SMTP domain)
New-ExchangeCertificate -KeySize 2048 -PrivateKeyExportable $true -SubjectName "cn=Microsoft Exchange Server Auth Certificate" -FriendlyName "Microsoft Exchange Server Auth Certificate" -DomainName "contoso.com"

3. Take note of your thumbprint, you’ll need it for the next command. Now run the rest of the commands.

Set-AuthConfig -NewCertificateThumbprint <ThumbprintFromStep1> -NewCertificateEffectiveDate (Get-Date)
Set-AuthConfig -PublishCertificate
Set-AuthConfig -ClearPreviousCertificate
Restart-Service MSExchangeServiceHost
Restart-WebAppPool MSExchangeOWAAppPool
Restart-WebAppPool MSExchangeECPAppPool

4. If you have multiple Exchange servers, you’ll need to run the following commands on each of them, but wait for the new Exchange Auth Certificate to be replicated to them first.

Restart-Service MSExchangeServiceHost
Restart-WebAppPool MSExchangeOWAAppPool
Restart-WebAppPool MSExchangeECPAppPool

5. Wait, this can take a few hours to replicate across (more than the one hour Microsoft state), but then everything will start working again. If you wish to confirm each server is aware of the new Auth configuration you can run “Get-AuthConfig” and validate the Thumbprint and effective date match your new certificate and the time you executed the first “Set-AuthConfig” command respectively. If you have a Hybrid Exchange environment you need to rerun the “Hybrid Configuration Wizard” again to update these changes to Azure Active Directory.

You may find that this isn’t enough, whilst Microsoft don’t reference it within their support document, I’ve also found expired Backend certificates causing this problem too, though based on the time delays mentioned above this could be coincidental. If your backend certificates have expired, this is also quite easy to replace, gather the Thumbprint of the certificate currently being used by the backend and then run the following command:

Get-ExchangeCertificate -thumbprint “Thumbprint” | New-ExchangeCertificate

Once this has been done, change the certificate binding on your Exchange Server by opening IIS, browsing to the “Exchange Back End “site, selecting “Bindings” and editing the HTTPS binding to use the new certificate. It will have the same name as the old certificate so ensure you click “View” to see the properties of your certificate, making sure the certificate has the new expiry.

Problem Two, Cannot log into OWA or ECP, login loop (Using load balancer)

If you’ve got a load balancer and you’re getting a sign-in loop after installing the SU, I also have good news for you, you’re doing things right in that you’re taking a phased rollout to minimize downtime (or so I hope!). But the bad news is that the July 2021 SU for Exchange 2013/2016/2019 does not support being partially deployed, once you’ve started the rollout you need to ensure you don’t maintain a “mixed pool” of patched and unpatched servers to resolve this issue.

Bonus Points: Update your AD Schema

Remember from earlier in the article I highlighted that the AD schema has changed, if you’re using Exchange Server 2013, you’re likely not used to updating your schema anymore as there aren’t any Cumulative Updates for Exchange that normally necessitated this. However if you didn’t update your schema for Exchange 2016/2019 during the June CU updates this applies to you too! To update your schema, open an administrative PowerShell, set yourself to the directory (cd command) of the Exchange Setup.exe. For example with Exchange Server 2013 the default is “C:\Program Files\Microsoft\Exchange Server\V15\Bin\”. Then run the following command.

Setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms

IMPORTANT: Remember when updating your schema, you require Schema Admin or Enterprise Admin privileges, Domain Admin isn’t sufficient.

Error Codes (To help others on their search to this article)

MSExchange Front End HTTP Proxy – Event ID 1003

“[Owa] An internal server error occurred. The unhandled exception was: Microsoft.Exchange.Diagnostics.ExAssertException: ASSERT: HMACProvider.GetCertificates:protectionCertificates.Length<1”

ASP.NET 4.0.30319.0 – Event ID 1309

“Event code: 3005
Event message: An unhandled exception has occurred.”

By micoolpaul

Technical Consultant at Nexus Open Systems. Focusing on Veeam, VMware & Microsoft Productivity and Infrastructure stacks.

4 comments

  1. Important note for future readers: the Set-AuthConfig -NewCertificateThumbprint “taking a few hours to work” depends on your timezone. So if you’re in NZ, it takes 12 hours to work because the command assumes you’re giving it UTC Effective Date. To get around this you can either change your server’s timezone to UTC temporarily or adjust the command.

    Like

    1. Hi Brenden, I’ve heard others say this too but as I’m not in such a time zone I haven’t experienced this first hand, but yes, if you run the Get-AuthConfig you can see the effective date for this as well to validate if you have a time delay for this to take effect. As it will give you two datetime outputs, one of them in UTC and one that should be localised. One way around this would be instead of using Get-Date directly in the command you could create a variable called $date by running “$date = Get-Date” and then manipulate the hours relative to your time zone by then running $date = $date.AddHours(-NumOfHours) and then using $date as your variable in the Set-AuthConfig command instead of Get-Date directly.

      Like

      1. Unfortunately, this method also doesn’t work, since we receive this:
        “The new certificate effective date is invalid. The effective date must be greater than the current time.”

        Like

      2. Hi Aerocom, the suggestion I made is for people on -UTC time zones, you can use a + instead of a minus to implement this on time zones ahead relative to UTC. Hope this helps!

        Like

Leave a comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s