Remember the Y2K bug that threatened to destabilize the internet as we knew it around the turn of the century? Microsoft quietly had its Y2K22 moment, albeit brief, as the year changed from 2021 to 2022 but thankfully, all is now back to normal as a fix has been issued.
Some folks woke up on the first day of 2022 to find their email’s inbox empty. While not everybody was in the mood to get some work done, it was still a concern. Even sending out an email was impossible.
The Exchange admins (sorry, you guys just can’t catch a break) quickly discovered that the issue was coming from emails getting stuck in the queue, which errors recorded in the event log like those below:
“Log Name: Application
Logged: 1/1/2022 1:03:42 AM
Event ID: 5300
Description: The FIP-FS “Microsoft” Scan Engine failed to load. PID: 23092, Error Code: 0x80004005. Error Description: Can’t convert “2201010001” to long.”
“Log Name: Application
Logged: 1/1/2022 11:47:16 AM
Event ID: 1106
Description: The FIP-FS Scan Process failed initialization. Error: 0x80004005. Error Details: Unspecified error.”
Microsoft itself has explained what was going wrong, and it has to do with numbers, or more precisely, how the application handles dates. “We have addressed the issue causing messages to be stuck in transport queues of on-premises Exchange Server 2016 and Exchange Server 2019. The problem relates to a date check failure with the change of the new year and it not a failure of the AV engine itself. This is not an issue with malware scanning or the malware engine, and it is not a security-related issue. The version checking performed against the signature file is causing the malware engine to crash, resulting in messages being stuck in transport queues.”
The software maker has published the fix, and admins can apply it manually or run it as a script. However, the solution could take some time to take effect. Also, with so many emails queued up, it might take a while for all of them to be delivered to the correct recipients.
Here is the announcement of the fix:
“We have now created a solution to address the problem of messages stuck in transport queues on Exchange Server 2016 and Exchange Server 2019 because of a latent date issue in a signature file used by the malware scanning engine within Exchange Server. Customer action is required to implement this solution. When the issue occurs, you’ll see errors in the Application event log on the Exchange Server, specifically event 5300 and 1106 (FIPFS).”
To solve the problem by running a script, download the file from here: https://aka.ms/ResetScanEngineVersion
However, for a manual fix, follow the steps below:
Remove existing engine and metadata
- Stop the Microsoft Filtering Management service. When prompted to also stop the Microsoft Exchange Transport service, click Yes.
- Use Task Manager to ensure that updateservice.exe is not running.
- Delete the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\amd64\Microsoft.
- Remove all files from the following folder: %ProgramFiles%\Microsoft\Exchange Server\V15\FIP-FS\Data\Engines\metadata.
Update to the latest engine
- Start the Microsoft Filtering Management service and the Microsoft Exchange Transport service.
- Open the Exchange Management Shell, navigate to the Scripts folder (%ProgramFiles%\Microsoft\Exchange Server\V15\Scripts), and run Update-MalwareFilteringServer.ps1 <server FQDN>.
Verify engine update info
- In the Exchange Management Shell, run Add-PSSnapin Microsoft.Forefront.Filtering.Management.Powershell.
- Run Get-EngineUpdateInformation and verify the UpdateVersion information is-2112330001.
After updating the engine, it is recommended that you also verify that mail flow is working and that FIPFS error events are not present in the Application event log.
Some resourceful admins had found a workaround, but the fix from Microsoft remains the recommended course of action.