InfoBridge just released a new version (3.2.141) of the Exchange Online Synchronizer which resolves the infamous ‘Exchange Server Down?’ message which is usually caused by Exchange throttling. This article provides some background information, explanation of new settings and recommendations.
Background
In the past years a lot of customers have switched from the Exchange Synchronizer v2 to the Exchange Online Synchronizer, which is now the recommended product for server based synchronization for Exchange 2007 SP1 and up. The new product no longer uses Extended MAPI to communicate with Exchange but uses Exchange Web Services, or EWS for short. This switch had several benefits:
- No more ‘ICS file corruption’;
- Future proof;
- Support for Office 365 (Exchange Online) and Hosted Exchange.
However, by using EWS a new factor came into play: client throttling policies.
Client throttling policies were introduced by Microsoft in Exchange to ensure a good performance of the Exchange server by limiting the bandwidth and resources clients can use. Each client has a budget for the amount of resources it can use and once the client exceeds this budget, throttling kicks in. The server will first start to delay the requests coming from the client which, in case of the Synchronizer, will slow down synchronization. If the client continues to breach the throttling limit the server will eventually block all incoming request for the client for a while. For an external application like the Synchronizer it would seem that the server is not available anymore, hence the message ‘Exchange Server Down?’.
The problem
You can imagine that the Synchronizer easily breaks the throttling limits as all requests are executed by the ‘syncuser’. The limits are designed with a single user accessing it’s mailbox in mind, but the Synchronizer might access dozens of mailboxes every second, depending on the number of users that are linked and the amount of changes made by those users.
To make things worse: In Exchange 2010 SP1 and up throttling policies are enabled by default.
The previous solution
The solution for this problem up until now was fairly simple and straightforward: create a new throttling policy that has no limits at all and assign this policy to the ‘syncuser’. This method is described in the InfoBridge knowledgebase in this article.
Whilst this solution works in most cases there are several problems with this method:
- Some (usually larger) organisations do not allow custom throttling policies;
- Most Hosted Exchange providers do not allow customers to either change throttling policies or access Exchange Management Shell at all;
- Office 365 (Exchange Online) currently does not support configuration of throttling policies.
The new solution
In version 3.2.141 of the Exchange Online Synchronizer 3 new options are introduced which should resolve throttling issues once and for all.
However, keep in mind that each option is designed with a specific area / cause of throttling in mind so just setting all options is not recommended: each option will have an impact on the performance of the Synchronizer.
The three options are added to the technical settings screen of the Synchronizer Admin:
* Optimize for High Server Load environments
This option enables response time monitoring of EWS requests. When the (weighted moving) average of the requests is larger than 5 seconds the Synchronizer will throttle its requests to the Exchange server by adding small delays.
* Use Incremental Change Synchronization Detection only
With this option the Synchronizer will only use ICS to detect changes. Each user will be checked (in sequential order) for changes through ICS. In between each user there is a delay of 3 seconds.
* Limit number of concurrent server requests
The Synchronizer will execute requests simultaneously up to the maximum specified. Additional requests will be queued when the maximum is reached.
The value must be equal to the value of ‘EWSMaxConcurrency’ in the throttling policy of the sync user. That value is by default set to 20 and this is also the default value in the Synchronizer.
Setting it to 0 in the Synchronizer will mean that there is no limit. This corresponds to the behaviour of synchronizer versions prior to 3.2.141.
To get the ‘EWSMaxConcurrency’’ for the policy of the ‘syncuser’, execute the following Exchange Management Shell commands:
- Get-ThrottlingPolicyAssociation “domain\syncuser”
- Remember the ‘ThrottlingPolicyId’
- get-throttlingpolicy ENTER_POLICY_ID_HERE
- Check the value ‘EWSMaxConcurrency’
Example:
Impact
‘Optimize for High Server Load environments’ and ‘Limit number of concurrent server requests’ (when set to the default value) have the smallest impact on the user experience. The Synchronizer will just take a bit longer to synchronize items from Exchange to SuperOffice and from SuperOffice to Exchange.
‘Use Incremental Change Synchronization Detection only’ has the biggest impact on the synchronization speed. This is due to the fact that new items will be synchronized through ICS instead of through events from Exchange. The other way around, changes from SuperOffice, is unaffected though.
Recommended settings
Office 365
We recommend turning on ‘Use Incremental Change Detection only’ in synchronizers connected to Office 365 environments. Also make sure that the 'Limit number of concurrent requests' is set to 20 (this is the default setting in both the Synchronizer and the default Office 365 throttling policy).
Closing notes
The new options will definitely solve a lot of throttling related issues. However, if a customer wants very fast synchronization and has the capacity on the Exchange Server for this, we recommend turning all throttling options in the Synchronizer off, setting ‘Limit number of concurrent server requests’ to 0 and making a throttling policy for the ‘syncuser’ with all throttling parameters set to null as described in this knowledge base article.