Friday, February 14, 2014

Office 365 Hybrid Mailbox Move Issues and Fixes

Microsoft team has completed their Service upgrade ( Wave 15) to most of the customers and post service upgrade the Exchange and Lync Servers in O365 are the latest 2013 Servers and this made some significant changes in the way we manage the Cloud environment. And Hybrid Exchange Environment is the most impacted one we could say as we are used to manage both On premises and Exchange Online from the same EMC console and some major changes occurred with the Service upgrade which eventually took out most of the Management tasks from EMC to the new EAC in Admin portal and also some new additional management changes are now made and this article discuss it along with the latest issue we have with Hybrid Mailbox moves from Cloud On premise which was not known to many as of now and the workaround available which was officially released by Microsoft today and other vital points to consider in the future of Cloud Management from On premises.


Microsoft team released a EHLO blog post which talks about the major changes made in the way Exchange Online is managed from On premises in a Hybrid deployment and how the Move requests needs to be initiated while moving a mailbox from Cloud to On premise via Exchange Online Powershell and the vice versa via EAC etc.

Refer the below post first and review it completely to know the changes as stated above explained descriptively.

Hybrid Mailbox moves and EMC changes

Unfortunately it does not contain any information about the issue which I am going to show you in the upcoming discussion.

Hybrid mailbox moves from Cloud to Onpremises and UCS 

I manage a Hybrid Exchange Environment and we normally would get request to move user back and forth between Cloud and On premises and everything worked seamlessly for us for a while, no impacts for faced for a while even after our successful service upgrade which was done way back and suddenly before few months we noticed some strange issues occurred while moving users from Cloud to On premise and we did used the Exchange Online cmdlet as stated by Microsoft as we have a Exchange 2010 SP3 based Hybrid Environment.

Below is the error received while performing the move


When you read the error completely it will make sense that it talks about user Lync contacts that are still in Exchange Server which are migrated as a part of the new feature called Unified Contact store (UCS) which gets enabled by default when a user who has Lync Online account along with a Mailbox hosted in Exchange Online for the first time he signs-in from a Lync 2013 client at the backend in cloud.

I have already written a post long back over UCS when I had  issue with UCS and Legacy Lync clients during the Service Upgrade and you can refer it below to know what is UCS and how it plays a role within office 365.

UCS and Legacy Clients

Lets get back to our topic, so in this state we were left without a solution of moving mailboxes from Cloud to On premise. When I searched the Internet in Microsoft forums unable to find any discussion available and finally, found the below blog post from Matt Green's blog where he was one of the Administrator who faced  same issue like me with his Environment and Ideally found a working solution on his own.

Access the post here: Hybrid Mailbox Moves – From Office 365 Back to On-Premises

But there were no official documentation from Microsoft over this issue at that time and for a Major Environment like the one I manage we cannot go and do stuffs that are not provided officially from Microsoft and as a result we ended up with a premier case and started our investigation with Microsoft Engineers.

The solution we had is to disable UCS for the user and Roll back his Lync Contacts back to Lync Server from Exchange server and once this is done we will run the Exchange Online Cmdlet shown below to validate whether UCS is disabled by checking the  "Imlistmigrationcompleted" parameter which is set to false indicates UCS is disabled.


The below TechNet article will show you how the UCS can be disabled and how the contacts can be Roll back in a Lync 2013 On premise Environment and the same applies to Cloud as the Servers are the same which is providing the Lync Online service to customers.

Configuring Microsoft Lync Server 2013 to Use the Unified Contact Store

Once we are good we will move the user mailbox to On premise and we were able to do this when we have list of users handy but this was a not a ideal solution on a long term as we might get cases on the move and we cannot delay the requests by contacting Microsoft with a Support request for those users UCS to get disabled and Lync contacts to rollback and this eventually takes time as the support team member needs to raise request with the Backend team to process and they need to work on it and then communicate to their support and then to us etc. And things were the same till yesterday and When working with major tenants we cannot disable UCS on a whole as it is a time consuming process for Microsoft and also few features may not be available for users which they use now when we go via this route and its advisable to do this task on a requirement basis.

Microsoft team and we were working on this for a while to identify is there any way that we ourselves could manage the way of disabling UCS and roll back and both is a not a easy task as it involves some degree of elevated access on the Cloud end to manage things and its was under validation and we were one among the customers outside who are experiencing the similar issue as we have with their Hybrid Environment.

Good News!!! 

Above events explained our state and the same can be applied to other affected tenants and the good news is today Microsoft Team finally released a KB article over this issue and provided a solution or we can even call this a workaround of disabling UCS on our own from Exchange Online by using the Set-Mailbox cmd let and specifying the "Imlistmigrationcompleted"  parameter to false which makes UCS to get disabled and we say to exchange that user Lync contacts are migrated ( Which is not true) and then proceeding with the mailbox move.


By doing this we are giving false information to Exchange Online and proceeding the move which will actually result in data loss where the user Lync contacts still available in the Cloud Mailbox and not migrated to Lync server to make it available post migration for the user .

For roll back we don't have any other option till date even with this Microsoft KB  explains the same above said points clearly and says data loss will be there if you go via the route of specifying the  "Imlistmigrationcompleted"  parameter to false and not actually performing the Contacts Roll back in the backend.


Review here : http://support.microsoft.com/kb/2932443/en-us

If you have a urgent request and user is updated about the impact and fine the data loss then proceed with the above task and with this you could at least complete the move on time.

At this stage we still rely on Microsoft team for the Roll back to get processed and if there can be a chance to grant the tenant admin to get assigned with additional permissions where they can do the roll back themselves or a alternative way where the Lync contacts automatically migrated  from Exchange Online when we disable UCS for a user on our end if possible which makes the real seamless move management to be made possible as before with the Hybrid Exchange deployment.

For other related information over Hybrid move requests from Cloud to On premises refer the below office 365 community post

Exchange Hybrid Deployment – Moving Cloud-Based Mailboxes to the On-Premises Organization

Future!!!

Change is inevitable, Sooner or later we can expect all the management activities to be moved to EAC which is already made available in the Admin portal for managing cloud objects and EMC will be slowly deprecated and we will still continue to have the same Management capabilities with Exchange Online powershell for most of the management activities which is a good news that most of us nowadays rely on Shell rather than GUI but we cannot take that as applicable with all Admins and still GUI is preferred one for many and even me at instances and the only change is, we need to start using the web based EAC and adopt it soon so that we will not have any headache when we have don't have the EMC  in mere future, and obviously one day or another you will upgrade your On premises Environment to Exchange 2013 then EAC is the only option, so get used to it from today and when its the right time it will be just a news and you will have no pain with your work to adopt a new thing at the last minute wherein you have already mastered it... :)

Review the TechNet post below to know more on EAC with Exchange Online

Exchange Admin Center in Exchange Online

I will update this post once I get some additional Information on the same.. standby for my updates soon.

No comments:

Post a comment