Azure Active Directory Sync – AAD Connect Disaster Recovery and High Availability

I just wanted to write and tell you all about a fantastic new feature built into the AAD Connect tool. It’s name is ‘Staging Mode’ and it has a dual purpose; a) it allows you to have a server which is essentially on standby, and b) it can be used just as it’s name suggests, in a kind of test mode where you can see what is being imported before it all gets sent off to Azure AD.

In real life it would be utilised thus:

Customer A has a functional installation of AAD Sync / AAD Connect which is synchronising objects and attributes between Azure Active Directory and the On Premise Active Directory. They then build an AAD Connect server in their DR datacentre (or wherever they fancy), and during the initial configuration, enable ‘Staging Mode’. Apart from this setting, they configure it just like their existing, live AAD Sync / AAD Connect server. They even leave the scheduled task enabled and running. All of a sudden, DR strikes, and the live AAD Sync / AAD Connect server goes offline forevermore, cast into the computer graveyard in the sky. Rather than restore the server from backup, they simply log into their second AAD Connect server and disable ‘Staging Mode’. This server then starts synchronising with Azure Active Directory in earnest, without having to miss a beat.

What Staging Mode does is very simple. It acts just like a functional AAD Connect installation, except for the fact that it exports nothing to Azure Active Directory or your on premise Active Directory. It also does not perform any password sync or password write-back functions. The metaverse is fully populated and ready to start exporting data, giving you the easiest possible way to have a server on standby. Unfortunately there is no replication between your two synchronisation servers, so any configuration changes need to be replicated manually, but this is another step to making AAD Connect fully HA, which is becoming much more desirable as Azure Active Directory gains traction.

AADConnect Staging Mode

Advertisements

ADFS 3.0 service will not start – error 1297

This issue is fairly well documented, but I wanted to put it here for my own purposes:

When installing a new ADFS farm, you may find that if you reboot the ADFS server, or restart the ADFS service, it will not restart and fails with a 1297 error code. In the Event Viewer you will see an error stating that;

A privilege that the service requires to function properly does not exist in the service account configuration

This error screams of an issue with the configuration of the service account…and that’s exactly what it is. On the affected ADFS server, open the Local Security Policy console (secpol.msc) and expand the following container:

Security Settings\Local Policies\User Rights Assignment

Go into the properties of the Generate Security Audits section and add the ADFS service account into here. If the option to add an account is grayed out, then that means that a Group Policy is controlling this access list, and you will need to find and modify the appropriate GP to add the ADFS service account into the group (usually the Default Domain Policy). While you are here, ensure that the ADFS service account has ‘Log on as a Service’ privileges.

Once this is done you should be able to start the ADFS service (although if you edited Group Policy then run gpudpdate first). Hopefully this helps you before you get to the point where you make the ADFS service account a Domain Admin! Remember, this account only needs Domain User privileges and should not be put into god mode!

Modify AADSync Default Schedule

When using the AADSync tool to synchronise your Active Directory environment with Azure Active Directory (AAD), the default schedule for an incremental sync is 3 hours. This is done using a Scheduled Task. There are many reasons why you may want to change this schedule; maybe you have a high rate of change in your AD environment and you need a 1 hour sync to keep Office 365 up to date, or it might be that you have such a slow rate of change in your AD environment that you only want to sync your identities once every few days. It is worth mentioning that Password Synchronisation does not follow this schedule and is done immediately following a change of password, so this shouldn’t play a part in your decision to modify any scheduling of sync tasks.

Whatever your reasons, you are likely to become a little befuddled when trying to modify the regularity of your scheduled task. If you go into Task Scheduler, find the Azure AD Sync task and go into Properties, you can change the frequency of the task to make it run more or less often. However when you try to save the task it will ask you for the password of the account under which the task runs, the name of which looks something like ‘AAD_a6a4cefedc741’. It uses a random hex code at the end of the name so this could be slightly different to the example I’m using.

Modify AADSync Schedule

This account is used to run the AADSync service, is the account used to access the MIIS client database, and also to run the Scheduled Task. It is a local account which is created during the initial installation of AADSync, and the password is randomly generated. It may be tempting to change the password of this account, but please don’t. I have only come across this happening twice but both times have involved the internal database becoming completely inaccessible, meaning that the service simply won’t start, even with correct credentials.

If you really must change the frequency of your sync, create a new Scheduled Task and configure it to run the following application:

“C:\Program Files\Microsoft Azure AD Sync\Bin\DirectorySyncClientCMD.exe”

Ensure the Task is running with the highest possible priveleges and configure the task to use a user account which is a member of the following groups:

  • ADSyncAdmins
  • ADSyncBrowse
  • ADSyncOperators
  • ADSyncPasswordSet

This new task will run under whatever schedule you fancy, and for good measure you can disable the original task if you’d like. When dealing with default configuration items in any piece of software, I would always recommend creating cloned configurations rather than modifying the default, as it gives you a way to back out of changes and allows you to compare old with new.

Synchronised User not Provisioned in Lync Online

Today I came across an issue with Lync Online during an implementation of Office 365. We synchronised the user accounts and password hashes using AADSync and applied E3 licenses to a few test users in the IT team. Their testing was successful with the exception of Lync Online, which they could not login to with their synchronised @contoso.com account. The error returned to them was as follows:

Lync couldn't find a Lync Server for contoso.com. There might be an issue with the Domain Name System (DNS) configuration for your domain. Please contact your support team.

Lync Online DNS Error

I tested using my own testuser@contoso.com account and could successfully login to Lync Online internally and externally, which showed me that the DNS records were created correctly. After looking at the firewall logs it appeared that the DNS name lyncdiscoverinternal.contoso.com was being unsuccessfully looked up. This DNS record is used for On Premise Lync installations. I also noted that in the Office 365 Lync Admin centre, the user had not been provisioned.

After quizzing the customer, we discovered that a proof of concept Lync server had been installed a while back and enabled for a few test users in the IT department. This server was not properly decommissioned so AD attributes still existed for these test users. In order to get Lync Online to provision the accounts we had to go into ADSIEdit and clear the following attributes so that they were empty:

msRTCSIP-DeploymentLocator
msRTCSIP-FederationEnabled
msRTCSIP-InternetAccessEnabled
msRTCSIP-Line
msRTCSIP-OptionFlags
msRTCSIP-PrimaryHomeServer
msRTCSIP-PrimaryUserAddress
msRTCSIP-UserEnabled
msRTCSIP-UserPolicies

We then ran a delta sync and hey presto, our user accounts were provisioned in Lync Online and the users could successfully login. The error message we initially given was fairly misleading, but this just shows you how important proper decommissioning is when it comes to essential services like Lync and Exchange.

Modify UPN to match Primary SMTP address

 

You may have read a previous blog I wrote in which I explained a new feature in AADSync which allows you to use the Primary SMTP address as a username in Office 365, therefore allowing you to leave the UPN value unchanged. However this method is not without it’s problems, and during a recent Hybrid Deployment of Office 365 it was decided to instead change the UPN prefix and suffix to match the users Primary SMTP address.

The risk to be aware of here is that if one of your LOB applications or a Service Account is using the UPN for authentication, you will break authentication for this app/service.The customer in question was confident that this wasn’t the case, but just to be sure we changed the UPN for a few test users prior to rolling this change out.

To clarify, the users Pre-Windows 2000 and UPN logon prefixes were firstinitialsurname, for example, ECoates, whereas the Primary SMTP address was firstname.lastname@contoso.com, for example, Emily.coates@contoso.com. We wanted users to be able to login to Office 365 using their ’email address’, so we decided to change all UPNs to match. There are 2 commands that could be used to accomplish this. The end result of the commands was that the users UPN would be changed from ddixon@contoso.local to Emily.coates@contoso.com. We also only wanted to change to be made to user accounts with a Primary SMTP address specified.

Pre-Requisites

Either of the two below commands can be run to achieve this goal. However, before either of these commands are run, it is important to make sure that the UPN suffix you wish to be populated exists in AD Domains and Trusts. To do this, open AD Domains & Trusts, and right click on Properties.

ADDT

Then enter the required domain, such as contoso.com and click Add and Apply. This makes the alternate UPN suffix available for use in the domain.

Alternate UPN Suffix

The below commands need to be run from an elevated Exchange Management Shell prompt. It is up to you which to run.

Command 1

This command finds all users who have a Primary SMTP address and then sets the UPN to be identical to this. It will ignore any user who does not have a Primary SMTP Address.

$users = Get-User | Where {-Not [string]::IsNullOrEmpty($_.WindowsEmailAddress) } 
$users | ForEach {Set-User –Identity $_.Guid.ToString() –UserPrincipalName $_.WindowsEmailAddress.ToString()}

I found this command to be a little hit or miss, but this may be due to the size of the user base I was working with.

Command 2

This command finds all users who have the specified UPN suffix (such as a non-routable suffix like contoso.local) and changes their UPN to match their Primary SMTP address. If a user has no Primary SMTP address, it will not change the properties of that user.

$users = Get-User –Filter {(UserPrincipalName –like '*@contoso.local')} 
$users | ForEach {Set-User –Identity $_.Guid.ToString() –UserPrincipalName $_.WindowsEmailAddress.ToString()}

My personal preference is to use Command 2 as I had the most success with it, however I would love to hear your feedback on which one worked best for you!

Once you have run the command you can then use the following command to discover if any users were not successfully modified:

Get-User –Filter {(UserPrincipalName –like '*@contoso.local')}

Edit

I have also used the following command after having some problems with command 2 on Exchange 2013 (errors regarding the filter expression):

$mbxs = Get-Mailbox | Where {$_.UserPrincipalName -like '@contoso.local'}
$mbxs | ForEach {Set-User –Identity $_.Guid.ToString() –UserPrincipalName $_.WindowsEmailAddress.ToString()}

 

Mixed Server 2012 R2 / Server 2003 Domain Controller Environment & Kerberos Issues

A few months ago, the firm I work for had an issue with one of our implementations of Server 2012 R2. It was our 2nd install of Domain Controllers running this OS, and the environment we installed was experiencing Kerberos errors which caused intermittent login issues on servers and workstations. After much troubleshooting we were stumped, and the workaround of ‘reboot the server/client and you will be able to login again’ wasn’t exactly a satisfactory fix for ourselves or the client. We logged a call with Microsoft, who were also stumped by the problem as we were the first partner to report this. A registry workaround was given to us which solved the clients woes temporarily, but now a hotfix is available! We have successfully deployed this hotfix and it has cured the problem.

I won’t write a huge spiel about this problem as Microsoft have already done this for me, but if you are experiencing login issues on Windows 7, Server 2008 R2 and/or Server 2012 R2 and you have recently deployed 2012 R2 Domain Controllers in a 2003 domain environment, you may be suffering from this issue. Look in your DCs event logs for the following event:

Event ID: 4
Source: Kerberos
Type: Error
“The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server host/myserver.domain.com.  This indicates that the password used to encrypt the Kerberos service ticket is different than that on the target server. Commonly, this is due to identically named machine accounts in the target realm (domain.com), and the client realm.   Please contact your system administrator.”

Please follow this link to read more about the problem and to download the hotfix:

http://blogs.technet.com/b/askds/archive/2014/07/23/it-turns-out-that-weird-things-can-happen-when-you-mix-windows-server-2003-and-windows-server-2012-r2-domain-controllers.aspx