I recently ran into a problem with a Skype for Business 2015 on-premises deployment where certain services randomly failed to start. After successfully deploying and testing the services it was time to patch the OS and reboot. This is where things became interesting.
As you can see from the screen grab below there were specific services like the Conferencing Announcement and Response Group services did not want to start. The Skype for Business Front End service did start but was taking an extremely long time. I attempted to manually start the stopped services but did not have any luck.

My first stop in troubleshooting the problem was to have a look at the event logs. The system log showed a timeout as the cause of the service not wanting to start. This made sense as the Skype For Business Front End service was taking an extremely long time to start. Unfortunately, the event logs were not very helpful in finding the root cause. At first glance, the services which failed to start have little in common. One commonality I did see was the location of the servers and the use of Read-Only Domain Controllers (RODC’s).
At this point, I switched over to using the native Skype for Business CLS Logging tool to pull detailed logs. Before choosing a component to log I sat down and thought about the dependencies for the failed services. The conferencing and response group related services rely on the RTC Service global settings container which is located in a distant central office.

 
To start I added ADConnect as a component to log. I then attempted to start the Skype for Business Response Group service with the logging running. Once the service timed out I pulled up the logs. From the logs, it became apparent that Skype was cycling through a number of RODC’s within the local AD site in search of some required attribute.
The Skype Server Try to Locate a Global Catalog/Domain Controller:

 TL_WARN(TF_COMPONENT) [0]423C.376C::11/18/2017-00:57:19.412.ffffffff (ADConnect,ADConnection.AnalyzeDirectoryError:adconnection.cs(692)) Caught LdapExceptionQ with 7143535(0x), message= TL_ERROR(TF_COMPONENT) [0]423C.376C::11/18/2017-00:57:19.412.ffffffff (ADConnect,SuitabilityVerifier.CreateConnectionAndBind:suitabilityverifier.cs(143)) BindingFailedTo fqdn: server.domain.com error:6357090 port:3014764 message:omn TL_ERROR(TF_COMPONENT) [0]423C.376C::11/18/2017-00:57:19.412.ffffffff (ADConnect,DirectoryServicesTopologyProvider.FindFirstSuitableDomainController:directoryservicestopologyprovider.cs(770)) Domain Controller server.domain,com Message-“The LDAP server is unavailable.” LdapError-“ServerDown” ” server.domain.com “:”389” was found not suitable. Will try to find another DC in the domain. Error: server.domain.com * in site Remote Site

Going Through List of Read-Only Domain Controllers in the AD Site:

TL_INFO(TF_COMPONENT) [0]423C.376C::11/18/2017-00:57:19.412.ffffffff (ADConnect,SuitabilityVerifier.IsServerSuitableIgnoreExceptions:suitabilityverifier.cs(46)) Trying to find if DC server.domain.com is suitable  TL_INFO(TF_COMPONENT) [1]423C.376C::11/18/2017-00:57:20.149.ffffffff (ADConnect,SuitabilityVerifier.IsServerSuitable:suitabilityverifier.cs(91)) Created a connection to DC server.domain.com TL_INFO(TF_COMPONENT) [1]423C.376C::11/18/2017-00:57:20.149.ffffffff (ADConnect,SuitabilityVerifier.IsOperatingSystemSuitable:suitabilityverifier.cs(170)) Checking if operating system is suitable for DC server.domain.com TL_ERROR(TF_COMPONENT) [2]423C.376C::11/18/2017-00:57:20.394.ffffffff (ADConnect,SuitabilityVerifier.LogRodcFoundEvent:suitabilityverifier.cs(458)) OsSuitability server.domain.com TL_ERROR(TF_COMPONENT) [2]423C.376C::11/18/2017-00:57:20.395.ffffffff (ADConnect,DirectoryServicesTopologyProvider.FindFirstSuitableDomainController:directoryservicestopologyprovider.cs(770)) Domain Controller server.domain.comlErrorIsServerSuitableRODC server.domain.com was found not suitable. Will try to find another DC in the domain. Error:

 
Why was it not chasing a referral to a domain controller that had the answer? DNS lookups and telnet confirmed name resolution and connectivity were not a problem. The customer’s Active Directory Team confirmed they had healthy replication across the environment. They also added additional replication links in the AD topology between the local sites and the forest root. None of this seemed to make a difference and it was apparent Skype was not leaving the local AD site for answers.
It was apparent that Skype, unlike an Exchange server, is not site aware. I needed a way of making Skype bypass the RODC’s in the local site and go directly to the forest root. From the start of my career, I recalled an old NT 4.0 setting in the registry called SetPrfDc. This was used to statically assign the login domain controller. That particular setting had long since become obsolete for good reason. I started looking for something similar to the old setting. I discussed the problem with one of my colleagues and he found an old article referring to a similar problem with a Lync 2010 appliance. You can read about it here – https://trogjels.wordpress.com/2013/05/08/lync-2010-sba-with-rodc-how-to-get-it-work/
The proposed fix was to add the local server IP address to the forest root in AD Sites and Services. The customer’s AD team was understandably reluctant to make the change. This prompted a call to Microsoft who recommended a simple change to the server that ultimately resolved the problem. They pointed me to a registry setting called SiteName. We made the recommended change and following a reboot, the services started as expected.
 

You can read more about the fix on Technet – https://technet.microsoft.com/en-us/library/cc937923.aspx.

Perficient partners with Integrated Research to deliver their UC Assessor and Path Insight detailed network and environmental analysis. UC Assessor is a cloud-based tool that is designed to work with not only Skype for Business, but many different vendors such as AudioCodes, Avaya, and Cisco.
So, why conduct an advanced network assessment? Here are a few reasons why you should consider a professional services engagement with Perficient as a partner:
 

1. You have conducted a basic network assessment and found areas where your metrics failed to meet the target values.
2. You’re experiencing problems with audio, video, conferencing and sharing when using Skype for Business either online or on-premises.
3. You’ve made significant changes to the environment such as network upgrades, or growth and acquisition.
4. You are planning to move to Office 365 and need to ensure the best experience possible.

The solution consists of a software agent on a Windows PC or server which is used to test, or stress the network with simulated media traffic. The data is collected and collated online in the portal where the analysis. You can see some of the sample data the tool generates below.

 

You can see from the images above that IR’s UC Assessor is running an analysis based on a learned path. It is evaluating each network device in the path (in this case routers) to determine if they provide the expected level of service. It can also analyze the Skype for Business Online server roles. You can also evaluate an on-premises Skype for Business or Lync Server deployment by loading the topology file.
In the prior images, you can clearly see all devices in the path to Skype Online showing green. This shows they are all meeting the expected level of service. You can drill down into any individual hop and pull more detailed information. If you control those devices like an on-premises router or Lync/Skype server you can pull up an even greater level of detail.
The value that Perficient provides is our deep understanding of unified communications. This allows us to turn the network analysis information into actionable data. You want to learn more about what Perficient can do for you on our website – https://www.perficient.com/

In my prior blog, I covered how to automate the process of running a Skype for Business Online network assessment. You can read about it here.
A network assessment using the free Microsoft tool generates a CSV file which is used to record the results. You can download the assessment tool from here.
We will be using Excel and some simple graphs to illustrate the results.
Let’s start by looking at our performance targets provided by Microsoft. For the best experience, we obviously want to meet or exceed the published metrics.

 
We will use the target values and the actual values in Excel to graph the results of each metric in an easy to digest format. Let’s start with results file generated when I ran a quick analysis:

 
The sample data was taken from a computer connected to the Microsoft Edge, meaning I am not going through an on-premises Lync or Skype for Business Edge server. It was connected to the Internet via Wifi to a home router. The goal for me is to meet or exceed the “Internal Client to Microsoft Edge” target metrics. Granted, the data consists of a very small sample size. Normally a network analysis is run through an entire workweek before the data is collated and analyzed. I will be graphing actual data taken over the course of a week.
 
It is clear from the graph that my home network did quite well on round-trip latency as the all measured values fall below the target line shown in orange.
 

Moving to the next measured value, packet loss rate you can see once again the network exceeded the target values. The blue line for the actual measured value is not displayed as it was zero.

The last measured value is packet reorder ratio. Once again, my home network did quite well when measured against the target value. The blue line for the actual measured value is not displayed as it was zero.

 
You may have not have noticed, Microsoft publishes some additional values like burst packet loss which the tool isn’t capable of capturing. Likewise, the free tool cannot deep dive into the root cause of any item outside the recommended values.
What do you do if you want a deeper analysis, or your measured values do not meet the requirements? This is where Perficient can assist you with a professional services engagement. Perficient has partnered with Integrated Research better known as IR to deliver Skype for a Business Online network assessments. The network assessments are run using IR’s UC Assessor tool to deliver outstanding value. You can read more about IR’s UC Assessor here – http://www.ir.com/skype-for-business-assessment
In my next blog post, I’ll cover using IR’s UC Assessor and how it delivers exceptional value as part of a network assessment services engagement.

My colleague Jeff Brown has already covered the reason for running a network assessment. He also covered how to run the assessment. I’d now like to show you how you can automate the process and scale it so that you can collect data from multiple end-points, or even sites. You can read more about Jeff’s network assessment info on the Perficient blog:
https://blogs.perficient.com/microsoft/2017/02/skype-online-why-you-need-a-network-assessment/
https://blogs.perficient.com/microsoft/2017/03/skype-online-how-to-run-a-network-assessment/
The free Skype for Business Network Assessment Tool measures network performance and network connectivity. Like most free tools, this one is limited in functionality. It does measure latency, packet loss, jitter, and packet reorder percentage all of which are important. Some downsides to the free tool include the following:

1. The requirement for an end-user to start the process.
2. The ability to run only a single assessment due to the fixed output file which is overwritten.
3. The need to modify a cofig file to change options.

So how do we overcome these limitations? By using the Windows Task Scheduler and a bit of PowerShell scripting we will automate the assessment.
Let’s start by downloading the tool and having a look. You can download the Skype for Business Network Assessment Tool from here – https://www.microsoft.com/en-us/download/details.aspx?id=53885.
The download is a single install file and there is no option to pick the install directory. The files are extracted to the “Program Files\Microsoft Skype for Business Network Assessment Tool” directory. You can copy the files to a different location or go with the default. You can also copy the extracted files to different computers.
Prior to running the tool, you need to get a better understanding of your user community. How does your company use Skype for Business or Teams? There are a lot of basic questions you want to ask. Here are just a few:

1. Do we use audio, video, conferencing or PSTN calling?
2. Is there a Quality of Service (QoS) policy and support in place?
3. What are the hours of operation, and the number of sites that need to be evaluated?

Let’s start by looking at the config file. Look for the file named NetworkAssessmentTool.exe.config.
Open the file in Notepad, the values we want to modify are the “NumIterations” and the IntervalInSeconds. Assuming we want to run an assessment every 30 minutes over a 12-hour period we need to set the NumIterations to 24 and the IntervalSeconds to 1800.

<add key="NumIterations" value="24"/>
 <add key="ResultsFilePath" value="C:\Networkassessment\performance_results.tsv"/>
 <add key="Delimiter" value=" "/>
 <add key="IntervalInSeconds" value="1800"/>
 <add key="OutputFilePath" value="C:\Networkassessment\connectivity_results.txt"/>

 
This covers us for a single day. What happens if you want to you want to run this tool on more than one day, say Monday through Friday? The output file name is configurable but remains static based on the config file. The next run of the analysis tool will overwrite the current data file which means you could lose the prior day’s data. This solution is obviously not scalable and requires a lot of manual intervention. This is where we can leverage the Windows Task Scheduler and a bit of PowerShell to automate the process.
Let’s have a look at the PowerShell Script. It is used to start the network assessment and rename the output file prior to the next iteration. The script is used to invoke the network assessment tool and then rename the output when done.

<#
===================================================
.NOTES
 Organization: Perficient Inc.
 Filename: NetworkAssessment.ps1
 ===================================================
PowerShell script to automate the Microsoft Network Assessment Tool
Step 1 - Create NetworkAssessment folder at root of C:
Step 2 - Copy network assessment PowerShell script to install directory of the network assessment tool
Step 3 - Create scheduled task to run the PowerShell(AS SHOWN BELOW)
Step 4 - Modify network assessment config file parameters
powershell -nologo -noninteractive -command & ("{C:\Program Files\Microsoft Skype for Business Network Assessment Tool\NetworkAssessment.ps1}")
#>
<#
Start the Network Assessment Process
Wait for process to complete before renaming the file in order to prevent overwriting prior data
#>
push-location "C:\Program Files\Microsoft Skype for Business Network Assessment Tool";
$exe = "NetworkAssessmentTool" 
$proc = (Start-Process $exe -PassThru)
$proc | Wait-Process
<#
Rename output file
#>
Get-Item C:\Networkassessment\performance_results.* | Rename-Item -newname {$_.LastWriteTime.toString("yyyy.MM.dd.ThhmmZ") + ".tsv"}

 
Next, we need to automate the network assessment so that is runs on a schedule. This is where the Windows Task Scheduler plays its role. We need to invoke PowerShell to start the assessment and then re-name the output file.

1. Start by creating a folder called Network Assessment. Next, create a scheduled task and give it a name.
2. Create the task and set it to run under an account that has administrator permissions on the local computer. Select the option to run whether a user is logged in or not.
3. Select the Triggers tab and specify the start and end date and time for running an assessment. Be sure to check the box to enable the trigger.
4. Select the Actions tab and click the New… button. This is where we execute the PowerShell script to run the assessment. The program to run is PowerShell.exe. For the add arguments field you need to use the following: “& ‘C:\Program Files\Microsoft Skype for Business Network Assessment Tool\NetworkAssessment.ps1′”
5. Lastly, you will want to wake the computer if it goes into a sleep, or power saving mode. Ideally, you want to exclude the computer from any group policies that prevent the assessment from running as expected.

Once the task is executed for the first what can you expect? How do you know it is working? You will need to check the output file directory from the config file. Based on the configuration file, we should see a file called performance_results.tsv. It will then be renamed to the date and time format following the completion of the daily network assessment.
If you open the file in Excel you should see some data being collected. I will cover how to make sense the data on a different blog.