Sitecore Corner

TL;DR; Clear your non-self-signed certificates from the Trusted Root Store

Yesterday after I started to slowly recover from my jet lag (yayyy #SitecoreSYM was awesome !) I decided to start installing the official release of Sitecore 9.

I downloaded all the required goodies from the Development Portal, imported the SIF modules, fixed my configurations, etc., the usual stuff 🙂

Around 30-45 minutes later I got my shiny brand new Sitecore 9 running together with his new pal xConnect. I browsed for a little bit and noticed it is running a bit slow, so as every Sitecore developer I went to check what is going on in the logs. I opened the logs folder and noticed my log is over 1 MB … that is never a good sign. After opening the log I noticed tons of the following exceptions:

Exception: Sitecore.XConnect.XdbCollectionUnavailableException
Message: The HTTP response was not successful: Forbidden
Source: Sitecore.Xdb.Common.Web
at Sitecore.Xdb.Common.Web.Synchronous.SynchronousExtensions.SuspendContextLock[TResult](Func`1 taskFactory)
at Sitecore.XConnect.Client.XConnectSynchronousExtensions.SuspendContextLock(Func`1 taskFactory)
at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.Initialize(XmlNode configNode)
at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert, IFactoryHelper helper)
at Sitecore.Configuration.DefaultFactory.CreateObject(XmlNode configNode, String[] parameters, Boolean assert)
at Sitecore.Configuration.DefaultFactory.CreateObject(String configPath, String[] parameters, Boolean assert)
at Sitecore.XConnect.Client.Configuration.SitecoreXConnectClientConfiguration.GetClient(String clientConfigPath)
at Sitecore.Analytics.Processing.AsyncPoolScheduler`2..ctor(IAsyncProcessingPool`1 pool, IXdbContextFactory xdbContextFactory, ExpandOptions options, Int16 maxBatchSize)
at Sitecore.Analytics.Aggregation.InteractionAggregationAgent.d__15.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Analytics.Core.Agent.d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Analytics.Core.AsyncBackgroundService.d__22.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Sitecore.Analytics.Core.AsyncBackgroundService.d__eNext()

Of course, having tons of these means that my xConnect connection was unhealthy. I know that Sitecore communicates with xConnect over SSL and that is uses certificate thumbprints to validate the certificate, so the first thing to check was if the ConnectionStrings thumbprint matches the xConnect one. Unlucky for me – it did…

As a 21st century developer my standard follow up was google. The first result seemed very promising – All about xConnect Security by Kam Figy. I suggest to everyone interested in how the xConnect security works to read it ! Sadly, after reading it I didn’t find what I was looking for as I didn’t have any of the 4XX errors symptoms mentioned in the post.

After that I asked our local SSL and Sitecore Expert Charlie Turano if he can give me a hand. After an hour and a half long troubleshooting session I felt bad from keeping him from the cool stuff he is doing and decided to try to look for help elsewhere.

So my next stop was #sif Sitecore Community Slack channel where Jon David from Sitecore responded to my call and started troubleshooting the issue with me. I think this is the place where I should thank Jon for spending his entire day trying to help me – this is what I call customer service! Meanwhile Chris Wojciech messaged me on Slack that he is having the same problem and he is currently working on it, so I felt a bit better that someone else is also trying to resolve it !

To sum it up – after nearly 5 hours of headbanging with mixed results I got a little tip from Hedgehog`s Infrastructure Specialist to take a look into the IIS logs for the xConnect website. I was aware that I was getting 403`s, but it was not until I opened the IIS logs when I saw I was actually getting 403.16 errors when Sitecore was trying to communicate with xConnect. Long story short – googled 403.16 and the first result was this stackoverflow article – HTTP error 403.16 – client certificate trust issue

I then executed the powershell mentioned in the article to see if I have any non-self signed certificates in my Trusted Root and it turned out I had a couple (generated by me during some of my tests). So I removed them and clicked F5 feeling skeptical because I tried at least 30 things during the day that didn’t work.

Still with fingers crossed, I opened the log with my tailing software and I waited for the dreadful Trying to load XML configuration /App_Config/Security/GlobalRoles.config line after which everything starts crashing.

So I waited for some time … and there was no exceptions in the logs !  Experience analytics even started synchronizing segments and my site was loading and behaving normally !

Let’s sum it up – it seems that Windows doesn’t like non-self signed certificates in its Trusted Root Certificates Store and you need to remove them in order to be able to run your shiny new xConnect applications locally !

Hope this saves you guys the combined total of 20 hours shared between 3 persons (that I know of) in resolving this !

Happy Sitecore nining ! 🙂