Handling Sitecore Multi-site Instance robots.txt

When dealing with Sitecore multi-site instances SEO can be a problem. One of the biggest challenges is to have a separate robots.txt file for each site. Fortunately in Sitecore there is an option to write a custom request processor which can deal with the matter.

Step 1 – Creating a Base Template

The first step is creating a template with one field which will store the robots.txt file contents.

Generic Robots Settings

The field type could be Multi-Line Text or Rich Text Editor – depending on the personal preferences (I prefer to use Multi-Line Text, because I don`t want html to accidentally appear in the robots file).

Generic Robots Settings Fields

Add Standard Values and set a default value for the robots.txt file. When using Sitecore it is important to prevent robots from crawling the”/sitecore” path, so a good default value for the robots is:


User-agent: *
Disallow: /sitecore

For development environments the value can be set to:


User-agent: *
Disallow: /

This way the crawlers won`t crawl the site at all, reducing the chance of leaking something important.

After the template is ready it should be inherited across all the different Home Page Templates (the templates that are used for the startItems in the sites configuration). Please note that the best practice when dealing with multi-site instances is to have some type of Generic Home Page Template which is inherited across the Concrete Home Page Templates. In this situation just inherit the template in the Generic Home Page Template.

After it is all set in Sitecore – it’s time for coding !

Step 2 – Creating a custom HttpRequestProcessor

Create a custom HttpRequestProcessor with the following code.


namespace Sandbox.Processors
{
    using Sitecore.Data.Items;
    using Sitecore.Pipelines.HttpRequest;
    using System;
    using System.Web;

    public class RobotsTxtProcessor : HttpRequestProcessor
    {
        public override void Process(HttpRequestArgs args)
        {
             HttpContext context = HttpContext.Current;

             if (context == null)
             {
                 return;
             }

             string requestUrl = context.Request.Url.ToString();

             if (string.IsNullOrEmpty(requestUrl) || !requestUrl.ToLower().EndsWith("robots.txt"))
             {
                 return;
             }

             string robotsTxtContent = @"User-agent: *"
                                       + Environment.NewLine +
                                       "Disallow: /sitecore";

             if (Sitecore.Context.Site != null && Sitecore.Context.Database != null)
             {
                  Item homeNode = Sitecore.Context.Database.GetItem(Sitecore.Context.Site.StartPath);

                  if (homeNode != null)
                  {
                      if ((homeNode.Fields["Site Robots TXT"] != null) &&
                          (!string.IsNullOrEmpty(homeNode.Fields["Site Robots TXT"].Value)))
                      {
                          robotsTxtContent = homeNode.Fields["Site Robots TXT"].Value;
                      }
                  }
             }

             context.Response.ContentType = "text/plain";
             context.Response.Write(robotsTxtContent);
             context.Response.End();
         }
    }
}

The code is pretty straightforward. If the requested URL has robots.txt in it – the request is intercepted and the value from the custom field is sent as a response. Also the processor assures that there will always be a default value for the robots.txt in case something goes wrong or the field was left blank.

Step 3 – Creating the custom configuration

The configuration file needs to register the new HttpRequestProcessor and it must allow txt files handling in the Sitecore.Pipelines.PreprocessRequest.FilterUrlExtensions processor.

The sample configuration can be seen bellow.


<?xml version="1.0" encoding="utf-8" ?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <preprocessRequest>
        <processor type="Sitecore.Pipelines.PreprocessRequest.FilterUrlExtensions, Sitecore.Kernel">
          <param desc="Allowed extensions (comma separated)">aspx, ashx, asmx, txt</param>
        </processor>
      </preprocessRequest>
      <httpRequestBegin>
        <processor type="Sandbox.Processors.RobotsTxtProcessor, Sandbox"
                   patch:before="processor[@type='Sitecore.Pipelines.HttpRequest.UserResolver, Sitecore.Kernel']"/>
      </httpRequestBegin>
    </pipelines>
  </sitecore>
</configuration>

And that`s it ! Now you can have a separate robots file for each of your sites and you are one step closer in being SEO Friendly !

For another implementation with ASP.NET HTTP Handler you can refer to this blog post: http://sitecoreclimber.wordpress.com/2014/07/27/sitecore-multisite-robots-txt/

Sitecore Custom Logging

Every Sitecore Developer faces a situation in which he will need to build a custom logging mechanism. One of the most common scenarios would be a multi-site solution in which the standard log will become huge and finding the exact trace information becomes painful.

On the bright side Sitecore built-in logging uses log4net and offers different ways to extend the logging process. In this post I will cover the easiest approach for dealing with custom logs.

In the web.config just beneath the <sitecore> node you can find the log4net configuration section. After examining it you will notice that you must do two things in order to create a new logger:

    • create a new appender that will append to a file of your choice
    • register a new logger that is going to use this appender and can be referenced in the code later on.

So here are the 3 easy steps to get your brand new logger up and running.

Step 1 – Configuring a custom appender

Configuring the custom appender is as easy as copying and pasting one of the existing sections and modifying it to suit the new logger`s needs.


 <appender name="MyCustomLogFileAppender" type="log4net.Appender.SitecoreLogFileAppender, Sitecore.Logging">
 <file value="$(dataFolder)/logs/mycustomlogfile.{date}.txt" />
 <appendToFile value="true" />
 <layout type="log4net.Layout.PatternLayout">
 <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
 </layout>
 <encoding value="utf-8" />
 </appender>

You can notice that it almost entirely reuses the standard LogFileAppender. The two changes are the appender name and the name of the file it will append to.

Step 2 – Configuring a custom logger

Configuring the custom logger is as simple as configuring the custom appender.


 <logger name="Sitecore.Diagnostics.MyCustomLogger" additivity="false">
 <level value="INFO" />
 <appender-ref ref="MyCustomLogFileAppender" />
 </logger>

Just make sure that the appender-ref matches your custom appender.

Step 3 – Creating a wrapper class to work with your custom logger

When dealing with custom logger it will be nice to have a way to log in the same manner as we do with the Sitecore.Diagnostics.Log. For this purpose we can build a class that covers all the functionality that the Sitecore.Diagnostics.Log offers.


using log4net;
using Sitecore.Diagnostics;
using System;

namespace Sandbox.Diagnostics
{
public static class Log
{
private static readonly ILog _logger = log4net.LogManager.GetLogger("Sitecore.Diagnostics.MyCustomLogger");

public static void Debug(string message)
{
Assert.ArgumentNotNull(message, "message");

_logger.Debug(message);
}

public static void Debug(string message, Exception exception)
{
Assert.ArgumentNotNull(message, "message");
Assert.ArgumentNotNull(exception, "exception");

_logger.Debug(message, exception);
}

public static void Info(string message)
{
Assert.ArgumentNotNull(message, "message");

_logger.Info(message);
}

public static void Info(string message, Exception exception)
{
Assert.ArgumentNotNull(message, "message");
Assert.ArgumentNotNull(exception, "exception");

_logger.Info(message, exception);
}

public static void Warn(string message)
{
Assert.ArgumentNotNull(message, "message");

_logger.Warn(message);
}

public static void Warn(string message, Exception exception)
{
Assert.ArgumentNotNull(message, "message");
Assert.ArgumentNotNull(exception, "exception");

_logger.Warn(message, exception);
}

public static void Error(string message)
{
Assert.ArgumentNotNull(message, "message");

_logger.Error(message);
}

public static void Error(string message, Exception exception)
{
Assert.ArgumentNotNull(message, "message");
Assert.ArgumentNotNull(exception, "exception");

_logger.Error(message, exception);
}

public static void Fatal(string message)
{
Assert.ArgumentNotNull(message, "message");

_logger.Fatal(message);
}

public static void Fatal(string message, Exception exception)
{
Assert.ArgumentNotNull(message, "message");
Assert.ArgumentNotNull(exception, "exception");

_logger.Fatal(message, exception);
}
}
}

Please keep in mind that for using the log4net functionality you will need to add reference to the Sitecore.Logging assembly in your project.

For more complex implementations you can refer to Alex Shyba`s Chain on the topic.

Hope this helps and happy logging !

Sitecore Sublayout Parameter Templates

I found out that many Sitecore Developers never used Sublayout Parameter Templates, so I decided to make a post about them.

Imagine that you have a situation in which you need to pass items from different datasources to a single stand alone sublayout. You obviously cannot use the Sublayout Datasource parameter and the content editors (especially new ones) might find the whole concept of adding Item IDs to the additional parameters a bit confusing. In such situations the Sublayout Parameter Templates come in handy.

Here are the simple steps that you need to execute in order to leverage the Sublayout Parameter Templates.

Step 1 – Creating a new template

Create a new template that uses “/System/Layout/Rendering Parameters/Standard Rendering Parameters” as a base template.

My First Rendering Parameters Template

Step 2 – Creating the template fields

Create the template fields the same way you will do with any normal template. By using Standard Rendering Parameters as a base you won`t lose any of the Standard Template behavior your are used to.

My First Rendering Parameters Template Fields

 

Step 3 – Attaching the new template to the sublayout

Attach your new template by using the “Parameters Template” field on your Sublayout.

Attaching to a sublayout

Step 4 – Try if it works and add some data!

To add some data you just need to add the Sublayout to the presentation details and edit its settings.

My First Rendering Parameters Template Datasource

Step 5 –  Handle it in code!


Sublayout sublayout = Parent as Sublayout;
NameValueCollection sublayoutParameters = Sitecore.Web.WebUtil.ParseUrlParameters(sublayout.Parameters);

string title = sublayoutParameters["My Component Title"];
Item firstDatasource = Sitecore.Context.Database.GetItem(Sitecore.Data.ID.Parse(sublayoutParameters["My First Datasource"]));

And you are all set !

When dealing with Sublayout Parameter Templates (or Sitecore in General) – please consider using ORMs like Glass Mapper to ease your work.