Using Time-based Partition Keys in #Azure Table Storage

In a previous post about storing Azure Storage Table entities in descending order I combined a time-based key with a guid in order to create a unique key. This is practical when you need to use combined keys for the Row Keys or Partition Key. But it’s not practical for logs.

A better solution for logs, is to generate a Partition Key based on time. This allows you to query for logs by time periods. There are many ways to generate time-based partitions, so I will cover the two that I use the most.

The first method comes in two flavors. It consists of taking the current time value in Ticks as the Partition Key value. Some round it up to the minute, which makes it possible to do batch inserts. Remember that batch inserts are limited to 100 entities and that they must all belong to the same table partition.

var time = new DateTime(2014,06,16,2,40,0,DateTimeKind.Utc);

// Some logging libraries add an extra zero in front of the ticks
var firstForm = "0" + time.Ticks;
// First Form: 0635384832000000000

// Other services use the ticks as-is
var secondForm = time.ToUniversalTime().Ticks;
// Second Form: 635384832000000000

The second method sorts entities in descending order. Building on my previous example, let’s calculate a Partition Key that’s rounded up to the minute.

var time = new DateTime(2014, 06, 16, 2, 40, 0, DateTimeKind.Utc);

var key = String.Format("{0:D19}", (DateTime.MaxValue - time.ToUniversalTime()).Ticks);
//key: 2519994143999999999

usin

This method, can be used to calculate the Partition Keys required to query against logs persisted by the Semantic Logging Application Block (SLAB). The following is a query against Azure Table Storage, that I would use to extract events that were persisted by SLAB between June 1 2014 at 5AM UTC and  June 1 2014 at 6AM UTC.

var fromTime = new DateTime(2014, 06, 1, 5, 0, 0, DateTimeKind.Utc);
var fromTicks = (DateTime.MaxValue - fromTime.ToUniversalTime()).Ticks;
var fromKey = fromTicks.ToString("d19", CultureInfo.InvariantCulture);

var from = TableQuery.GenerateFilterCondition("PartitionKey",
                                               QueryComparisons.LessThanOrEqual,
                                               fromKey);

var toTime = new DateTime(2014, 06, 1, 6, 0, 0, DateTimeKind.Utc);
var toTicks = (DateTime.MaxValue - toTime.ToUniversalTime()).Ticks;
var toKey = toTicks.ToString("d19", CultureInfo.InvariantCulture);

var to = TableQuery.GenerateFilterCondition("PartitionKey",
                                            QueryComparisons.GreaterThan,
                                            toKey);

var query = TableQuery.CombineFilters(from, TableOperators.And, to);

// Generated query string:
// (PartitionKey le '2520007019999999999') and (PartitionKey gt '2520006983999999999')

This query string can be used in Cerebrata’s Azure Management Studio or in Visual Studio to get the desired data set. It can also be used to retrieve data through the Azure Storage Client library.