Windows Azure - 清理 WADLogsTable

Question

我读过关于 Windows Azure 中的 DiagnosticMonitor 使用的 WADLogsTable 表是否会自动删除旧日志条目的相互冲突的信息。

我猜它不会，反而会永远增长——让我花钱。 :)

如果是这种情况，是否有人有一个很好的代码示例来说明如何手动从此表中清除旧日志条目？也许基于时间戳？我会定期从辅助角色运行此代码。

Answer 1

Windows Azure 诊断创建的表中的数据不会自动删除。

但是，Windows Azure PowerShell Cmdlet 包含专门针对这种情况的 cmdlet。

PSD：\>帮助清除-WindowsAzureLog

姓名
清除-WindowsAzureLog

概要
从存储帐户中删除 Windows Azure 跟踪日志数据。

语法
Clear-WindowsAzureLog [-DeploymentId ] [-From ] [-To ] [-StorageAccountName ] [-StorageAccountKey ] [-UseD
开发存储] [-StorageAccountCredentials] []

Clear-WindowsAzureLog [-DeploymentId ] [-FromUtc ] [-ToUt
c <日期时间>] [-StorageAccountName <字符串>] [-StorageAccountKey <字符串>]
[-UseDevelopmentStorage] [-StorageAccountCredentials ] [<通用参数>]

您需要指定-ToUtc参数，该日期之前的所有日志都将被删除。

如果需要在辅助角色内的 Azure 上执行清理任务，则可以重用 C# cmdlet 代码。 PowerShell Cmdlet 是根据许可的 MS 公共许可证发布的。

基本上，只需要 3 个文件，没有其他外部依赖项：DiagnosticsOperationException.cs、WadTableExtensions.cs、WadTableServiceEntity.cs。

Answer 2

更新了Chriseyre2000的功能。对于需要删除数千条记录的情况，这提供了更高的性能：通过 PartitionKey 搜索和分块分步过程。请记住，最好的选择是在存储附近（在云服务中）运行它。

public static void TruncateDiagnostics(CloudStorageAccount storageAccount, 
    DateTime startDateTime, DateTime finishDateTime, Func<DateTime,DateTime> stepFunction)
{
        var cloudTable = storageAccount.CreateCloudTableClient().GetTableReference("WADLogsTable");

        var query = new TableQuery();
        var dt = startDateTime;
        while (true)
        {
            dt = stepFunction(dt);
            if (dt>finishDateTime)
                break;
            var l = dt.Ticks;
            string partitionKey =  "0" + l;
            query.FilterString = TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.LessThan, partitionKey);
            query.Select(new string[] {});
            var items = cloudTable.ExecuteQuery(query).ToList();
            const int chunkSize = 200;
            var chunkedList = new List<List<DynamicTableEntity>>();
            int index = 0;
            while (index < items.Count)
            {
                var count = items.Count - index > chunkSize ? chunkSize : items.Count - index;
                chunkedList.Add(items.GetRange(index, count));
                index += chunkSize;
            }
            foreach (var chunk in chunkedList)
            {
                var batches = new Dictionary<string, TableBatchOperation>();
                foreach (var entity in chunk)
                {
                    var tableOperation = TableOperation.Delete(entity);
                    if (batches.ContainsKey(entity.PartitionKey))
                        batches[entity.PartitionKey].Add(tableOperation);
                    else
                        batches.Add(entity.PartitionKey, new TableBatchOperation {tableOperation});
                }

                foreach (var batch in batches.Values)
                    cloudTable.ExecuteBatch(batch);
            }
        }
}

Answer 3

您可以根据时间戳执行此操作，但这会非常低效，因为需要扫描整个表。下面是一个代码示例，可能有助于生成分区键以防止“全”表扫描。 http://blogs.msdn.com/b/avkashchauhan/archive/2011/06/24/linq-code-to-query-windows-azure-wadlogstable-to-get-rows-which-存储在特定日期时间之后.aspx

Answer 4

这是一个根据时间戳截断的解决方案。 （针对 SDK 2.0 进行测试）

它确实使用表扫描来获取数据，但如果每天运行一次就不会太痛苦：

    /// <summary>
    /// TruncateDiagnostics(storageAccount, DateTime.Now.AddHours(-1));
    /// </summary>
    /// <param name="storageAccount"></param>
    /// <param name="keepThreshold"></param>
    public void TruncateDiagnostics(CloudStorageAccount storageAccount, DateTime keepThreshold)
    {
        try
        {

            CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

            CloudTable cloudTable = tableClient.GetTableReference("WADLogsTable");

            TableQuery query = new TableQuery();
            query.FilterString = string.Format("Timestamp lt datetime'{0:yyyy-MM-ddTHH:mm:ss}'", keepThreshold);
            var items = cloudTable.ExecuteQuery(query).ToList();

            Dictionary<string, TableBatchOperation> batches = new Dictionary<string, TableBatchOperation>();
            foreach (var entity in items)
            {
                TableOperation tableOperation = TableOperation.Delete(entity);

                if (!batches.ContainsKey(entity.PartitionKey))
                {
                    batches.Add(entity.PartitionKey, new TableBatchOperation());
                }

                batches[entity.PartitionKey].Add(tableOperation);
            }

            foreach (var batch in batches.Values)
            {
                cloudTable.ExecuteBatch(batch);
            }

        }
        catch (Exception ex)
        {
            Trace.TraceError(string.Format("Truncate WADLogsTable exception {0}", ex), "Error");
        }
    }

Answer 5

这是我的 @Chriseyre2000 解决方案的稍微不同的版本，使用异步操作和 PartitionKey 查询。在我的例子中，它被设计为在辅助角色中连续运行。如果您有很多条目需要清理，这可能会更容易记忆。

static class LogHelper
{
    /// <summary>
    /// Periodically run a cleanup task for log data, asynchronously
    /// </summary>
    public static async void TruncateDiagnosticsAsync()
    {
        while ( true )
        {
            try
            {
                // Retrieve storage account from connection-string
                CloudStorageAccount storageAccount = CloudStorageAccount.Parse(
                    CloudConfigurationManager.GetSetting( "CloudStorageConnectionString" ) );

                CloudTableClient tableClient = storageAccount.CreateCloudTableClient();

                CloudTable cloudTable = tableClient.GetTableReference( "WADLogsTable" );

                // keep a weeks worth of logs
                DateTime keepThreshold = DateTime.UtcNow.AddDays( -7 );

                // do this until we run out of items
                while ( true )
                {
                    TableQuery query = new TableQuery();
                    query.FilterString = string.Format( "PartitionKey lt '0{0}'", keepThreshold.Ticks );
                    var items = cloudTable.ExecuteQuery( query ).Take( 1000 );

                    if ( items.Count() == 0 )
                        break;

                    Dictionary<string, TableBatchOperation> batches = new Dictionary<string, TableBatchOperation>();
                    foreach ( var entity in items )
                    {
                        TableOperation tableOperation = TableOperation.Delete( entity );

                        // need a new batch?
                        if ( !batches.ContainsKey( entity.PartitionKey ) )
                            batches.Add( entity.PartitionKey, new TableBatchOperation() );

                        // can have only 100 per batch
                        if ( batches[entity.PartitionKey].Count < 100)
                            batches[entity.PartitionKey].Add( tableOperation );
                    }

                    // execute!
                    foreach ( var batch in batches.Values )
                        await cloudTable.ExecuteBatchAsync( batch );

                    Trace.TraceInformation( "WADLogsTable truncated: " + query.FilterString );
                }
            }
            catch ( Exception ex )
            {
                Trace.TraceError( "Truncate WADLogsTable exception {0}", ex.Message );
            }

            // run this once per day
            await Task.Delay( TimeSpan.FromDays( 1 ) );
        }
    }
}

要启动该流程，只需从辅助角色的 OnStart 方法中调用此方法即可。

// start the periodic cleanup
LogHelper.TruncateDiagnosticsAsync();

Answer 6

如果您不关心任何内容，只需删除该表即可。 Azure 诊断只会重新创建它。

Answer 7

稍微更新了 Chriseyre2000 的代码：

• 使用 ExecuteQuerySegmented 而不是 ExecuteQuery

• 观察 TableBatchOperation 100 次操作的限制

• 清除所有 Azure 表

  public static void TruncateAllAzureTables(CloudStorageAccount storageAccount, DateTime keepThreshold)
  {
     TruncateAzureTable(storageAccount, "WADLogsTable", keepThreshold);
     TruncateAzureTable(storageAccount, "WADCrashDump", keepThreshold);
     TruncateAzureTable(storageAccount, "WADDiagnosticInfrastructureLogsTable", keepThreshold);
     TruncateAzureTable(storageAccount, "WADPerformanceCountersTable", keepThreshold);
     TruncateAzureTable(storageAccount, "WADWindowsEventLogsTable", keepThreshold);
  }
  
  公共静态无效TruncateAzureTable（CloudStorageAccount storageAccount，字符串aTableName，DateTime keepThreshold）
  {
     常量 int maxOperationsInBatch = 100;
     var tableClient = storageAccount.CreateCloudTableClient();
  
     var cloudTable = tableClient.GetTableReference(aTableName);
  
     var query = new TableQuery { FilterString = $"Timestamp lt datetime'{keepThreshold:yyyy-MM-ddTHH:mm:ss}'" };
     TableContinuationToken 延续Token = null;
     做
     {
        var queryResult = cloudTable.ExecuteQuerySegmented(query, continuationToken);
        ContinuationToken = queryResult.ContinuationToken;
  
        var items = queryResult.ToList();
        var batches = new Dictionary>();
        foreach（项目中的var实体）
        {
           var tableOperation = TableOperation.Delete(entity);
  
           if (!batches.TryGetValue(entity.PartitionKey, out var batchOperationList))
           {
              batchOperationList = new List();
              批次.Add(entity.PartitionKey,batchOperationList);
           }
  
           var batchOperation = batchOperationList.FirstOrDefault(bo => bo.Count < maxOperationsInBatch);
           if (batchOperation == null)
           {
              批处理操作 = new TableBatchOperation();
              批量操作列表.Add(批量操作);
           }
           批量操作.Add(表操作);
        }
  
        foreach (var 批量批量.Values.SelectMany(l => l))
        {
           cloudTable.ExecuteBatch(批处理);
        }
     while (continuationToken!= null);
  }