Execution Analytics

This example demonstrates how to use the ExecutionAnalyticsPlugin to automatically track agent execution performance, providing detailed insights into agent operations.

Overview

The execution analytics example shows how to:

• Automatically track agent execution performance
• Monitor AI provider calls and tool executions
• Analyze performance metrics and error tracking
• Access analytics through different methods
• Clear analytics data when needed

Code Example

import { Robota, ExecutionAnalyticsPlugin } from '@robota-sdk/agent-core';
import { OpenAIProvider } from '@robota-sdk/agent-provider-openai';
import OpenAI from 'openai';
import dotenv from 'dotenv';

// Load environment variables
dotenv.config();

/**
 * Example: Execution Analytics Plugin
 *
 * This example demonstrates how to use the ExecutionAnalyticsPlugin
 * to automatically track agent execution performance.
 *
 * The plugin automatically tracks:
 * - Agent run executions
 * - AI provider calls
 * - Tool executions
 * - Performance metrics
 * - Error tracking
 */

async function main() {
  console.log('🔍 Execution Analytics Plugin Example');
  console.log('=====================================\n');

  // Validate API key
  const apiKey = process.env.OPENAI_API_KEY;
  if (!apiKey) {
    throw new Error('OPENAI_API_KEY environment variable is required');
  }

  // Create ExecutionAnalyticsPlugin for automatic tracking
  const analyticsPlugin = new ExecutionAnalyticsPlugin({
    maxEntries: 100,
    trackErrors: true,
    performanceThreshold: 2000, // 2 seconds
    enableWarnings: true,
  });

  // Create OpenAI client
  const openaiClient = new OpenAI({ apiKey });

  // Create agent with analytics plugin
  const agent = new Robota({
    provider: 'openai',
    model: 'gpt-4o-mini',
    currentModel: 'gpt-4o-mini',
    systemMessage: 'You are a helpful assistant that provides detailed responses.',
    aiProviders: {
      openai: new OpenAIProvider({
        client: openaiClient,
        model: 'gpt-4o-mini',
      }),
    },
    plugins: [analyticsPlugin], // Plugin will automatically track all executions
  });

  console.log('🤖 Agent created with ExecutionAnalyticsPlugin');

  // Method 1: Access plugin through agent (recommended)
  console.log('\n📊 Method 1: Access through agent.getPlugin()');

  try {
    // Execute some tasks - plugin automatically tracks everything
    const queries = ['What is AI?', 'Tell me about ML.']; // Minimal queries

    for (let i = 0; i < queries.length; i++) {
      console.log(`Executing task ${i + 1}...`);
      await agent.run(queries[i]);
    }

    // Access analytics through the agent's plugin
    const pluginFromAgent = agent.getPlugin('ExecutionAnalyticsPlugin');
    if (pluginFromAgent && 'getAggregatedStats' in pluginFromAgent) {
      const stats = (pluginFromAgent as any).getAggregatedStats();
      displayAnalytics('Agent Plugin Access', stats);
    }

    // Also demonstrate getting raw data
    if (pluginFromAgent && 'getData' in pluginFromAgent) {
      const rawData = (pluginFromAgent as any).getData();
      console.log(`\n📋 Raw execution data: ${rawData.length} entries`);

      // Show last execution details
      if (rawData.length > 0) {
        const lastExecution = rawData[rawData.length - 1];
        console.log(
          `   └─ Last execution: ${lastExecution.operation} (${lastExecution.duration}ms, ${lastExecution.success ? 'success' : 'failed'})`,
        );
      }
    }

    // Demonstrate plugin status
    if (pluginFromAgent && 'getStatus' in pluginFromAgent) {
      const status = (pluginFromAgent as any).getStatus();
      console.log(`\n🔍 Plugin Status:`);
      console.log(`   ├─ Name: ${status.name}`);
      console.log(`   ├─ Version: ${status.version}`);
      console.log(`   ├─ Enabled: ${status.enabled}`);
      if ('totalRecorded' in status) {
        console.log(`   ├─ Total Recorded: ${status.totalRecorded}`);
        console.log(`   ├─ Active Executions: ${status.activeExecutions}`);
        console.log(`   └─ Memory Usage: ${status.memoryUsage} items`);
      }
    }
  } catch (error) {
    console.error('❌ Task execution failed:', error);

    // Even on error, analytics are tracked
    const pluginFromAgent = agent.getPlugin('ExecutionAnalyticsPlugin');
    if (pluginFromAgent && 'getAggregatedStats' in pluginFromAgent) {
      const stats = (pluginFromAgent as any).getAggregatedStats();
      console.log(
        `\n📊 Analytics after error (${stats.totalExecutions} total, ${stats.failedExecutions} failed)`,
      );
    }
  }

  // Method 2: Direct plugin access (for comparison)
  console.log('\n📊 Method 2: Direct plugin access');
  const directStats = analyticsPlugin.getAggregatedStats();
  displayAnalytics('Direct Plugin Access', directStats);

  // Demonstrate clearing analytics
  console.log('\n🧹 Clearing analytics data...');
  analyticsPlugin.clearStats();
  const clearedStats = analyticsPlugin.getAggregatedStats();
  console.log(`   └─ After clearing: ${clearedStats.totalExecutions} executions`);

  // Cleanup
  await agent.destroy();
  process.exit(0);
}

function displayAnalytics(method: string, stats: any) {
  console.log(`\n📊 ${method} Analytics:`);
  console.log(`   ├─ Total Executions: ${stats.totalExecutions}`);
  console.log(`   ├─ Successful: ${stats.successfulExecutions}`);
  console.log(`   ├─ Failed: ${stats.failedExecutions}`);
  console.log(`   ├─ Success Rate: ${(stats.successRate * 100).toFixed(1)}%`);
  console.log(`   ├─ Average Duration: ${stats.averageDuration.toFixed(0)}ms`);
  console.log(`   └─ Total Duration: ${stats.totalDuration}ms`);

  // Show operation breakdown
  if (Object.keys(stats.operationStats).length > 0) {
    console.log(`\n   Operation Breakdown:`);
    Object.entries(stats.operationStats).forEach(([operation, opStats]: [string, any]) => {
      console.log(
        `   ├─ ${operation}: ${opStats.count} calls (${opStats.averageDuration.toFixed(0)}ms avg)`,
      );
    });
  }

  // Show errors if any
  if (Object.keys(stats.errorStats).length > 0) {
    console.log(`\n   Error Breakdown:`);
    Object.entries(stats.errorStats).forEach(([errorType, count]) => {
      console.log(`   ├─ ${errorType}: ${count} occurrences`);
    });
  }
}

// Run the example
main().catch(console.error);

Expected Output

🔍 Execution Analytics Plugin Example
=====================================

🤖 Agent created with ExecutionAnalyticsPlugin

📊 Method 1: Access through agent.getPlugin()
Executing task 1...
Executing task 2...

📊 Agent Plugin Access Analytics:
   ├─ Total Executions: 4
   ├─ Successful: 4
   ├─ Failed: 0
   ├─ Success Rate: 100.0%
   ├─ Average Duration: 1250ms
   └─ Total Duration: 5000ms

   Operation Breakdown:
   ├─ run: 2 calls (1800ms avg)
   ├─ provider-call: 2 calls (700ms avg)

📋 Raw execution data: 4 entries
   └─ Last execution: provider-call (698ms, success)

🔍 Plugin Status:
   ├─ Name: ExecutionAnalyticsPlugin
   ├─ Version: 1.0.0
   ├─ Enabled: true
   ├─ Total Recorded: 4
   ├─ Active Executions: 0
   └─ Memory Usage: 4 items

📊 Method 2: Direct plugin access
📊 Direct Plugin Access Analytics:
   ├─ Total Executions: 4
   ├─ Successful: 4
   ├─ Failed: 0
   ├─ Success Rate: 100.0%
   ├─ Average Duration: 1250ms
   └─ Total Duration: 5000ms

🧹 Clearing analytics data...
   └─ After clearing: 0 executions

Key Features

1. Automatic Execution Tracking

The plugin automatically tracks all agent operations without manual intervention:

const analyticsPlugin = new ExecutionAnalyticsPlugin({
  maxEntries: 100, // Maximum number of entries to store
  trackErrors: true, // Track error occurrences
  performanceThreshold: 2000, // Threshold for performance warnings
  enableWarnings: true, // Enable performance warnings
});

2. Comprehensive Metrics

The plugin tracks multiple types of operations:

• Agent runs: Complete user interactions
• Provider calls: AI provider API calls
• Tool executions: Function tool calls
• Performance metrics: Duration, success rates
• Error tracking: Failure types and frequencies

3. Multiple Access Methods

Access analytics through different approaches:

// Method 1: Through agent (recommended)
const pluginFromAgent = agent.getPlugin('ExecutionAnalyticsPlugin');
const stats = pluginFromAgent.getAggregatedStats();

// Method 2: Direct plugin access
const directStats = analyticsPlugin.getAggregatedStats();

4. Detailed Analytics Data

The plugin provides comprehensive statistics:

{
    totalExecutions: 4,
    successfulExecutions: 4,
    failedExecutions: 0,
    successRate: 1.0,
    averageDuration: 1250,
    totalDuration: 5000,
    operationStats: {
        'run': { count: 2, averageDuration: 1800 },
        'provider-call': { count: 2, averageDuration: 700 }
    },
    errorStats: {}
}

5. Raw Execution Data

Access detailed execution information:

const rawData = plugin.getData();
// Returns array of execution records with:
// - operation: type of operation
// - duration: execution time
// - success: success/failure status
// - timestamp: when it occurred
// - metadata: additional context

Plugin Configuration

Basic Configuration

const analyticsPlugin = new ExecutionAnalyticsPlugin({
  maxEntries: 100, // Limit memory usage
  trackErrors: true, // Include error tracking
  performanceThreshold: 2000, // Warn on slow operations
  enableWarnings: true, // Enable console warnings
});

Advanced Configuration

const analyticsPlugin = new ExecutionAnalyticsPlugin({
  maxEntries: 500,
  trackErrors: true,
  performanceThreshold: 1000,
  enableWarnings: true,
  // Optional: Custom operation filters
  operationFilters: ['run', 'provider-call'],
  // Optional: Custom metadata collection
  collectMetadata: true,
});

Use Cases

1. Performance Monitoring

Track agent response times and identify bottlenecks:

const stats = plugin.getAggregatedStats();
if (stats.averageDuration > 2000) {
  console.warn('Agent performance is below threshold');
}

2. Success Rate Analysis

Monitor agent reliability:

const successRate = stats.successRate * 100;
if (successRate < 95) {
  console.warn(`Success rate: ${successRate}% - investigate issues`);
}

3. Operation Breakdown

Analyze where time is spent:

Object.entries(stats.operationStats).forEach(([operation, opStats]) => {
  console.log(`${operation}: ${opStats.averageDuration}ms average`);
});

4. Error Analysis

Identify common failure patterns:

Object.entries(stats.errorStats).forEach(([errorType, count]) => {
  console.log(`${errorType}: ${count} occurrences`);
});

Best Practices

1. Memory Management: Set appropriate maxEntries to prevent memory leaks
2. Performance Thresholds: Configure thresholds based on your application requirements
3. Regular Analysis: Review analytics data regularly to identify trends
4. Error Tracking: Enable error tracking to identify and resolve issues
5. Data Cleanup: Clear analytics data when needed to reset metrics

The ExecutionAnalyticsPlugin provides invaluable insights for optimizing agent performance and ensuring reliable operation.