Monitoring

Your autonomous engineer needs to see your AI to fix it. Monitoring gives your engineer complete visibility into your AI agents so it can detect issues, analyze root causes, and generate proven fixes.

The Problem: AI Black Boxes

2am failures wake you up. Your AI starts giving bad responses, customers complain, and you’re debugging blind. Did the language model change its behavior? Is a tool returning bad data? Is there a logic error in your reasoning chain? Without visibility, you’re playing whack-a-mole with quality issues.

Manual debugging doesn’t scale. You spot-check responses, find problems, and wonder about the thousands of interactions you didn’t see. By the time you notice issues, they’ve already impacted users.

How Monitoring Works

Issue Detection: By analyzing your AI’s behavior, your autonomous engineer spots when quality drops, which interactions fail, and what causes problems.

Root Cause Analysis: Detailed data helps your autonomous engineer understand whether issues stem from prompt problems, tool failures, or logic errors.

Fix Validation: Before creating pull requests, your autonomous engineer tests improvements against historical data to ensure fixes actually work.

Continuous Learning: As fixes get deployed, new data shows their effectiveness, helping your autonomous engineer improve over time.

What Gets Monitored

When you set up monitoring, Handit automatically captures:

• Complete execution flow from user request to final response
• All LLM interactions with prompts, responses, and performance data
• Tool executions with parameters, results, and timing
• Error details with full context for debugging
• Performance metrics showing bottlenecks and optimization opportunities

Quick Setup

Time required: Under 5 minutes
Prerequisites: Handit.ai Account , Node.js

Step 1: Install the Handit CLI

npm install -g @handit.ai/cli

Step 2: Start Monitoring Setup

Navigate to your AI project directory and run:

handit-cli setup

The CLI will guide you through connecting monitoring to your AI system:

🔧 Initial Connection

• Connect your Handit.ai account
• Install the Handit SDK in your project
• Configure your API key for monitoring

📱 Test Your Connection

• The CLI will ask you to run your app to verify everything works
• Your autonomous engineer immediately starts monitoring your AI
• You’ll see real-time data flowing in your dashboard

Example of code the CLI generates for you:

The CLI adds tracing only to your main agent entry points - the functions that start agent execution:

Python:

# Auto-generated by handit-cli setup
from handit_ai import tracing, configure
import os
 
configure(HANDIT_API_KEY=os.getenv("HANDIT_API_KEY"))
 
# Tracing added to your main agent function (entry point)
@tracing(agent="customer-service-agent")
async def process_customer_request(user_message: str):
    # Your existing agent logic (unchanged)
    intent = await classify_intent(user_message)      # Not traced individually
    context = await search_knowledge(intent)          # Not traced individually  
    response = await generate_response(context)       # Not traced individually
    return response

JavaScript:

// Auto-generated by handit-cli setup
import { configure, startTracing, endTracing } from '@handit.ai/handit-ai';
 
configure({
  HANDIT_API_KEY: process.env.HANDIT_API_KEY
});
 
// Tracing added to your main agent function (entry point)
export const processCustomerRequest = async (userMessage) => {
  startTracing({ agent: "customer-service-agent" });
  try {
    // Your existing agent logic (unchanged)
    const intent = await classifyIntent(userMessage);     // Not traced individually
    const context = await searchKnowledge(intent);       // Not traced individually
    const response = await generateResponse(context);     // Not traced individually
    return response;
    } finally {
    endTracing();
  }
};

Key point: The CLI only adds tracing to your main agent functions, not to every individual function. Everything inside the traced function gets captured automatically.

Verify Your Setup

✅ Check your dashboard: Go to dashboard.handit.ai  - you should see:

• Tracing data flowing in real-time as your AI processes requests
• Agent Performance showing your AI’s baseline metrics

Next Steps

Complete Setup: For full autonomous engineer setup including evaluation and optimization, use our Complete Setup Guide.

Advanced Options:

• Manual Setup - Add decorators yourself
• Advanced Monitoring - Node-by-node monitoring