Set Up Your Autonomous Engineer

The autonomous engineer that fixes your AI 24/7
Handit catches failures, writes fixes, tests them, and ships PRs, automatically. Like having an on-call engineer dedicated to your AI, except it works 24/7.

Your autonomous engineer monitors your AI 24/7, detects issues, creates fixes, and ships them as pull requests—automatically.

Time required: 10-15 minutes
Prerequisites: Handit.ai Account , Node.js, and admin access to your GitHub repository

Set up your autonomous engineer

Your autonomous engineer setup happens in one seamless flow through the Handit CLI. Here’s what will happen:

Step 1: Install the Handit CLI

npm install -g @handit.ai/cli

Step 2: Start the Setup Process

Navigate to your AI project directory and run:

handit-cli setup

The CLI will guide you through connecting your autonomous engineer to your AI system:

🔧 Initial Connection

• Connect your Handit.ai account
• Install the Handit SDK in your project
• Configure your API key for monitoring

📱 Test Your Connection

• The CLI will ask you to run your app to verify everything works
• Your autonomous engineer immediately starts monitoring your AI
• You’ll see real-time data flowing in your dashboard

Example of code the CLI generates for you:

The CLI adds tracing only to your main agent entry points - the functions that start agent execution:

Python:

# Auto-generated by handit-cli setup
from handit_ai import tracing, configure
 
configure(HANDIT_API_KEY=os.getenv("HANDIT_API_KEY"))
 
# Tracing added to your main agent function (entry point)
@tracing(agent="customer-service-agent")
async def process_customer_request(user_message: str):
    # Your existing agent logic (unchanged)
    intent = await classify_intent(user_message)      # Not traced individually
    context = await search_knowledge(intent)          # Not traced individually  
    response = await generate_response(context)       # Not traced individually
    return response

JavaScript:

// Auto-generated by handit-cli setup
import { configure, startTracing, endTracing } from '@handit.ai/handit-ai';
 
configure({
  HANDIT_API_KEY: process.env.HANDIT_API_KEY
});
 
// Tracing added to your main agent function (entry point)
export const processCustomerRequest = async (userMessage) => {
  startTracing({ agent: "customer-service-agent" });
  try {
    // Your existing agent logic (unchanged)
    const intent = await classifyIntent(userMessage);     // Not traced individually
    const context = await searchKnowledge(intent);       // Not traced individually
    const response = await generateResponse(context);     // Not traced individually
    return response;
    } finally {
    endTracing();
  }
};

Key point: The CLI only adds tracing to your main agent functions, not to every individual function. Everything inside the traced function gets captured automatically.

🧠 Connect Evaluation Models

• The CLI will ask if you want to set up quality evaluation
• Connect evaluation models (OpenAI, Together AI, etc.) so your autonomous engineer can assess quality
• Configure which evaluators to use and how often to run them

🔗 Connect Your GitHub Repository

• Finally, the CLI will ask you to connect your GitHub repository
• Install the Handit GitHub app so your autonomous engineer can create pull requests
• Configure repository permissions for automated fixes

Verify Your Setup

✅ Check your dashboard: Go to dashboard.handit.ai  - you should see:

• Tracing data flowing in real-time as your AI processes requests
• Quality scores appearing for evaluated interactions
• Agent Performance showing your AI’s baseline metrics

✅ Confirm GitHub integration: Check your repository - you should see:

• Handit app installed in your repository settings
• Branch protection respecting your existing rules
• Ready for PRs - your autonomous engineer can now create pull requests

What happens after setup

Once your autonomous engineer is active, it works automatically:

🔍 Monitors your AI interactions and detects quality issues 🛠️ Generates fixes and tests them against real data
📝 Creates pull requests with proven improvements

Example: If your AI starts giving incomplete responses, your autonomous engineer detects this, generates a better prompt, tests it, and creates a PR with the fix.

What happens next

✅ Your autonomous engineer is now active

Check your dashboard  to see real-time monitoring data and quality scores.

Within the first week: If issues are detected, you’ll receive pull requests with fixes Ongoing: Your AI improves automatically while you focus on building features

Next steps

Monitor progress: Dashboard  | Learn more: Tracing, Evaluation, Autonomous Fixes | Get help: Discord , Support

Troubleshooting

CLI Setup Issues?

• Ensure Node.js is installed: node --version
• Try reinstalling the CLI: npm uninstall -g @handit.ai/cli && npm install -g @handit.ai/cli
• Check your Handit.ai account credentials

Tracing Not Working?

• Run handit-cli setup again to regenerate configuration
• Verify your generated code is being executed
• Check that your API key was set correctly during setup

Evaluations Not Running?

• Re-run handit-cli evaluators-setup to verify connections
• Confirm model tokens are valid and have sufficient credits
• Verify LLM nodes are receiving traffic

GitHub Integration Issues?

• Ensure you have admin access to the repository
• Try running handit-cli github again to reinstall the app
• Check repository permissions in your GitHub settings

Need Help?

• Visit our Support page
• Join our Discord community 
• Check individual quickstart guides for detailed troubleshooting