Skip to Content
🎉 Welcome to handit.ai Documentation!
Quickstart

Set Up Your Autonomous Engineer

The autonomous engineer that fixes your AI 24/7
Handit catches failures, writes fixes, tests them, and ships PRs, automatically. Like having an on-call engineer dedicated to your AI, except it works 24/7.

What you’re setting up: Your autonomous engineer that monitors your AI, detects issues, creates fixes, and ships them via pull requests—all automatically.

What Your Autonomous Engineer Will Do

Once set up, your autonomous engineer will:

  • Monitor every AI interaction for quality issues and failures
  • Detect patterns in problems before they become critical
  • Generate improved system prompts that fix the issues
  • Test the fixes against real data to ensure they work
  • Create pull requests in your repository with the improvements
  • Work 24/7 so you don’t have to wake up to AI failures

The Result: Your AI continuously improves itself through your normal GitHub workflow, with zero manual intervention required.

Prerequisites

Before setting up your autonomous engineer:

  • A Handit.ai Account  (sign up if needed)
  • Node.js installed (for the CLI)
  • Admin access to your GitHub repository
  • 10-15 minutes for the complete setup

Setting Up Your Autonomous Engineer

Your autonomous engineer setup happens in one seamless flow through the Handit CLI. Here’s what will happen:

Step 1: Install the Handit CLI

terminal
npm install -g @handit.ai/cli

Step 2: Start the Setup Process

Navigate to your AI project directory and run:

terminal
handit-cli setup

The CLI will guide you through connecting your autonomous engineer to your AI system:

🔧 Initial Connection

  • Connect your Handit.ai account
  • Generate integration code for your AI project
  • Set up monitoring and tracing automatically

📱 Test Your Connection

  • The CLI will ask you to run your app to verify everything works
  • Your autonomous engineer immediately starts monitoring your AI
  • You’ll see real-time data flowing in your dashboard

🧠 Connect Evaluation Models

  • The CLI will ask if you want to set up quality evaluation
  • Connect evaluation models (OpenAI, Together AI, etc.) so your autonomous engineer can assess quality
  • Configure which evaluators to use and how often to run them

🔗 Connect Your GitHub Repository

  • Finally, the CLI will ask you to connect your GitHub repository
  • Install the Handit GitHub app so your autonomous engineer can create pull requests
  • Configure repository permissions for automated fixes

Setup Complete! Your autonomous engineer is now monitoring your AI, evaluating quality, and ready to create pull requests with fixes whenever issues are detected.

How Your Autonomous Engineer Works

Once setup is complete, here’s what happens automatically:

🔍 Continuous Monitoring

  • Your autonomous engineer watches every AI interaction
  • Tracks quality scores, response times, and error patterns
  • Builds understanding of what good vs. bad performance looks like

🚨 Issue Detection

  • Detects when quality drops or new types of failures appear
  • Identifies root causes by analyzing patterns in your data
  • Prioritizes the most impactful issues to fix first

🛠️ Automatic Fix Generation

  • Creates improved system prompts that address the detected issues
  • Tests fixes against your real data to ensure they actually work
  • Validates improvements with statistical confidence

📝 Pull Request Creation

  • Opens a PR in your repository with the improved system prompt
  • Includes detailed performance data showing the improvement
  • Follows your existing code review and deployment process

Example: If your customer service AI starts giving incomplete responses, your autonomous engineer detects this pattern, generates a better system prompt that addresses completeness, tests it against real conversations, and creates a PR to replace the old prompt in your codebase.

Viewing Your Autonomous Engineer’s Work

Your autonomous engineer’s monitoring and fixes are visible in your Handit dashboard:

📊 Real-Time Monitoring AI Agent Tracing Dashboard

  • Tracing: See every AI interaction with complete execution details
  • Agent Performance: Track quality trends and improvement over time
  • Issue Detection: View detected problems and their severity
  • Fix History: See all improvements your autonomous engineer has made

🔧 Active Fixes

  • Monitor pull requests created by your autonomous engineer
  • Review performance improvements before merging
  • Track the impact of deployed fixes on your AI’s quality

What You’ve Accomplished

Congratulations! You now have your autonomous engineer working 24/7 to improve your AI:

✅ Your Autonomous Engineer is Active

  • Monitoring every AI interaction for quality and performance issues
  • Evaluating responses automatically using advanced AI models
  • Detecting problems before they become critical failures
  • Generating fixes for issues it discovers in your AI system
  • Creating pull requests with improvements directly in your codebase

✅ Continuous AI Improvement

  • Your AI will automatically get better over time through GitHub PRs
  • No more manual prompt engineering or quality monitoring required
  • Issues are caught and fixed 24/7, even while you sleep
  • All improvements go through your normal code review process

Your Autonomous Engineer is Now Working

Your setup is complete! Here’s what happens next:

🚀 Immediate Benefits

  • Start seeing AI interactions and quality scores in your dashboard right away
  • Your autonomous engineer is learning your AI’s patterns and baseline performance
  • Any existing issues will be detected and queued for fixing

📈 Ongoing Improvements

  • Over the coming days and weeks, you’ll receive pull requests with AI improvements
  • Each PR includes detailed data showing exactly how the fix improves performance
  • Your AI’s quality will steadily improve through this autonomous feedback loop

Next Steps

🎯 Monitor Your Progress

  • Check your Handit Dashboard  to see your autonomous engineer at work
  • Review any pull requests created by your autonomous engineer
  • Watch your AI quality improve over time

📚 Learn More

💬 Get Support

Your autonomous engineer is now working! It’s monitoring your AI, detecting issues, and will start creating pull requests with improvements. Welcome to autonomous AI development.

Troubleshooting

CLI Setup Issues?

  • Ensure Node.js is installed: node --version
  • Try reinstalling the CLI: npm uninstall -g @handit.ai/cli && npm install -g @handit.ai/cli
  • Check your Handit.ai account credentials

Tracing Not Working?

  • Run handit-cli setup again to regenerate configuration
  • Verify your generated code is being executed
  • Check that your API key was set correctly during setup

Evaluations Not Running?

  • Re-run handit-cli evaluators-setup to verify connections
  • Confirm model tokens are valid and have sufficient credits
  • Verify LLM nodes are receiving traffic

GitHub Integration Issues?

  • Ensure you have admin access to the repository
  • Try running handit-cli github again to reinstall the app
  • Check repository permissions in your GitHub settings

Need Help?

Last updated on
OverviewIntroduction