Error Reference Guide

This document catalogs errors you may encounter when using the Neon agent evaluation platform, organized by component. Each entry includes the error message, what causes it, and how to resolve it.

CLI Errors

The Neon CLI (agent-eval) uses exit code 0 for success and 1 for any failure. User cancellations raise typer.Abort().

Exit Codes

File & Validation Errors

Agent & Execution Errors

Authentication Errors

Comparison Errors

Warnings (Non-Fatal)

These are displayed in yellow and do not cause the CLI to exit with an error:

API Errors

The Neon API returns JSON error responses with an error field and an HTTP status code.

Error Response Format

{
  "error": "Human-readable error description",
  "details": "Additional context (on 500/503 errors)",
  "hint": "Suggestion for fixing the issue (on 401/403 errors)",
  "status": 400
}

400 Bad Request

Returned when the request is missing required fields or contains invalid data.

401 Unauthorized

Returned when authentication is missing or invalid.

Supported authentication methods:

• JWT Bearer Token: Authorization: Bearer <token>
• API Key: X-API-Key: ae_<env>_<key>

403 Forbidden

Returned when the authenticated user lacks permission for the requested operation.

404 Not Found

Returned when the requested resource does not exist. For security, 404 is also returned when the user lacks access to the resource (to prevent enumeration).

422 Unprocessable Entity

Returned when request body validation fails (Zod schema validation).

{
  "error": "Validation error",
  "code": "VALIDATION_ERROR",
  "message": "Field validation failed",
  "details": [
    { "field": "field_name", "message": "error message" }
  ]
}

Common causes: missing required fields, invalid data types, invalid enum values.

500 Internal Server Error

Returned when an unexpected error occurs. The details field contains the error message.

Alert Rule API Errors

503 Service Unavailable

Returned when an external dependency (database, workflow engine) is not reachable.

Connection error detection: The API checks for ECONNREFUSED, ETIMEDOUT, UNAVAILABLE, and connect errors in exception messages to distinguish infrastructure issues from application errors.

SDK Errors

TypeScript SDK (@neon/sdk)

Custom Error Classes

CloudSyncError Thrown when cloud sync operations fail (network issues, authentication problems, timeouts).

class CloudSyncError extends Error {
  statusCode?: number;
  cause?: unknown;
}

CorrelationAnalysisError Thrown when ClickHouse correlation queries fail.

class CorrelationAnalysisError extends Error {
  code: CorrelationErrorCode;
  cause?: unknown;
}

Client Errors

Validation Errors

Debug Client Errors

Export Errors

Python SDK (neon-sdk)

Import Errors (Optional Dependencies)

Client Errors

Validation Errors

Infrastructure Errors

Temporal Workflow Errors

Worker Connection

The worker uses exponential backoff with configurable retry:

• Max reconnect attempts: 10 (override with MAX_RECONNECT_ATTEMPTS env var)
• Reconnect delay: 5000ms (override with RECONNECT_DELAY_MS env var)

Activity Timeouts

Activities have configured timeouts and retry policies. If an activity exceeds its timeout, Temporal will retry according to the retry policy.

Workflow-Level Errors

Workflow Control Signals

LLM Provider Errors

Thrown when LLM provider SDKs are missing or misconfigured in the Temporal worker.

Health Check API

The GET /api/health endpoint returns the overall system status:

ClickHouse Errors

PostgreSQL Errors

Troubleshooting

Services Won’t Start

1. Check Docker is running: docker ps
2. Start all infrastructure: docker compose up -d
3. Start with Temporal: docker compose --profile temporal up -d
4. Check container logs: docker compose logs <service-name>
5. Verify ports are free: ClickHouse (8123), PostgreSQL (5432), Temporal (7233)

Eval Runs Stuck in “Running” State

1. Check the Temporal UI (default: http://localhost:8233) for workflow status
2. Verify the Temporal worker is running: bun run workers
3. Check worker logs for activity failures
4. If needed, cancel the run: POST /api/runs/{id}/control with {"action": "cancel"}

Scores Are All Zero

1. Verify the scorer is configured correctly in the suite definition
2. Check if the agent is producing output (non-empty responses)
3. Review trace spans to confirm the agent executed
4. For LLM judges, ensure the ANTHROPIC_API_KEY environment variable is set
5. Check scorer error reasons in the run results

API Returns 503 Errors

1. Identify which service is unavailable from the error message
2. Check Docker container status: docker compose ps
3. Restart the failing service: docker compose restart <service>
4. Review container logs for startup errors

CLI Authentication Issues

1. Run agent-eval auth status to check current credentials
2. Re-authenticate: agent-eval auth login
3. Verify the API URL is correct in your configuration
4. Check that your API key has not expired

Suite Validation Fails

1. Validate YAML syntax with a YAML linter
2. Ensure all required fields are present (name, cases, scorers)
3. Check that scorer names match available scorers
4. Run agent-eval suite validate <file> for detailed validation errors