feat: v1 — boilerplate WxO + web
Boilerplate completo para construir soluciones agénticas sobre IBM watsonx Orchestrate (ADK 2.x) con una capa web encima. Destilado de cotemar-poc-n1 y dun-casos-prueba. Incluye: - 11 docs (best practices, architecture patterns, ADK cheatsheet, observability, tool authoring, deployment, RUNBOOK, DEPLOY_TO_NEW_WOX, known-issues con 16 errores reales y fix, eval strategy, INDEX) - 13 templates WxO (orchestrator/specialist/single-meta agents, 3 connections, KB + runbook, observable_tool decorator, coercion helpers, Python tools, OpenAPI spec, backend filter endpoint, webhook validator HMAC, MCP connection) - 6 scripts (deploy idempotente con fallback ADK, undeploy, reset, check-adk-version, new-specialist scaffold, eval-agents runner) - 4 eval pieces (linter de best practices, runner, smoke-test, direct backend probe) + scenario templates - 8 subagentes Claude (.claude/agents/) — wxo-architect, wxo-agent-author, wxo-tool-author, runbook-author, mock-builder, backend-tool-builder, eval-author, web-layer-builder - Skill bundle fit-wxo-bootstrap/ (SKILL.md + 3 templates) listo para copiar a ~/.claude/skills/ - Web layer default FastAPI+HTMX con vista timeline observable + endpoint receptor de trazas - Docker compose Coolify-ready (healthcheck wget -qO-, sin labels Traefik, network split internal:true) - CI Gitea workflow con lint + smoke Best practices enforced por evals/lint_wxo_yaml.py: - A1: máx 10 tools por agente - A3: orchestrator sin tools de remediación - A6: agente sin propósito (react sin tools ni KB) - T1: @observable_tool obligatorio, no @tool directo - T3: _compat shim inline en tools.py - D1: docker-compose sin wget --spider - I-005: OpenAPI con description per-operation - I-009: sin labels Traefik manuales Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
72
evals/eval-agents.sh
Executable file
72
evals/eval-agents.sh
Executable file
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env bash
|
||||
# Runner completo de evals. Corre las 4 capas en orden.
|
||||
# Exit 0 si todo pasa, 1 si alguna falla.
|
||||
|
||||
set -uo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
|
||||
FAILED=0
|
||||
|
||||
# ─── Capa 0: Lint ───────────────────────────────────────────────────────────
|
||||
echo "[lint] best practices"
|
||||
if python3 evals/lint_wxo_yaml.py; then
|
||||
echo " ✓ lint PASS"
|
||||
else
|
||||
echo " ✗ lint FAIL"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# ─── Capa 1: Native (ADK) ───────────────────────────────────────────────────
|
||||
echo "[native] orchestrate agents test"
|
||||
NATIVE_PASS=0; NATIVE_FAIL=0
|
||||
for f in evals/scenarios/*.input.json; do
|
||||
[ -f "$f" ] || continue
|
||||
# Derivar nombre del agente (convención: scenario_X_agentName.input.json o leer del JSON)
|
||||
agent=$(python3 -c "import json,sys; d=json.load(open('$f')); print(d.get('_agent',''))" 2>/dev/null)
|
||||
if [ -z "$agent" ]; then
|
||||
echo " ⚠ $f sin '_agent' field, skip"
|
||||
continue
|
||||
fi
|
||||
if orchestrate agents test "$agent" --input-file "$f" > /tmp/eval_native_$$.log 2>&1; then
|
||||
NATIVE_PASS=$((NATIVE_PASS + 1))
|
||||
else
|
||||
NATIVE_FAIL=$((NATIVE_FAIL + 1))
|
||||
echo " ✗ $f failed (see /tmp/eval_native_$$.log)"
|
||||
fi
|
||||
done
|
||||
echo " native: $NATIVE_PASS pass / $NATIVE_FAIL fail"
|
||||
[ "$NATIVE_FAIL" -gt 0 ] && FAILED=$((FAILED + 1))
|
||||
echo ""
|
||||
|
||||
# ─── Capa 2 + 3: Behavior + Final state (runner.py) ─────────────────────────
|
||||
echo "[behavior+final] custom runner"
|
||||
if python3 evals/runner.py; then
|
||||
echo " ✓ runner PASS"
|
||||
else
|
||||
echo " ✗ runner FAIL"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# ─── Capa 4: Smoke test ─────────────────────────────────────────────────────
|
||||
echo "[smoke] end-to-end"
|
||||
if ./evals/smoke-test.sh > /tmp/eval_smoke_$$.log 2>&1; then
|
||||
echo " ✓ smoke PASS"
|
||||
else
|
||||
echo " ✗ smoke FAIL (see /tmp/eval_smoke_$$.log)"
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# ─── Resumen ────────────────────────────────────────────────────────────────
|
||||
echo "════════════════════════════════════════"
|
||||
if [ "$FAILED" -eq 0 ]; then
|
||||
echo " ✓ ALL EVALS PASS"
|
||||
exit 0
|
||||
else
|
||||
echo " ✗ $FAILED layer(s) FAILED"
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user