Feedback Engine

The feedback engine collects ratings on query results and uses them to improve accuracy over time. It is the self-improving loop that makes Epistom better with use — good queries get promoted to verified status, bad patterns get flagged for review.

How It WorksCopy Anchor Link

Submitting Feedback (feedback_engine.py)Copy Anchor Link

The Feedback dataclass captures a rating for a specific query:

from pramiti.contexts.epistom.feedback.infrastructure.feedback_engine import Feedback, submit_feedback

feedback = Feedback(
    query_id="q-123",
    feedback_type="CORRECT",
    rating=5,               # int 1-5 (1 = negative, 5 = positive)
    correction=None,
    actor="analyst-1",
)
submit_feedback(feedback, workspace_id=workspace_id)

The Feedback dataclass uses rating: int (1-5) plus correction text. String vocabularies such as correct/wrong/partial belong to the stdio MCP tool, which maps them onto that 1-5 scale.

Analyzing FeedbackCopy Anchor Link

The analyze_feedback() function reviews recent feedback history and generates ImprovementSuggestion objects:

from pramiti.contexts.epistom.feedback.infrastructure.feedback_engine import analyze_feedback

suggestions = analyze_feedback(days=30)
# Returns ImprovementSuggestion objects with:
#   - type: "promote_vq", "review_definition", "fix_join_path"
#   - details: specific recommendations
#   - confidence: strength of the suggestion

Feedback Classifier (feedback_classifier.py)Copy Anchor Link

Classifies feedback patterns to identify systemic issues:

• Repeated negative feedback on the same concept suggests a definition problem
• Negative feedback on a specific join path suggests a schema mapping issue
• Consistently positive feedback qualifies queries for verified query promotion

Feedback Analyzer (feedback_analyzer.py)Copy Anchor Link

Aggregates feedback into actionable insights:

• Accuracy rate by concept domain
• Most common failure patterns
• Trending positive/negative query categories

VQ Promotion on FeedbackCopy Anchor Link

The check_vq_promotion_on_feedback() function evaluates whether a query should be promoted to verified status based on its feedback history. A query needs consistent positive ratings above a threshold with no recent negative feedback.

ArchitectureCopy Anchor Link

The feedback loop:

1. Agent calls ask_question and receives results with a query_id
2. Agent (or human) calls report_feedback with the query_id and a rating
3. Feedback is stored in PostgreSQL
4. Periodic analysis runs analyze_feedback() to generate improvement suggestions
5. Suggestions are surfaced to data stewards for review
6. Approved improvements update the knowledge model or promote verified queries

API / UsageCopy Anchor Link

REST API endpoints:

GET  /api/v1/feedback/pending-verifications
POST /api/v1/feedback/pending-verifications/{item_id}/approve
POST /api/v1/feedback/pending-verifications/{item_id}/reject
GET  /api/v1/feedback/flagged-queries
POST /api/v1/feedback/flagged-queries/{item_id}/resolve
POST /api/v1/feedback/flagged-queries/{item_id}/dismiss

MCP tool:

report_feedback(query_id, rating, comment?)

Accuracy MetricsCopy Anchor Link

The get_accuracy_metrics() function returns aggregate statistics:

• Total queries in period
• Correct/wrong/partial breakdown
• Accuracy rate (correct / total)
• Trend over time (improving or degrading)

Technical DetailsCopy Anchor Link

• Feedback is workspace-isolated in multi-tenant deployments
• The feedback database is initialized lazily on first submission (_init_feedback_db())
• Feedback history queries support configurable time windows (default 30 days)
• Humans always approve changes — the system generates suggestions but never auto-modifies the knowledge model