Feedback Engine

The feedback engine collects ratings on query results and uses them to improve accuracy over time. It is the self-improving loop that makes Epistom better with use — good queries get promoted to verified status, bad patterns get flagged for review.

How It Works

Submitting Feedback (`feedback_engine.py`)

The Feedback dataclass captures a rating for a specific query:

from epistom.semantic_core.governance.feedback_engine import Feedback, submit_feedback
 
feedback = Feedback(
    query_id="q-123",
    rating="correct",       # correct, wrong, partial
    comment="Accurate results",
    user_id="analyst-1"
)
submit_feedback(feedback, workspace_id=workspace_id)

Feedback is persisted to the query_feedback PostgreSQL table with the query ID, rating, optional comment, and timestamp.

Analyzing Feedback

The analyze_feedback() function reviews recent feedback history and generates ImprovementSuggestion objects:

from epistom.semantic_core.governance.feedback_engine import analyze_feedback
 
suggestions = analyze_feedback(days=30)
# Returns ImprovementSuggestion objects with:
#   - type: "promote_vq", "review_definition", "fix_join_path"
#   - details: specific recommendations
#   - confidence: strength of the suggestion

Feedback Classifier (`feedback_classifier.py`)

Classifies feedback patterns to identify systemic issues:

Repeated negative feedback on the same concept suggests a definition problem
Negative feedback on a specific join path suggests a schema mapping issue
Consistently positive feedback qualifies queries for verified query promotion

Feedback Analyzer (`feedback_analyzer.py`)

Aggregates feedback into actionable insights:

Accuracy rate by concept domain
Most common failure patterns
Trending positive/negative query categories

VQ Promotion on Feedback

The check_vq_promotion_on_feedback() function evaluates whether a query should be promoted to verified status based on its feedback history. A query needs consistent positive ratings above a threshold with no recent negative feedback.

Architecture

The feedback loop:

Agent calls ask_question and receives results with a query_id
Agent (or human) calls report_feedback with the query_id and a rating
Feedback is stored in PostgreSQL
Periodic analysis runs analyze_feedback() to generate improvement suggestions
Suggestions are surfaced to data stewards for review
Approved improvements update the knowledge model or promote verified queries

API / Usage

REST API endpoints:

POST /api/v1/feedback          — Submit feedback
GET  /api/v1/feedback/history  — Get feedback history
GET  /api/v1/feedback/analysis — Get improvement suggestions
GET  /api/v1/feedback/accuracy — Get accuracy metrics

MCP tool:

report_feedback(query_id, rating, comment?)

Accuracy Metrics

The get_accuracy_metrics() function returns aggregate statistics:

Total queries in period
Correct/wrong/partial breakdown
Accuracy rate (correct / total)
Trend over time (improving or degrading)

Technical Details

Feedback is workspace-isolated in multi-tenant deployments
The feedback database is initialized lazily on first submission (_init_feedback_db())
Feedback history queries support configurable time windows (default 30 days)
Humans always approve changes — the system generates suggestions but never auto-modifies the knowledge model

On this page