Laravel React PostgreSQL / pgvector OpenAI / Anthropic

RAGistrat

A RAG-powered assessment tool for legal education that provides evidence-based evaluations of student work using the IRAC method. By grounding analysis in authoritative legal sources, it bridges AI reasoning with the demands of legal scholarship.

// 01

Overview

RAGistrat employs Retrieval-Augmented Generation to analyse law students' written submissions against a curated knowledge base of legal documents. The system evaluates work using the IRAC method (Issue, Rule, Application, Conclusion) and a configurable set of 17 assessment rules, producing structured feedback with norm citations and rule violation flags for instructor review.

RAGistrat is a collaborative project between the School of Governance, Law and Society and the School of Digital Technologies at Tallinn University.

Workflow

  1. Knowledge base construction — legal documents (PDF/TXT) are chunked, embedded, and stored as vectors for semantic search
  2. Case definition — instructors create legal scenarios and exercises
  3. Student enrollment — instructors generate invite codes per case; students self-enroll and submit their analyses
  4. AI analysis — submissions are evaluated against the knowledge base and assessment rules, producing IRAC-structured feedback with norm citations and violation flags
  5. Flag review — critical issues are flagged for instructor review with accept/dismiss workflow
  6. Instructor grading — review AI analysis, adjust scores (0–18 German legal grading scale), provide feedback
  7. Student feedback — once the instructor publishes, students see their grade, IRAC analysis, and improvement suggestions
  8. Notifications — in-app notifications fire when analyses complete and when grades are published, with an unread badge in the nav bar
  9. PDF export — download structured analysis reports

// 02

Technical Architecture

Backend
Laravel, PHP
Frontend
React via Inertia.js (JSX), Tailwind CSS
Database
PostgreSQL with pgvector extension (cosine similarity search)
Queue
Redis, monitored via Laravel Horizon
AI / Embeddings
OpenAI embeddings (text-embedding-3-small), switchable LLM (OpenAI GPT / Anthropic Claude)
Deployment
Docker Compose (multi-stage build) targeting VPS
CI
GitHub Actions (tests + lint)
License
All rights reserved

Core Services

Service Responsibility
DocumentChunker Splits legal documents into overlapping chunks for embedding
VectorSearch Cosine similarity search over pgvector embeddings
LlmManager Switchable LLM provider abstraction (OpenAI / Anthropic)
PromptBuilder Assembles analysis prompts from submission, retrieved context, and assessment rules
RuleEvaluator Evaluates submissions against 17 configurable assessment rules
EvaluationRunner Fixture-based quality measurement for analysis output
Audit observers Analysis, AnalysisGrade, and Submission observers write an audit trail of state changes

// 03

RAG Pipeline

The retrieval-augmented generation pipeline operates in two phases:

Ingestion — legal source documents are uploaded, text is extracted (PDF/TXT), split into overlapping chunks by the DocumentChunker, embedded via OpenAI's text-embedding-3-small model, and stored as vectors in PostgreSQL with the pgvector extension.

Analysis — when a student submission is processed, the system performs semantic search over the vector store to retrieve the most relevant legal context, assembles a structured prompt combining the submission text, retrieved passages, and the 17 assessment rules, then sends it to the configured LLM (OpenAI GPT or Anthropic Claude) for IRAC-structured evaluation.

Analysis jobs run asynchronously via Redis queues monitored through Laravel Horizon. Instructors receive notifications when analyses complete.

// 04

IRAC Assessment Framework

The system evaluates legal writing using the IRAC method — a standard framework in legal education:

  • Issue — has the student correctly identified the legal issue(s)?
  • Rule — are the relevant legal rules and norms accurately stated?
  • Application — is the law properly applied to the facts of the case?
  • Conclusion — does the analysis reach a logically sound conclusion?

Each dimension is evaluated against configurable assessment rules. The AI produces structured feedback with specific norm citations from the knowledge base, flags rule violations for instructor review, and suggests scores on the 0–18 German legal grading scale.

Assessment Rules

The system ships with 17 assessment rules that define what constitutes quality legal analysis. Rules are configurable by instructors and cover aspects such as norm citation accuracy, logical argumentation structure, completeness of issue identification, and proper legal methodology.

// 05

Quality Assurance

RAGistrat includes a built-in evaluation harness for measuring analysis quality:

  • Fixture-based testing — predefined submissions with expected outcomes allow systematic quality measurement
  • Mock providers — all 305 tests (892 assertions) run without API keys using mock embedding and LLM providers
  • Evaluation CLI — evaluate:run command measures analysis accuracy against fixtures, runnable with mock or real providers
  • Policy-based authorization — 6 authorization policies with 41 dedicated tests ensure proper access control across all resources
  • Defence in depth — security headers middleware (X-Frame-Options, HSTS, Referrer-Policy, Permissions-Policy) and rate limiting on analysis endpoints (10/min) and source uploads (20/min)

// 06

Instructor & Student Interfaces

Instructor dashboard — overview statistics, recent activity, case management, source/knowledge base administration, invite-code generation for student enrollment, submission review with IRAC breakdown and violation flags, grading with score adjustment, flag accept/dismiss workflow, PDF report export, and notification center.

Student interface — a dedicated student dashboard listing enrolled cases and feedback status, invite-code enrollment flow, submission upload, and a filtered analysis view that surfaces grade and IRAC feedback once the instructor publishes.

Access control — role-based (admin, instructor, student) with policy-based authorization on all resources. Horizon dashboard restricted to admin/instructor roles.