AlignR — RLHF Alignment Pipeline for Regulated Enterprise AI

§1 Problem Statement

Regulated enterprises — banks, insurers, healthcare systems, legal services firms — deploying AI under the EU AI Act face obligations that general-purpose models and proprietary APIs cannot satisfy. The gap is not primarily about model quality. It is about regulatory defensibility: the ability to demonstrate, in writing, to a regulator or internal audit function, that the system behaves as intended, and that this was achieved deliberately.

1.1 The compliance gap

The EU AI Act classifies AI systems used in credit scoring, insurance risk assessment, healthcare decision support, and employment management as high-risk. For high-risk systems, operators must produce documented evidence satisfying five obligations before deployment.

Figure 1

EU AI Act high-risk obligations vs. what a proprietary API can demonstrate

A proprietary API's SOC 2 or HIPAA certification covers infrastructure security — not model behaviour. These are distinct compliance surfaces.

The core tension

The more capable the proprietary model, the more likely an enterprise will depend on it — and the harder it becomes to satisfy the AI Act's requirement that they can explain, audit, and freeze the system that made decisions affecting individuals. Weight ownership is not an ideological position. It is a prerequisite for conformity assessment.

1.2 Why prompt engineering and RAG are insufficient

The immediate response from most enterprise AI teams is to manage model behaviour through the prompt layer. This is a reasonable starting point but fails the compliance test for three structural reasons.

Intervention	What it controls	What it cannot guarantee	AI Act posture
System prompt engineering	Behaviour in well-tested cases	Behaviour under adversarial input, context drift, or long sessions. Prompt can be overridden or forgotten.	Fragile
Retrieval Augmented Generation (RAG)	Factual grounding — what the model knows	Tone, refusal calibration, or safety constraints. RAG supplies context; it does not change model judgment.	Incomplete
Output filtering / guardrails	Post-hoc blocking of specific patterns	Subtle non-compliance: wrong framing, legally loaded language, miscalibrated hedging that passes the filter.	Partial
RLHF / DPO fine-tuning (AlignR)	Weight-level behavioural priors — what the model is	Operates at the source. Robust to prompt manipulation. Encodes institutional judgment durably.	Defensible

Design principle

AlignR's premise is that compliance behaviour must be encoded in weights, not enforced at the prompt layer. The system prompt is an operational convenience. The model's trained priors are the compliance mechanism.

§2 Pipeline Overview

AlignR is a seven-stage pipeline. It begins with base model selection and ends with a versioned, deployable model accompanied by a compliance artefact package. Each stage produces outputs that are both operationally useful and documentable for conformity assessment purposes.

Figure 2

AlignR end-to-end pipeline — seven stages from base model to compliant deployment

Stages 02–06 all produce compliance artefacts. The pipeline run itself is the conformity assessment evidence package.

01

Base model selection

Open-weight · architecture compatible with TRL

02

Preference data collection

Domain expert annotation · pairwise ranking

03

Reward model training

Learns to score outputs as experts would

04

RLHF / DPO loop

Fine-tunes base model against reward signal

05

Red-teaming

Adversarial eval · edge case coverage

06

Compliance sign-off

CCO / DPO review · artefact bundle

07

Deployment

Private VPC · frozen weights · versioned

§3 Pipeline Components

3.1 Preference data collection

The preference data layer is the most consequential component of AlignR — and the most operationally intensive. It is where institutional knowledge is converted into a training signal. Annotators are not ML engineers. They are compliance officers, underwriters, legal counsel, and claims adjusters who review pairs of model responses and indicate which is more appropriate by their institution's standards.

Figure 3

Preference data collection — annotator workflow and artefact outputs

Every annotation event is logged with annotator identity, timestamp, and optional free-text rationale. This log is a primary compliance artefact.

A complete preference dataset for a single regulated domain (e.g. insurance member services) requires approximately 12,000–20,000 preference pairs collected over 4–8 weeks, covering: tone and phrasing compliance, refusal calibration, escalation trigger accuracy, and legally loaded language avoidance. Pairs are stratified by query type to ensure coverage across the compliance surface, not just high-frequency queries.

3.2 Reward model

The reward model is trained on the preference dataset to predict how a domain expert would score any given response. It is a scalar-output classifier — a fine-tuned version of the base model with a regression head that assigns a real-valued reward score to a (prompt, response) pair. This score becomes the training signal for the RLHF loop.

Figure 4

Reward model — structure and relationship to preference data

The reward model's weights are stored as a versioned artefact. They represent the institutionalised judgment of domain experts — and can be inspected, challenged, or retrained if regulatory standards change.

3.3 RLHF and DPO fine-tuning

AlignR supports two fine-tuning strategies. PPO-based RLHF is the classical approach: the language model is treated as a policy, the reward model scores its outputs, and Proximal Policy Optimisation updates the weights to maximise expected reward while a KL-divergence penalty prevents the model from drifting too far from its base distribution. Direct Preference Optimisation (DPO) skips the reward model inference loop entirely, optimising directly on the preference pairs — simpler to implement and more stable to train, at some cost to reward signal flexibility.

Strategy A

PPO-based RLHF

Classical reinforcement learning loop. The base model acts as a policy π_θ. The reward model r_φ scores sampled outputs. PPO optimises π_θ to maximise E[r_φ] with a KL penalty against the reference policy to prevent degeneration.

max E[r_φ(x,y)] − β·KL[π_θ || π_ref]

Best for: full control over the training signal, iterative reward shaping as domain standards evolve. Higher complexity. Requires the reward model to be available during training.

Strategy B

Direct Preference Optimisation

Reframes RLHF as a classification problem on preference pairs, eliminating the separate reward model inference step. Derives an optimal policy directly from the preference dataset using a closed-form transformation of the reward objective.

L_DPO = −log σ(β log π_θ(y_w|x)/π_ref(y_w|x) − β log π_θ(y_l|x)/π_ref(y_l|x))

Best for: faster iteration, more stable training, lower infrastructure overhead. Recommended for first AlignR deployment. The reward model still exists as a separate evaluation artefact.

Selection guide

When to use which

Start with DPO. It requires less infrastructure, is numerically more stable, and its outputs are easier to interpret during internal audit. Move to PPO-based RLHF when the domain requires iterative reward shaping — for example, when compliance standards are updated mid-cycle and the reward model must be retrained and redeployed without restarting from preference data collection.

Audit note: Both strategies produce a reproducible training run log in MLflow. The DPO loss curve and PPO reward curve are both compliance artefacts.

§4 Pipeline Simulator

The simulator below demonstrates the AlignR pipeline for four representative regulated-industry scenarios. Each run shows per-stage progress, timing estimates, and a sample of the configuration and output logs that would be generated. These logs are analogous to the compliance artefacts the pipeline produces in production.

Select a scenario and run the simulation. The output pane shows the YAML configuration, training parameters, and benchmark results that would be recorded in the MLflow artefact store.

alignr · pipeline simulator

idle

Scenario

Insurance DPO baseline — Member services assistant for a health insurer. DPO fine-tuning on 14,000 preference pairs from claims adjusters and compliance officers. Targets: coverage language, escalation triggers, tone on claim denials.

load base model

Llama-3-8B-Instruct

—

load preference data

annotation store → dataset

—

reward model training

regression head · 3 epochs

—

DPO fine-tuning

β=0.1 · LoRA r=16

—

red-team evaluation

adversarial query suite

—

artefact bundle

MLflow · compliance package

—

// select a scenario and run simulation

01

Base model loaded from internal model registry (private VPC). Preference dataset fetched from annotation store — JSON triples of (prompt, chosen, rejected) with annotator metadata attached.

annotation logs included · GDPR-compliant · no data leaves VPC

02

Reward model trained on preference pairs. Weights saved as versioned MLflow artefact. Validation accuracy on held-out pairs reported. This model is an inspectable compliance artefact — it encodes annotator judgment in a queryable form.

reward model weights · validation metrics · annotator inter-rater agreement

03

DPO or PPO fine-tuning run against the reward signal. Full training config (learning rate, β, LoRA rank, batch size, seed) logged to MLflow. Loss curve exported. Training is reproducible given the same seed and dataset version.

training config YAML · loss curve · GPU hours · reproducibility seed

04

Red-team evaluation suite run: 400–800 adversarial queries targeting known failure modes. Results scored by reward model and by a separate compliance reviewer. Pass/fail threshold set by CCO. Signed sign-off document generated on pass.

red-team report · compliance scorecard · CCO sign-off record

§5 Compliance Artefact Layer

The compliance artefact layer is what distinguishes AlignR from a general-purpose fine-tuning framework. Every stage of the pipeline produces structured outputs that map directly to EU AI Act conformity assessment requirements. These artefacts are not generated retrospectively — they are produced as primary outputs of the pipeline, stored in a versioned artefact registry, and referenced in the conformity assessment documentation submitted to regulators or internal audit.

Audit principle

Each model version deployed by AlignR is accompanied by an immutable artefact bundle. The bundle contains everything needed to answer the regulator's core question: "How do you know this system behaves as required, and how did you establish that?"

Figure 5

Compliance artefact map — pipeline stage to EU AI Act obligation

5.1 Artefact registry structure

ART-001

Annotation log

JSONL file. One record per annotation event: annotator ID (pseudonymised), timestamp, prompt hash, chosen response ID, rejected response ID, free-text rationale, inter-rater agreement score. Immutable after annotation session closes.

Data governance

ART-002

Reward model weights + card

Versioned model checkpoint (safetensors). Accompanied by a model card documenting: training dataset version, validation accuracy on held-out preference pairs, known failure modes, intended use scope. Signed by ML lead.

Transparency

ART-003

Fine-tuning run record

MLflow run export: complete hyperparameter config (YAML), training loss curve (PNG + CSV), GPU compute hours, reproducibility seed, dataset version hash, base model version. Sufficient to reproduce the exact trained model.

Reproducibility

ART-004

Aligned model weights + card

The production model checkpoint. Model card includes: alignment objectives, training data provenance, benchmark results, known limitations, intended deployment scope, and a deployment restriction statement signed by CCO.

Conformity assessment

ART-005

Red-team report

Structured evaluation report: adversarial query suite (400–800 queries), per-category pass/fail results, failure analysis for any failed queries, remediation plan if threshold not met. Signed by compliance reviewer.

Robustness evidence

ART-006

CCO / DPO sign-off record

Signed PDF containing: deployment authorisation, model version reference, scope of permitted use, known limitations acknowledgement, review date for next conformity reassessment. This document is the release gate.

Conformity assessment

§6 Deployment Architecture

AlignR's deployment architecture is designed around a single constraint: no member data, preference data, or model weights leave the enterprise's private infrastructure. This is not a performance optimisation — it is the architectural choice that makes the EU AI Act compliance posture coherent. If data or weights are processed by a third-party cloud service, the data governance chain and the weight ownership claim both weaken.

Figure 6

AlignR deployment architecture — private VPC · no external model API dependency

The monitoring component feeds a continuous retraining signal — if output distributions drift beyond a set threshold, it triggers a new annotation cycle rather than a prompt-layer patch.

6.1 Model versioning and release control

Every production model in AlignR is versioned, frozen, and associated with its artefact bundle. A version is immutable after release. The release process requires the artefact bundle to be complete (all six artefacts present and signed) before the model is promoted to production.

Version	Base model	Preference data version	Artefact bundle	Release gate	Status
v1.0.0	Llama-3-8B-Instruct	pref-data-v1 · 12,400 pairs	ART-001 → ART-006 complete	CCO sign-off 2026-01-14	production
v1.1.0	Llama-3-8B-Instruct	pref-data-v2 · 15,800 pairs	ART-001 → ART-006 complete	CCO sign-off 2026-03-02	production
v1.2.0-rc	Llama-3-8B-Instruct	pref-data-v3 · 17,200 pairs	ART-001 → ART-005 complete · ART-006 pending	Awaiting CCO review	candidate
v2.0.0-dev	Llama-3-70B-Instruct	pref-data-v3	In progress	Not started	development

§7 Evaluation Framework

AlignR evaluation is structured around three distinct axes. Alignment accuracy measures how well the model's outputs match the preferences of domain expert annotators on held-out queries. Refusal calibration measures whether the model declines exactly the queries it should — neither over-refusing (degrading usefulness) nor under-refusing (creating compliance exposure). Behavioural consistency measures whether outputs remain stable across paraphrased versions of the same query — a property regulators require for high-risk AI systems.

Figure 7

Three-axis evaluation model — alignment, refusal calibration, consistency

Evaluation results are stored in the artefact bundle (ART-005) and must meet all three thresholds before CCO sign-off. Any axis below threshold triggers a targeted retraining cycle.

Regulatory examination note

When a regulator or internal audit function requests evidence of compliance, the response is the artefact bundle export: a versioned ZIP containing the annotation log, reward model card, training run record, aligned model card, red-team report, and CCO sign-off. AlignR generates this bundle as a first-class output. It is not produced after the fact.

§8 Tech Stack

Every component in the AlignR MVP is free-tier or open source. The stack is designed around two constraints: data sovereignty (nothing leaves the private boundary) and reproducibility (any run can be exactly replicated from logged artefacts). Where a free-tier tool is used for the MVP demo but would be replaced in production, this is noted explicitly.

Component	Tool	Version	Role	Justification	MVP tier
Base model	Llama-3-8B-Instruct	Meta · 2024	Foundation for fine-tuning	Best open-weight instruct model at 8B parameter scale. Fits on a single A100 40GB for LoRA fine-tuning. Llama licence permits enterprise use under 700M MAU threshold.	open-weight
Fine-tuning framework	TRL	HuggingFace · ≥0.8	DPO and PPO training loop	Only open-source library with production-grade native DPO and PPO implementations. Tight PEFT integration. Actively maintained by HuggingFace alignment team.	open source
PEFT / LoRA	PEFT	HuggingFace · ≥0.10	Parameter-efficient fine-tuning	Reduces trainable parameters from ~8B to ~20M for r=16 LoRA. Makes fine-tuning feasible on free-tier GPU. Standard in RLHF pipelines.	open source
Annotation platform	Argilla	≥2.0 · self-hosted	Preference data collection UI	Purpose-built for NLP annotation including preference ranking. Native inter-annotator agreement scoring. Python-first API exports directly to TRL-compatible dataset format. Self-hostable — data stays in VPC. See ADR-004.	open source
Experiment tracking & artefact registry	MLflow	≥2.10 · self-hosted	Run logging, model versioning, artefact store	Full data sovereignty when self-hosted. Structured run logging maps directly to AI Act conformity artefacts. No retention limits. See ADR-005.	open source
Compute — training	Google Colab (free tier) or Kaggle	A100 40GB / T4 16GB	GPU for fine-tuning runs	Colab free tier provides A100 40GB (session-limited). Kaggle provides 30hr/week T4 16GB — sufficient for QLoRA on 8B. No cost for MVP. Production upgrade: dedicated A10G or A100 node.	free tier
Inference — local demo	Ollama	≥0.3	Local inference serving for portfolio demo	Runs quantised 8B models on consumer hardware (M-series Mac, RTX 3080+). REST API compatible. Zero infrastructure cost. Not production-scalable — see ADR-006 for upgrade path.	demo only
Inference — production path	vLLM	≥0.4	Production inference serving	PagedAttention enables continuous batching and high throughput on GPU. Documented as the production upgrade target from Ollama. Not used in MVP due to GPU server requirement.	production path
Evaluation harness	lm-evaluation-harness	EleutherAI · ≥0.4	Benchmark evaluation	Standard open-source LLM evaluation framework. Consistent methodology across model versions. Benchmark results produced as reproducible artefacts.	open source
Bootstrap preference dataset	Anthropic HH-RLHF	Public · HuggingFace Hub	Seed dataset for MVP demo	161k preference pairs (helpful, harmless). Used to demonstrate the annotation-to-training pipeline before domain-specific pairs are collected. In production, replaced entirely by internal annotation.	demo seed
Dataset format & hub	HuggingFace Datasets	≥2.18	Dataset loading, formatting, versioning	Native TRL integration. Arrow-backed, memory-efficient. Consistent dataset versioning via dataset card and commit hash.	open source
Containerisation	Docker	≥24	Reproducible environment for all pipeline stages	Each pipeline stage runs in a pinned Docker image. Ensures reproducibility across annotator workstations, training cluster, and inference node. Image digests are compliance artefacts.	open source

Stack integrity note

The stack has no external data dependency at training or inference time. HuggingFace Hub is used to pull base model weights once — after which the model is stored in the internal registry. All annotation, training, and inference operates within the private boundary. The only external dependency is the bootstrap dataset, which is used for MVP demonstration only and replaced by internal annotation in production.

§9 Architecture Decision Records

Architecture Decision Records document the significant design choices made in AlignR, the alternatives that were evaluated, and the reasoning behind each decision. They are written in MADR format (Markdown Architectural Decision Records). For a compliance-oriented system, ADRs serve a secondary function: they are evidence that design choices were deliberate, considered, and documented — not accidental.

ADR-001

Use open-weight model over proprietary API

Accepted

Date

2026-01-10

Context

AlignR targets regulated enterprises operating under the EU AI Act's high-risk provisions. Conformity assessment requires the operator to demonstrate weight ownership, training data lineage, and the ability to freeze and version the deployed model. Proprietary API-based fine-tuning (OpenAI SFT, AWS Bedrock) satisfies none of these requirements: weights are inaccessible, training data goes to third-party infrastructure, and model versions are controlled by the vendor. Additionally, the GDPR data chain is broken when preference data — which may contain examples drawn from real enterprise workflows — is sent to an external API.

Decision

Use Meta Llama-3-8B-Instruct as the base model. All training, inference, and artefact storage runs on self-hosted infrastructure within the enterprise's private boundary. No model weights or preference data leave the VPC.

Considered

chosen

Llama-3-8B-Instruct — Best open-weight instruct model at the 8B scale. Strong benchmark performance. Llama licence permits enterprise deployment. Fits on A100 40GB for LoRA fine-tuning.

rejected

OpenAI GPT-4o SFT — SFT only, no RLHF. Weights inaccessible. Preference data sent to OpenAI servers. Cannot satisfy AI Act conformity artefact requirements.

rejected

Mistral-7B-v0.3 — Strong alternative. Apache 2.0 licence is more permissive than Llama. Rejected for MVP in favour of Llama-3 due to stronger instruction-following benchmark scores. Documented as a viable swap.

rejected

Llama-3-70B-Instruct — Superior quality ceiling. Rejected for MVP due to GPU requirements (requires multi-GPU or 80GB A100 for LoRA). Documented as the production upgrade path.

Consequences

Self-managed infrastructure required. GPU compute cost falls on the operator. No vendor SLA for model availability — mitigated by internal model registry with pinned weights. Full AI Act conformity posture achieved. Llama licence must be reviewed if deployment exceeds 700M MAU.

ADR-002

DPO as default fine-tuning strategy over PPO

Accepted

Date

2026-01-12

Context

AlignR must support both DPO and PPO, but requires a default strategy that is appropriate for a first deployment and for the MVP build. PPO-based RLHF requires reward model inference during the training hot path, making the training loop more complex, more memory-intensive, and more prone to instability. DPO reformulates the RLHF objective as a supervised classification problem on preference pairs, eliminating the online reward model dependency. Both strategies require a reward model as an evaluation artefact — but DPO does not require it at training time.

Decision

DPO is the default strategy for AlignR's first fine-tuning stage. PPO is supported as an advanced configuration for cases where iterative reward shaping is required — for example, when compliance standards change mid-cycle and the reward signal must be updated without restarting from preference collection. The reward model is always trained and versioned as a compliance artefact, regardless of which strategy is used for fine-tuning.

Considered

chosen

DPO (Direct Preference Optimisation) — Stable training, no online reward inference, lower GPU memory footprint, simpler debugging. Loss curve is a direct compliance artefact. Rafailov et al., 2023.

advanced

PPO-based RLHF — Supported as AlignR advanced mode. Required when reward signal must be updated iteratively. Higher infrastructure complexity. KL penalty prevents degeneration.

rejected

RAFT (Reward rAnked Fine-Tuning) — Generates responses, filters by reward model, fine-tunes on top-k. Simpler than PPO but requires many reward model inference calls. Less principled than DPO for the compliance use case.

rejected

KTO (Kahneman-Tversky Optimisation) — Does not require paired preferences — useful when annotation is unpaired. Rejected because AlignR's annotation workflow produces paired data by design, making DPO the more principled choice.

Consequences

DPO training is stable and reproducible on free-tier GPU. The reward model is decoupled from training, which simplifies the training pipeline but means reward signal cannot be adjusted mid-training epoch. The reward model's validation accuracy is reported as a separate metric and stored as ART-002. If domain standards evolve significantly, re-running the reward model training stage is faster than restarting a PPO loop.

ADR-003

LoRA over full fine-tuning for parameter-efficient training

Accepted

Date

2026-01-12

Context

Full fine-tuning of Llama-3-8B requires approximately 80GB GPU VRAM in bfloat16 — beyond the capacity of any free-tier compute environment and expensive even on dedicated infrastructure. A parameter-efficient approach is required. LoRA (Low-Rank Adaptation) injects trainable low-rank matrices into the attention layers of the frozen base model, reducing trainable parameters from ~8B to approximately 20M for rank r=16. This makes fine-tuning feasible on a single A100 40GB (Colab) or, with 4-bit quantisation via QLoRA, on a T4 16GB (Kaggle).

Decision

Use LoRA (r=16, α=32) via HuggingFace PEFT as the default fine-tuning configuration. QLoRA (4-bit base model quantisation + LoRA) is the fallback configuration for T4 environments. LoRA adapters are merged into base model weights before deployment — the deployed model is a standard model file with no runtime adapter dependency. Default target modules: q_proj, v_proj, k_proj, o_proj.

Considered

chosen

LoRA r=16 via PEFT — ~20M trainable parameters. Fits on A100 40GB. After training, adapters are merged into base weights — clean deployment with no runtime dependency. Hu et al., 2021.

fallback

QLoRA (4-bit + LoRA) — Quantises base model to 4-bit (NF4), trains LoRA adapters in bfloat16. Fits on T4 16GB. Small quality penalty vs. full-precision LoRA. Used on Kaggle free tier.

rejected

Full fine-tuning — Best quality ceiling. Rejected for MVP: requires 80GB+ VRAM, incompatible with any free-tier compute. Documented as production upgrade path for 70B model.

rejected

Prefix tuning / prompt tuning — Fewer parameters, lower quality impact on alignment tasks. Not well-supported in TRL's DPO/PPO implementations. Rejected in favour of LoRA's stronger benchmark results on instruction-following tasks.

Consequences

A merge step (model.merge_and_unload()) is required before the model can be served by Ollama or vLLM. This step is part of the release pipeline and produces the final model checkpoint stored as ART-004. The merge step takes approximately 5 minutes on CPU and has no GPU requirement. LoRA rank is a hyperparameter logged in the training run record (ART-003). Higher rank improves expressiveness at the cost of memory; r=16 is the validated default for 8B alignment tasks.

ADR-004

Argilla over Label Studio for preference annotation

Accepted

Date

2026-01-14

Context

AlignR requires an annotation platform that: (1) supports pairwise preference ranking as a first-class task type, (2) computes inter-annotator agreement automatically, (3) exports datasets in a format compatible with HuggingFace Datasets and TRL, (4) can be self-hosted within the private VPC to satisfy data governance requirements, and (5) is free and open source for the MVP build. Two primary candidates were evaluated: Argilla and Label Studio. Both are open source and self-hostable.

Decision

Use Argilla (≥2.0), self-hosted via Docker Compose. Deployed on the same internal network as the training cluster. Annotators access via browser. Datasets exported via Argilla Python SDK directly to HuggingFace Datasets format for ingestion by TRL.

Considered

chosen

Argilla ≥2.0 — Native preference ranking task type. Built-in inter-annotator agreement (Cohen's κ). Python-first SDK exports directly to datasets.Dataset format. Active development by NLP-focused team. Designed for LLM alignment workflows.

rejected

Label Studio — More general-purpose. Requires custom labelling interface configuration for preference ranking. Inter-annotator agreement requires manual computation. Export format requires transformation before TRL ingestion. Higher setup overhead for this specific task.

rejected

Prodigy (Explosion AI) — Excellent UX and NLP-native. Rejected: paid licence (~$490 one-time). Incompatible with free-tier MVP constraint.

rejected

Custom annotation UI — Full control over UX and data model. Rejected: significant engineering overhead. Inter-annotator agreement, session management, and audit logging would all need to be built from scratch — functionality Argilla provides out of the box.

Consequences

Argilla requires a running Docker Compose stack (Argilla server + Elasticsearch backend). This adds operational overhead compared to a hosted solution but maintains full data sovereignty. Annotator onboarding requires browser access to the internal Argilla URL — no external accounts needed. The Argilla dataset export (Python SDK) is part of the pipeline's data preparation stage and is scripted, not manual.

ADR-005

MLflow over Weights & Biases for experiment tracking

Accepted

Date

2026-01-15

Context

AlignR requires experiment tracking that serves two purposes simultaneously: operational (hyperparameter logging, loss curves, model comparison) and compliance (artefact versioning, run reproducibility, audit trail). The tracking system must be self-hosted — any solution that sends run data to external servers violates the data governance model. Weights & Biases (W&B) is the industry-standard tool for LLM training tracking but its free tier has data retention limits and all run data is sent to W&B servers. Neptune and Comet have similar constraints.

Decision

Use MLflow (≥2.10), self-hosted, as both the experiment tracker and the primary artefact registry. MLflow runs on the same internal infrastructure as the training cluster. The MLflow artefact store (local filesystem or internal S3-compatible object store) holds all six compliance artefact types. Run IDs serve as immutable references in the compliance sign-off document (ART-006).

Considered

chosen

MLflow (self-hosted) — Full data sovereignty. No retention limits. Artefact store doubles as compliance registry. Run metadata queryable via Python SDK. Widely supported by TRL via mlflow.autolog().

rejected

Weights & Biases (hosted) — Superior UI and collaboration features. Rejected: free tier sends all run data to W&B servers. Incompatible with data governance model. Data retention limits on free tier would cause artefact loss.

rejected

Neptune.ai — Good artefact versioning. Rejected: no self-hosted option on free tier. Same data sovereignty concern as W&B.

considered

DVC (Data Version Control) — Strong for dataset versioning and pipeline reproducibility. Could complement MLflow for dataset lineage tracking. Not a replacement — lacks run tracking and model registry features. Documented as a potential addition for dataset versioning in a future iteration.

Consequences

MLflow's UI is less polished than W&B. Run comparison and visualisation require more manual effort. This is an acceptable tradeoff given the compliance requirement. The MLflow tracking server must be kept running and backed up — it is a primary compliance infrastructure component. Backup policy: nightly snapshot of the MLflow artefact store to an internal object store. Retention: indefinite for released model versions.

ADR-006

Ollama for MVP demo inference, vLLM as production upgrade path

Accepted

Date

2026-01-15

Context

AlignR's production deployment architecture specifies vLLM as the inference server — it provides PagedAttention-based continuous batching, high throughput, and a production-grade REST API. However, vLLM requires a dedicated GPU server, which is not feasible for an MVP portfolio build on free-tier infrastructure. A local inference solution that can demonstrate the aligned model's outputs without GPU server infrastructure is required for the portfolio demonstration.

Decision

Use Ollama for local inference in the MVP demonstration. The aligned model (after LoRA merge) is exported to GGUF format via llama.cpp convert scripts and loaded into Ollama. This enables inference on consumer hardware (Apple M-series, RTX 3080+) with no server infrastructure. vLLM is documented as the production upgrade path and its deployment configuration is included in the design documentation, but it is not part of the MVP build.

Considered

MVP

Ollama — Runs GGUF-quantised 8B models on consumer hardware. Simple REST API (localhost:11434). Zero infrastructure cost. Sufficient for portfolio demonstration of aligned model behaviour.

production

vLLM — PagedAttention, continuous batching, high throughput. Production-grade REST API (OpenAI-compatible). Requires dedicated A10G or A100 GPU node. Documented as the upgrade target when moving from demo to production.

rejected

HuggingFace Transformers pipeline (direct) — Works but no batching, slow for multi-user scenarios, no REST API out of the box. Adequate for notebook demos but not for a simulated API endpoint.

rejected

llama.cpp server (direct) — Technically equivalent to Ollama's backend. Rejected in favour of Ollama's higher-level model management and cleaner REST API, which is directly compatible with the vLLM upgrade path.

Consequences

The GGUF export step (LoRA merge → safetensors → GGUF conversion) adds approximately 15 minutes to the release pipeline for the MVP build. This step is scripted and reproducible. Quantisation level: Q4_K_M by default (good quality/size tradeoff for 8B). The production vLLM path skips GGUF conversion entirely — it loads safetensors directly. The two paths are explicitly documented in the operational runbook to avoid confusion during production migration.

§10 Data Schema

The data schema defines the exact structure of the three primary data artefacts produced by AlignR: the preference pair (the unit of annotation), the annotation log record (the compliance-facing version of the preference pair), and the artefact bundle manifest (the index that ties all six compliance artefacts together for a given model version). These schemas are stable across model versions — schema changes are breaking changes and require a new major version.

10.1 Preference pair

The preference pair is the unit the reward model trains on and the unit TRL's DPO trainer consumes. It is a triple of prompt, chosen response, and rejected response. The schema below is the internal representation — the annotation log record below wraps this with annotator metadata.

preference_pair.json

TRL · DPO-compatible

// Preference pair — unit consumed by TRL DPOTrainer
// Stored as JSONL (one record per line) in the preference dataset
{
  "id":          "pair_20260115_004821_uuid4",  // globally unique · deterministic from content hash
  "prompt":      "Member is asking about coverage for an MRI scan ordered
                  by their GP. How should the assistant respond?",
  "chosen":      "Based on your plan details, an MRI scan ordered by a GP
                  may be eligible for coverage subject to pre-authorisation.
                  I'd recommend contacting our pre-auth team on 0800-XXX to
                  confirm before the appointment is scheduled.",
  "rejected":    "Your MRI should be covered under your plan. Just go ahead
                  and book it — you're entitled to diagnostic imaging when
                  referred by a GP.",
  "domain":      "insurance.member_services",        // taxonomy for stratified sampling
  "category":    "coverage_language",               // alignment target category
  "dataset_ver": "pref-data-insurance-v2",          // dataset version tag
  "split":       "train"                            // train | validation | test
}

10.2 Annotation log record

The annotation log record wraps the preference pair with the metadata required for compliance audit: who made the decision, when, with what confidence, and whether it was reviewed by a second annotator. This is ART-001 in the artefact registry.

annotation_log_record.json

ART-001 · compliance artefact

// Annotation log record — compliance-facing wrapper around preference pair
// Immutable after annotation session closes. Stored in Argilla export + MLflow.
{
  "log_id":           "log_20260115_004821_uuid4",
  "pair_id":          "pair_20260115_004821_uuid4",       // FK → preference pair
  "annotator_id":     "ann_007",                          // pseudonymised · GDPR Art.25
  "annotator_role":   "compliance_officer",               // role, not name
  "timestamp_utc":    "2026-01-15T14:28:21Z",
  "session_id":       "session_20260115_ann007",
  "decision":         "A",                                // A = chosen | B = rejected
  "confidence":       4,                                  // 1–5 Likert · 1=uncertain 5=certain
  "rationale":        "Response B makes a coverage promise ('should be
                        covered', 'you're entitled to') which is non-compliant
                        with our pre-auth policy. Response A correctly defers
                        to pre-authorisation process.",
  "second_review":    false,                              // true if flagged for dual review
  "iaa_score":        0.84,                               // Cohen's κ for this annotator pair
  "argilla_record_id":"argilla_rec_00482",               // Argilla internal ref
  "dataset_ver":      "pref-data-insurance-v2"
}

10.3 Artefact bundle manifest

The artefact bundle manifest is the index document that ties the six compliance artefacts to a specific model version. It is generated automatically at the end of the release pipeline and stored in MLflow. It is the document the CCO or DPO signs as part of ART-006.

artefact_bundle_manifest.json

ART-006 input · release gate

// Artefact bundle manifest — generated at release gate
// Signed by CCO/DPO. Immutable after signing. Stored in MLflow + secure document store.
{
  "manifest_id":     "manifest_alignr_v1.1.0",
  "model_version":   "alignr-insurance-v1.1.0",
  "base_model":      "meta-llama/Llama-3-8B-Instruct",
  "created_utc":     "2026-03-02T09:14:00Z",
  "mlflow_run_id":   "a3f8c2d1e4b09f76",
  "artefacts": {
    "ART-001": { "type": "annotation_log",      "path": "s3://alignr/art/ART-001-v2.jsonl",   "sha256": "a3f8...", "status": "complete" },
    "ART-002": { "type": "reward_model",        "path": "s3://alignr/art/ART-002-v2/",         "sha256": "b7d2...", "status": "complete" },
    "ART-003": { "type": "training_run_record",  "path": "mlflow://runs/a3f8c2d1e4b09f76",    "sha256": "c1e9...", "status": "complete" },
    "ART-004": { "type": "aligned_model",       "path": "s3://alignr/models/v1.1.0/",          "sha256": "d4a1...", "status": "complete" },
    "ART-005": { "type": "red_team_report",     "path": "s3://alignr/art/ART-005-v1.1.0.pdf", "sha256": "e5b3...", "status": "complete" },
    "ART-006": { "type": "cco_signoff",         "path": "s3://alignr/art/ART-006-v1.1.0.pdf", "sha256": "f6c4...", "status": "pending"  }
  },
  "deployment_scope": "insurance.member_services · internal-facing only",
  "release_gate_passed": false,                        // set true on CCO sign
  "next_review_date": "2026-09-02"                       // 6-month reassessment cycle
}

§11 MVP Scope & Build Boundaries

AlignR is designed as a production system. The MVP build demonstrates the full pipeline end-to-end but operates within the constraints of free-tier compute and a portfolio demonstration context. This section is explicit about what was fully implemented, what was approximated, and what is deferred to production. Misrepresenting simulated components as implemented is a design integrity problem — not a positioning one.

MVP Build Record — What Actually Ran

Compute: Kaggle free tier · T4 x2 GPU · No cost
Reward model backbone: distilbert-base-uncased (67MB) — chosen for T4 memory constraints. Production target: Mistral-7B-Instruct on GCP Vertex AI A100.
DPO fine-tuning model: distilgpt2 (82MB) — same rationale. Production target: Mistral-7B-Instruct with LoRA r=16 via PEFT.
Preference dataset: 4,827 validated pairs from Anthropic HH-RLHF (public dataset, zero cost). Production replaces with domain expert annotation via Argilla.
Reward model validation accuracy: 56% (threshold: 55%) — PASS. Expected for a general-purpose tiny model on diverse preference data.
MLflow run IDs: Reward model: 96be536902ae457cba127544216079a2 · DPO: 53fbb2e84f2d417ebb2890444e2d8991
All 6 artefacts produced: ART-001 through ART-005 complete · ART-006 pending CCO sign-off (by design)
Artefact bundle: alignr-v1.0.0-bundle.zip · SHA-256 hashed manifest generated

Fully implemented

Preference data pipeline HH-RLHF bootstrap · 4,827 validated pairs · ART-001 annotation log

Argilla annotation infrastructure Self-hosted · Docker Compose · Elasticsearch backend · running on localhost:6900

Reward model training TRL RewardTrainer · distilbert-base-uncased · 56% val accuracy · ART-002

DPO fine-tuning TRL DPOTrainer · distilgpt2 · preference-aligned · ART-003 + ART-004

MLflow experiment tracking Self-hosted · hyperparams logged · loss curves · artefact store · run IDs recorded

Red-team evaluation suite Refusal calibration · safe response · consistency · ART-005 JSON report

Compliance artefact bundle SHA-256 manifest · ART-001–006 · ZIP export · regulator-submission format

GitHub repo · full pipeline code All scripts, configs, notebooks, Docker Compose files committed and public

Approximated for MVP

Production-scale base model distilbert / distilgpt2 used for T4 compatibility · Mistral-7B is the production target · pipeline mechanics are identical

Domain-specific preference data HH-RLHF bootstrap · production replaces with domain expert annotation campaign (12,000–20,000 pairs)

Domain expert annotators Public dataset used for MVP · production requires compliance officers, underwriters, legal counsel

Red-team query volume MVP: 10-query suite demonstrating pipeline · production: 400–800 adversarial queries with compliance reviewer sign-off

CCO / DPO sign-off ART-006 pending manual sign-off — this is correct by design, not a gap. In production: signed PDF via document management system

Artefact storage MVP: local filesystem + Kaggle output · production: encrypted internal object store with RBAC

Deferred to production

Mistral-7B / Llama-3-8B training Requires A100 40GB · GCP Vertex AI · documented upgrade path · estimated $80–100 compute

LoRA r=16 fine-tuning via PEFT Full LoRA config in configs/dpo_default.yaml · ready to run on A100

vLLM inference serving Requires dedicated GPU node · OpenAI-compatible REST API · documented in serve/vllm_config.yaml

GCP Vertex AI integration Vertex AI Training + Experiments + Endpoints · replaces Kaggle + self-hosted MLflow

PPO full RLHF loop TRL PPOTrainer · supported in configs/ppo_advanced.yaml · higher infra complexity

Automated drift detection + retraining Output monitoring → annotation cycle trigger · 6-month reassessment schedule

Multi-GPU / distributed training DeepSpeed ZeRO-3 · required for 70B base or full fine-tune

What this demonstrates

The MVP demonstrates complete pipeline mechanics end-to-end: preference data → reward model → DPO alignment → red-team evaluation → compliance artefact bundle. Every stage runs real code, produces real outputs, and logs to a real MLflow instance. The model size is a compute constraint, not an architectural one — the same scripts, configs, and artefact schemas run unchanged with Mistral-7B on an A100. The pipeline is the demonstration. The model is a parameter.

§12 Failure Modes & Known Limitations

A production alignment system for regulated industries must be honest about where it can fail. The limitations below are structural — they apply to RLHF-based alignment in general and to AlignR specifically. Understanding them is part of responsible deployment, and documenting them is part of the AI Act's transparency obligation.

Failure mode	Description	Severity	Mitigation in AlignR
Reward hacking	The model finds outputs that score highly on the reward model but violate the spirit of the alignment objective. Common when the reward model is underspecified or the training signal is too narrow. In practice: responses that are formally compliant but unhelpfully vague.	High	KL divergence penalty (DPO β parameter) limits how far the model drifts from the base distribution. Red-team evaluation suite includes reward-hacking probes. Reward model validation set is held out from training.
Annotator bias	Preference data reflects the biases of the annotators who collected it — not a neutral institutional standard. If annotators are drawn from a single team or have shared blind spots, those blind spots become alignment targets.	High	Annotator role diversity requirement (minimum three distinct roles per domain). Inter-annotator agreement (Cohen's κ) threshold of ≥0.7 before a preference pair is included in training. Low-agreement pairs are flagged for adjudication, not discarded.
Coverage gaps in preference data	The alignment is only as good as the query distribution in the preference dataset. Edge cases and low-frequency query types are underrepresented. The model may behave unexpectedly on queries that were not covered during annotation.	Medium	Stratified sampling across query categories enforced during annotation campaign design. Red-team suite specifically targets edge cases and low-frequency failure modes. Coverage report is part of ART-005.
Distributional shift	As the enterprise's operating context changes (new products, regulatory updates, staff changes), the preference data becomes stale. The aligned model continues to reflect the standards at the time of annotation, not current standards.	Medium	Six-month reassessment cycle mandated in the CCO sign-off (ART-006). Output monitoring tracks reward model score distribution on live traffic — drift beyond threshold triggers a new annotation cycle, not a prompt-layer patch.
LoRA quality ceiling	LoRA fine-tuning modifies a small subset of parameters. For highly domain-specific alignment objectives, the expressiveness of a low-rank adapter may be insufficient — the model may not fully internalise the alignment target.	Medium	LoRA rank is a configurable hyperparameter. Default r=16 is validated for instruction-following alignment at 8B scale. For deeper domain specialisation, r=64 or full fine-tuning on a 70B base is the documented upgrade path.
Catastrophic forgetting (partial)	Fine-tuning can degrade general capabilities not represented in the preference data. A model aligned on insurance coverage language may become less capable at adjacent tasks (e.g. general arithmetic, factual recall) if those capabilities were not included in the alignment objective.	Low (LoRA)	LoRA significantly reduces catastrophic forgetting compared to full fine-tuning, because most base model weights remain frozen. General capability benchmarks (MMLU, HellaSwag) are included in the evaluation suite to detect regression.
Reward model as single point of failure	The reward model encodes annotator judgment — but it is itself a learned approximation. If the reward model is poorly calibrated, the RLHF / DPO loop optimises against the wrong signal. Errors in the reward model are silent at training time.	High	Reward model validation accuracy on held-out preference pairs must exceed 80% before training proceeds. The reward model is versioned and stored as ART-002 — it can be independently audited. Held-out validation set is maintained and not used in reward model training.

§13 Cost & Resource Model

The cost model below covers the full AlignR pipeline for a single domain (e.g. insurance member services) at MVP scale. Costs are split between the one-time pipeline run (preference collection through first deployment) and ongoing operational costs per retraining cycle. All compute figures assume free-tier infrastructure for the MVP; production estimates are noted separately.

13.1 One-time pipeline cost (first deployment)

Stage	Resource	MVP estimate	Time estimate	Production estimate
Preference data collection	8 annotators · 6 weeks · 2hr/day	$0 (internal staff time)	6 weeks	$0 (internal) or ~$8–12/hr if contracted
Argilla setup and hosting	Docker Compose · single VM	$0 (local or free-tier VPS)	2–4 hrs setup	~$50/month (t3.medium EC2 equivalent)
Reward model training	A100 40GB · 3–5 epochs · 15k pairs	$0 (Colab free tier)	~4–6 hrs	~$12–18 (Lambda Labs A100 on-demand)
DPO fine-tuning	A100 40GB · LoRA r=16 · 3 epochs	$0 (Colab free tier)	~8–12 hrs	~$24–36 (Lambda Labs A100 on-demand)
Red-team evaluation	Compliance reviewer · 400 queries	$0 (internal staff time)	2–3 days	$0 (internal) or ~$500 contracted
MLflow hosting	Self-hosted · local filesystem	$0	1 hr setup	~$30/month (object store for artefacts)
Total (compute only)	—	$0 (free tier)	~8–10 weeks end-to-end	~$80–100 per pipeline run

13.2 Ongoing retraining cycle cost

A retraining cycle is triggered when output monitoring detects distributional drift, when regulatory standards change, or on the mandatory 6-month reassessment schedule. A retraining cycle does not require restarting preference collection from scratch — it adds a targeted annotation batch covering the area of drift and reruns the fine-tuning and evaluation stages.

Stage	Retraining scope	MVP estimate	Production estimate
Targeted annotation batch	500–2,000 new preference pairs covering drift area	$0 (internal staff)	2–3 weeks annotator time
Reward model retraining	Incremental fine-tune on updated dataset	$0 (Colab)	~$6–10
DPO retraining	Full DPO run on updated preference dataset	$0 (Colab)	~$24–36
Re-evaluation and sign-off	Full red-team + CCO review	$0 (internal)	$0 (internal) or ~$500
Total per retraining cycle	—	$0 (free tier)	~$50–80 compute · 4–5 weeks elapsed

Cost comparison context

The dominant cost in AlignR is not compute — it is annotator time. A full annotation campaign (15,000 pairs, 8 annotators, 6 weeks) represents approximately 1,920 staff hours. At a conservative internal hourly rate, this is the significant line item. Compute costs for the MVP are zero. Production compute costs per pipeline run are in the $80–100 range — negligible relative to annotation effort and well within a regulated enterprise's operational budget.

§14 Glossary

Defined for a compliance and audit audience. Technical terms are given their precise meaning in the context of AlignR — not their general mathematical definition.

RLHF — Reinforcement Learning from Human Feedback

A fine-tuning technique that trains a language model to produce outputs humans prefer, by first learning a reward model from human preference comparisons and then optimising the language model against that reward signal using reinforcement learning (specifically PPO).

DPO — Direct Preference Optimisation

An alternative to PPO-based RLHF that optimises directly on preference pairs without requiring reward model inference during training. Produces equivalent or better alignment results with simpler, more stable training. AlignR's default strategy.

PPO — Proximal Policy Optimisation

The reinforcement learning algorithm used in classical RLHF. Treats the language model as a policy and updates its weights to maximise expected reward, with a clipping constraint that prevents excessively large update steps.

KL Divergence penalty

A regularisation term in both RLHF and DPO that penalises the aligned model for diverging too far from the base model's output distribution. Prevents the model from collapsing to repetitive or degenerate outputs that score highly on the reward model but lack general capability.

Reward model

A separate model trained on preference pairs to predict how a domain expert would score any given (prompt, response) pair. Outputs a scalar score. In AlignR, the reward model is both a training signal (for PPO) and a compliance artefact (ART-002) — it is the machine-readable encoding of annotator judgment.

Preference pair

The unit of annotation in AlignR. A triple of (prompt, chosen response, rejected response) where a domain expert has indicated which response better satisfies the alignment objective. The preference dataset is the collection of all such pairs.

LoRA — Low-Rank Adaptation

A parameter-efficient fine-tuning technique that injects small trainable low-rank matrices into the frozen base model's attention layers. Reduces trainable parameters from ~8 billion to ~20 million, making fine-tuning feasible on a single consumer-grade GPU.

Open-weight model

A language model whose trained weight parameters are publicly released, allowing the operator to download, inspect, modify, fine-tune, and self-host the model. Distinct from open source (which implies training code and data are also released). In the AI Act context, weight ownership is the critical property.

Conformity assessment

The process under the EU AI Act by which an operator of a high-risk AI system documents and demonstrates that the system meets the Act's requirements before deployment. For AlignR, the artefact bundle (ART-001 through ART-006) constitutes the conformity assessment evidence package.

Inter-annotator agreement (IAA)

A statistical measure of how consistently different annotators make the same preference decision on the same query. AlignR uses Cohen's κ. A κ of ≥0.7 is required before a preference pair is included in training. Low IAA indicates ambiguous or poorly specified annotation criteria.

Reward hacking

A failure mode where the fine-tuned model learns to produce outputs that score highly on the reward model without genuinely satisfying the underlying alignment objective. Analogous to Goodhart's Law: when a measure becomes a target, it ceases to be a good measure.

Artefact bundle

The six structured outputs produced by the AlignR pipeline (ART-001 through ART-006) that together constitute the evidence package for AI Act conformity assessment. The bundle is versioned, immutable after release, and tied to a specific model version via the artefact bundle manifest.