The organisational knowledge graph is the most valuable and least governed data asset in the enterprise. It has no schema, no owner, no SLA, and no recovery plan. Left unstructured, knowledge behaves like entropy — it diffuses, degrades, and becomes irrecoverable across geographies, roles, and incentive boundaries. This is a value stream obstruction with measurable downstream cost, and it has a structural, architectural solution.
KnowledgeFlow is a four-layer agentic and generative AI architecture — experience, orchestration, intelligence, and infrastructure — designed to instrument, govern, and accelerate enterprise knowledge flow at regulated-industry scale. Every agent decision is explainable. Every human checkpoint is intentional. Every access policy is auditable. Deployed on Google Cloud Platform, governed by TOGAF ADM Phases A–H, and compliant by structural design with GDPR and the EU AI Act.
Knowledge hoarding and siloing manifest as structurally distinct failure patterns across verticals — but the root cause and the architectural remedy are invariant.
These are not personality types — they are emergent behaviours produced by the absence of a governing knowledge architecture.
Every design decision in KnowledgeFlow traces to this artefact. Drivers inform assessment. Assessment informs goals. Goals constrain principles. Principles govern every subsequent architectural choice through ADM Phases B–H.
KnowledgeFlow is decomposed into four strict layers — each independently deployable, independently scalable, and independently governable. Layer boundaries are service interfaces, not logical groupings.
The System Context diagram positions KnowledgeFlow relative to its users, external systems, and the enterprise boundary. All interfaces are described with protocol, data classification, and trust zone. (Simon Brown, C4 Model, 2018)
Each container is a separately deployable process or data store. Trust zones, network boundaries, and data classification are explicit. Multi-tenancy model: single-tenant per enterprise client — each client receives an isolated GCP project with dedicated Neo4j Aura instance, VPC, and CMEK key ring. Cross-tenant access is structurally impossible by project boundary design.
Each ADR records: the context, the decision, the alternatives considered, and the consequences. ADRs are the evidence artefacts for TOGAF Phase H governance gate reviews.
The ontology is the architecture. This is the canonical schema definition for the KnowledgeFlow property graph model. Every node type, edge type, property, cardinality constraint, and index specification is declared here. Schema changes are governed artefacts — every modification requires an ADR and Architecture Board review. (W3C OWL 2 Web Ontology Language specification, 2012; Robinson et al., Graph Databases, O'Reilly 2nd ed. 2015)
The original document named "the knowledge graph" without defining its schema, making GDPR erasure cascade specifications architecturally incomplete and graph query performance analysis impossible. This tab provides the full ontology specification: node types with typed property schemas, edge types with cardinality constraints, index strategy, schema governance policy, and the MDM authority model for entity resolution (C-3).
When two source systems emit conflicting representations of the same real-world entity, the MDM authority model determines the system of record for each attribute. The entity resolution pipeline applies record linkage before any graph write. (Dong & Srivastava, Big Data Integration, Morgan & Claypool, 2015)
Without entity resolution, the ingestion pipeline accumulates duplicates: one Person node per source system rather than one per real-world person. This table defines the authority model and the conflict resolution policy for every Person node attribute. A formal entity linkage pipeline (blocking → similarity → merge decision) precedes every graph write.
| Attribute | System of Record | Fallback source | Conflict resolution | Refresh cadence |
|---|---|---|---|---|
| person_id | HRIS (Workday) employee_id | None — HRIS is authoritative | If not in HRIS, person node is not created. CRM-only contacts use a separate :ExternalContact node type. | Real-time via SCIM 2.0 event |
| display_name | HRIS preferred name field | CRM full name | HRIS wins. CRM alias stored as alternate_name property. | SCIM PATCH event |
| geo | HRIS work location country | CRM billing country | HRIS wins. OPA policy evaluates HRIS geo for access control — not CRM geo. | SCIM PATCH event |
| seniority_level | HRIS job band | CRM title field | HRIS wins. Seniority drives RBAC role assignment; CRM title stored as display_title only. | HRIS job change event |
| expertise (linked :Expertise nodes) | No single SOR — multi-source | N/A — evidence-weighted merge | Expertise nodes are created per source. The :HAS_EXPERTISE edge carries source and verified_by. Policy agent queries by verified=true for high-sensitivity access. | Source system change event |
| consent_ts | KnowledgeFlow consent service (authoritative) | None — KF owns consent | Consent cannot be sourced from any external system. GDPR Art. 7 requires unambiguous, freely-given consent recorded by the processor. | Real-time on consent event |
A synchronous cascade across Neo4j, Vertex AI Search, BigQuery, and Chronicle is architecturally impossible with ACID guarantees. The correct pattern is the saga with explicit compensating transactions and an idempotency key per step. Each step emits an audit event to Chronicle regardless of outcome. (Richardson, Microservices Patterns, Manning, 2018, ch. 4)
The original diagram showed a linear synchronous erasure flow with no compensation mechanism. A partial failure (e.g., Neo4j node deleted but Vertex AI embedding not yet purged) leaves the data subject in a partially-erased state — itself a potential GDPR violation. This section replaces the synchronous diagram with an orchestration saga: each step is independently retriable, carries an idempotency key, and its compensating transaction is specified.
This section provides the LangGraph state machine graph topology (missing from the original), the per-query-category HITL confidence threshold matrix with calibration methodology, and the FMEA table for every agent node — addressing the "happy path only" architecture critique. (Guo et al., "On Calibration of Modern Neural Networks," ICML 2017)
C-2: A single global confidence threshold of 0.85 is architecturally naive. Uncalibrated cosine similarity scores are not probabilities (Guo et al., ICML 2017). This section defines the per-query-category threshold matrix, the confidence score computation pipeline, and the Platt scaling calibration protocol.
C-5: The original document described only the happy path. Every LangGraph edge is an implicit failure boundary. This section provides the FMEA table for every agent node, the fallback behaviours, and the recovery SLOs.
A single global threshold of 0.85 fails the proportionality requirement of EU AI Act Art. 9. A query seeking EU financial services consultant recommendations carries different epistemic risk than an onboarding documentation query. This matrix defines thresholds per category, the metric each threshold is computed over, and the calibration methodology. (Minderer et al., "Revisiting the Calibration of Modern Neural Networks," NeurIPS 2021)
| Query category | Sensitivity ceiling | Confidence metric | HITL threshold | Calibration method | Rationale |
|---|---|---|---|---|---|
| Expert identification (career-relevant) | confidential | Composite: retrieval MRR@5 × synthesis BERTScore (F1) · Platt-scaled to [0,1] | 0.90 | Platt scaling on 500-query labelled eval set. Temperature parameter re-calibrated quarterly. | Highest epistemic risk — influences career-relevant decisions under EU AI Act Art. 6 high-risk classification. Proportionality requires higher bar. |
| Engagement history lookup | restricted | Retrieval NDCG@10 · Platt-scaled | 0.85 | Platt scaling on 300-query eval set. Accepts lower bar than expert ID as engagement data is factual, not inferential. | Factual record retrieval with lower inferential component. Threshold aligned to original baseline with calibration applied. |
| Methodology / document retrieval | restricted | Retrieval recall@10 · no synthesis component | 0.75 | Recall@10 is directly interpretable. No Platt scaling required — already a probability proxy. | Document retrieval has no inferential synthesis step. Lower threshold appropriate. Reviewer intervention adds latency without proportionate risk reduction. |
| Onboarding / general knowledge | public / restricted | Retrieval recall@10 | 0.70 | Standard recall metric. No calibration required for public-sensitivity content. | Public-sensitivity content. HITL intervention reserved for genuine knowledge gaps, not routine retrieval uncertainty. |
| Cross-geo access (OPA exception path) | any | N/A — policy-triggered HITL, not confidence-triggered | ALWAYS HITL | Not confidence-based. Any query touching geo-exclusion OPA rules is unconditionally routed to HITL. | Geo-exclusion policies protect data sovereignty obligations. No confidence score can override a policy-layer intercept. |
| Departed employee knowledge nodes | any | N/A — lifecycle-triggered HITL | ALWAYS HITL | Not confidence-based. Knowledge transfer protocol triggered on departure; retrieval of departed employee nodes requires explicit DPO review. | GDPR Art. 17 retention review obligation. Cannot be confidence-bypassed. |
Every agent node in the LangGraph state machine is an implicit failure boundary. This FMEA table specifies the failure mode, detection mechanism, fallback behaviour, SLO impact, and recovery procedure for each node. (Newman, Building Microservices, O'Reilly 2nd ed. 2021, ch. 12)
| Agent node | Failure mode | Sev | Detection | Fallback | SLO impact | Recovery |
|---|---|---|---|---|---|---|
| Auth Agent | IAP token validation failure / IdP timeout | CRITICAL | HTTP 4xx returned to gateway · Chronicle alert in <5s | Fail-closed. No query proceeds without valid auth. User receives 401 with support link. | 100% of affected queries fail. No partial degradation. | IAP auto-retries IdP 3×. If IdP recovers, queries resume automatically. If sustained: incident P1, on-call page, SRE intervention. |
| Policy Agent (OPA) | OPA sidecar crash / Rego eval timeout | CRITICAL | GKE liveness probe fail → Pod restart. Chronicle alert. | Fail-closed. If OPA unavailable, zero nodes are authorised. Queries route to HITL with "policy evaluation unavailable" reason code. | All queries escalate to HITL until OPA recovers. HITL queue load spike. | GKE auto-restarts Pod. Policy bundle re-loaded from GCS in <30s. Chronicle records downtime window for compliance review. |
| Retrieval Agent — Neo4j | Cypher query timeout / Neo4j connection pool exhaustion | HIGH | Query latency > 500ms → Circuit breaker opens. Chronicle metric alert. | ANN-only retrieval (Vertex AI Search) proceeds. Result quality degrades — graph relationships not traversed. HITL threshold automatically lowered by 0.05 during Neo4j degradation. | Latency SLO breached for graph-dependent queries. ANN recall compensates partially. | Circuit breaker retry after 30s. Connection pool config reviewed. If sustained: Neo4j instance scaling or query plan investigation. |
| Retrieval Agent — Vertex AI Search | ANN endpoint unavailable / rate limit exceeded | HIGH | 503 response → retry with exponential backoff. Chronicle alert at 3 consecutive failures. | Graph-only retrieval (Cypher) with fulltext index fallback. Coverage degrades for semantic queries. | Semantic recall degrades. Keyword-only retrieval proceeds. | Exponential backoff ×5 (max 8s). If sustained: Vertex AI quota increase request or request throttling at Apigee. |
| Synthesis Agent (Gemini) | Gemini API timeout / hallucination detection trigger | HIGH | Timeout: 30s hard limit per request. Hallucination: groundedness score < 0.6 via Vertex AI Grounding API. | Timeout: retry ×2. Hallucination: route to HITL with "grounding confidence low" reason code. Never deliver ungrounded response. | Affected query SLA extends by HITL review time (4h business hours). | Retry on timeout. HITL review on hallucination. RLHF label added for model improvement. Recurrent hallucination pattern triggers model evaluation review. |
| Audit Agent (Chronicle) | Chronicle ingestion failure / event emission timeout | CRITICAL | gRPC emit returns error. Audit agent enters retry loop. Chronicle alert on sustained failure. | Query delivery is HELD until audit emission succeeds or 3-retry deadline exceeded. No unaudited response is delivered to user (compliance obligation). | Query delivery delayed. User receives "processing — please wait" status. | Retry ×3, 100ms exponential backoff. If Chronicle unavailable >2 min: incident P1. All held queries resume once Chronicle recovers. Idempotency key prevents duplicate audit records. |
| HITL Gate (LangGraph interrupt) | Reviewer queue timeout / no reviewers available | HIGH | Queue age > SLA threshold → Looker alert → architecture owner paged. | User notified of expected review time. Query held in queue — not abandoned. Escalation to KF_Admin role if primary reviewer queue is stale. | Query SLA extends beyond 4h business hours. User experience degrades. | On-call KF_Admin role can review any queue item. Reviewer pool adequacy reviewed in weekly HITL metrics review. |
The RLHF pipeline is itself a high-risk AI system under EU AI Act Art. 6 — it modifies the synthesis agent that influences career-relevant information access. Its architecture must be as explicit as the system it trains. (Ziegler et al., "Fine-Tuning Language Models from Human Feedback," arXiv 2019; Bai et al., Anthropic, 2022; Gao et al., "Scaling Laws for Reward Model Overoptimisation," ICML 2023)
RLHF cannot be a black box labelled "better labels → better model." Reward hacking, reviewer population bias, and training instability are documented failure modes. This section specifies the reward signal definition, the reviewer calibration protocol, the DPO variant used (Direct Preference Optimisation — Rafailov et al., NeurIPS 2023), and the evaluation gate before any model version is promoted.
The EU AI Act's obligations for high-risk AI systems entered force August 2024. KnowledgeFlow spans two risk tiers simultaneously. This section provides the formal conformity assessment approach referencing ISO/IEC 42001:2023 (AI Management Systems), with honest status tracking distinguishing implemented controls from designed commitments.
The original document asserted conformity without a conformity assessment methodology. EU AI Act Art. 43 requires high-risk systems to undergo a conformity assessment. The relevant harmonised standard is ISO/IEC 42001:2023 (AI Management System). This section makes explicit: (1) the assessment is a self-assessment at current stage (pre-deployment), (2) the distinction between "Structural — implemented" and "Designed — committed not yet independently verified", and (3) the path to third-party assessment at production deployment.
| Article | Obligation | KnowledgeFlow architectural response | Status | ISO/IEC 42001 clause | Evidence artefact |
|---|---|---|---|---|---|
| Art. 9 — Risk management | Continuous risk management system throughout lifecycle | LangGraph state machine provides observable, testable risk surface. FMEA table (AG-03) is the risk register. HITL intercepts anomalous agent behaviour. Vertex Model Monitor detects drift. TOGAF ADM Phase H governance gate. | Structural | 6.1.2, 8.4 | AG-03 · ADR-006 |
| Art. 10 — Data governance | Training and input data relevant, representative, free of errors | Dataplex enforces data quality rules at ingestion. PII detection via DLP API. Ontology validation before graph write. MDM authority model (MDM-01) ensures data consistency. Lineage tags trace every element to source and consent record. | Structural | 8.3, 8.4 | MDM-01 · ONT-01 |
| Art. 11 — Technical documentation | Maintain technical documentation demonstrating conformity | This architecture portfolio constitutes the technical documentation artefact. Terraform state provides infrastructure documentation. Model registry provides ML documentation with bias evaluation gate. ADRs 001–006 provide decision documentation. | Designed | 7.5 | Full portfolio · ADRs 1–6 |
| Art. 12 — Record keeping | Automatic logging sufficient for post-hoc reconstruction | Chronicle SIEM retains every agent state transition, policy evaluation, and HITL decision for 24 months. BigQuery retains structured analytical logs. Log schema is defined and version-controlled. Audit Agent is non-bypassable (FMEA: query held if Chronicle unavailable). | Structural | 9.1 | ADR-005 · AG-03 |
| Art. 13 — Transparency | Users must be informed they are interacting with AI | Every synthesised response carries a mandatory disclosure header identifying AI generation, source knowledge nodes, and confidence score. Implemented as a non-optional synthesis agent output constraint — not a UI flag. | Structural | 8.6 | Synthesis agent spec |
| Art. 14 — Human oversight | Natural persons able to oversee, interpret, and override | HITL is a LangGraph interrupt node (ADR-006) — not a UI feature. Non-bypassable by architecture. Per-category threshold matrix (AG-02) ensures proportionate oversight. Reviewer actions (approve/edit/reject) are all first-class audit events. | Structural | 8.5, 9.2 | ADR-006 · AG-02 |
| Art. 15 — Accuracy and robustness | High-risk AI must achieve appropriate accuracy, robust against errors | Model performance SLOs defined in Vertex Model Monitor. Drift detection triggers automatic HITL threshold lowering during investigation. RLHF DPO pipeline with four-gate promotion criteria (AG-03 RLHF section). Fallback to HITL when confidence below threshold. | Structural | 8.4, 9.1 | RLHF spec · Vertex Monitor |
GDPR's six data processing principles under Art. 5 are satisfied by architecture. Each principle maps to structural controls. Refer to Tab 2 (Ontology) for the erasure saga implementation of Art. 17 right to erasure.
Five categories of human intercept are defined. Three are confidence-based (per AG-02 threshold matrix). Two are unconditional policy-triggered intercepts. All are first-class LangGraph interrupt nodes — not UI features.
| Intercept type | Trigger | Reviewer role | Actions | SLA | Audit record |
|---|---|---|---|---|---|
| Confidence intercept | Platt-scaled confidence score below per-category threshold (AG-02). Triggered when retrieved knowledge is sparse, contradictory, or outside model training distribution. | KF_Curator | Approve Edit Reject → knowledge gap flag | 4h business hours | Decision · rationale · reviewer · timestamp → Chronicle |
| Sensitivity escalation | Nodes tagged sensitivity=confidential, or cross-geo OPA override required, or departed employee nodes involved. | KF_Admin or Data Steward | Release with justification · Redact nodes · Deny | 2h (confidential) · 24h (cross-geo) | Policy rule triggered · nodes · access decision · justification |
| GDPR erasure gate | Right-to-erasure DSR received. Saga step S2 — DPO approves cascade scope before any deletion occurs (ERR-01 saga). | Data Protection Officer (KF_Compliance) | Authorise full · Authorise partial · Defer pending legal hold | 72h statutory (GDPR Art. 12) | DSR ref · authorisation · cascade scope · completion timestamp — immutable Chronicle |
| Policy change review | OPA Rego PR affecting geo-exclusion, sensitivity ceilings, or cross-org access. | Architecture owner + DPO co-review | Approve & merge · Approve with conditions · Reject | 5 business days | Policy diff · impact assessment · approval evidence → Git + Chronicle |
| Anomaly alert | Chronicle YARA-L rule detects: bulk queries, unusual geo combination, repeated denied-access, agent behaviour outside baseline. | Security operations (on-call KF_Admin) | Investigate · Suspend session · Escalate to CISO | 30 minutes (P1 alert) | Alert rule · detection evidence · response actions — Chronicle forensic record |
Each audit layer serves a distinct stakeholder with distinct latency and query requirements.
This simulation demonstrates graph traversal behaviour, confidence scoring, and HITL routing. Scalability caveat (M-5): the simulation uses 18 synthetic nodes and is illustrative only. Real-world graphs at 50,000+ nodes exhibit qualitatively different characteristics — query plans switch to index-backed traversals, ANN recall degrades at boundary conditions, and blocking parameters in entity resolution require recalibration. Production performance claims require load testing at target scale before commitment.
Walk through the full KnowledgeFlow query pipeline — each step shows the agent, the action, the data, and the audit event emitted.
The original document presented unsourced unit economics without derivation methodology, conflating infrastructure cost with TCO and presenting projections without confidence intervals. This section provides a bottom-up monthly infrastructure cost estimate with GCP pricing references, a TCO model that includes HITL labour, and a revenue bridge grounded in McKinsey productity research with appropriate epistemic humility about assumptions.
The critique noted: (1) no derivation methodology, (2) conflation of infrastructure cost with TCO, (3) fabricated unit economics. This section resolves all three: infrastructure costs are derived from GCP pricing calculator inputs, TCO includes the full cost stack (infra + HITL labour + change management), and unit economics are computed from stated assumptions with those assumptions made explicit and challengeable.
| Component | Driver | Pilot (100 users) | Mid (1,000 users) | Enterprise (5,000 users) |
|---|---|---|---|---|
| Query volume (monthly) | 10 queries/user/day × 22 working days | 22,000 | 220,000 | 1,100,000 |
| Infrastructure cost/mo | See cost derivation table | €5,800 | €14,200 | €38,900 |
| Infrastructure cost/query | Infra ÷ query volume | €0.264 | €0.065 | €0.035 |
| HITL labour cost/query | 5% intercept × 15min × €80/hr | €0.100 | €0.100 | €0.100 |
| Full TCO cost/query | Infra + HITL (excl. engineering) | €0.364 | €0.165 | €0.135 |
| Value recovered per query | €85k/yr ÷ (220 days × 10 queries/day) × 60% search time reduction | €2.32 / query (McKinsey productivity basis — see assumption note) | ||
The original roadmap was a phase list without dependency constraints. This creates the illusion of parallelism where serial dependencies exist. This version makes dependencies explicit. The critical path runs through identity federation → OPA policy deployment → every downstream access-controlled feature — no access-controlled query is possible until this path completes. (PMBoK, A Guide to the Project Management Body of Knowledge, 7th ed., 2021)
The original roadmap had no dependency graph. Critical path analysis: the longest chain from start to first production query is approximately 26 weeks — gated by identity federation (Workday SCIM + Google Workspace SSO) → OPA policy bundle → GKE pod deployment → Neo4j graph population → first query. Every stream in Phase 2 is blocked until Phase 1 identity and policy foundations complete. This has direct resource planning implications: the identity integration team cannot be de-prioritised without pushing the entire delivery timeline.
Each phase is decomposed into three streams: technical architecture, data & governance, and HITL/operations. Dependencies between phases are explicit (see dependency graph above). Phase exits are gate-based — each phase requires a TOGAF Architecture Board review before the next phase commences.
Architecture that cannot be operated is incomplete. This section specifies the operational model: 10 operational scenarios, cost circuit breakers, HITL fatigue detection, SLO definitions, and the on-call escalation structure. Cross-references to FMEA (AG-03) and the audit architecture.
The ingestion pipeline is where data quality, entity resolution, and lineage tagging converge. This C4-L3 component diagram shows the internal structure of the Ingestion Agent container, exposing the components that were opaque at L2.
Architecturally complete documents acknowledge what they do not yet know. The following are known limitations and open questions at the time of this portfolio submission. Each has a stated resolution path.