diff --git a/README.ko.md b/README.ko.md index f6e5aff..34edd90 100644 --- a/README.ko.md +++ b/README.ko.md @@ -1,6 +1,6 @@ # Synaptic Memory -**인덱싱에 API 호출 0회. 인프라 0. 락인 0.** +**기본 경로는 인덱싱 API 호출 0회. 인프라 0. 락인 0.** LLM 에이전트용 지식 그래프 + MCP 도구 서버. 하이브리드 검색, CDC 기반 실시간 DB 동기화, 한국어 FTS 내장. [![PyPI](https://img.shields.io/pypi/v/synaptic-memory)](https://pypi.org/project/synaptic-memory/) @@ -82,7 +82,7 @@ MCP 도구 → LLM 에이전트가 그래프 기반 멀티턴으로 탐색 ``` **라이브러리가 하는 건 딱 두 가지:** -1. **그래프를 잘 구축한다** — 인덱싱에 LLM 비용 0원 +1. **그래프를 잘 구축한다** — 기본은 비용 없는 deterministic extraction 2. **LLM에게 좋은 도구를 쥐어준다** — 판단은 LLM이, 코드는 데이터만 --- @@ -90,9 +90,12 @@ MCP 도구 → LLM 에이전트가 그래프 기반 멀티턴으로 탐색 ## 설치 ```bash -# 추천 — README의 모든 예제가 작동하는 조합 +# 일반 로컬 그래프 + MCP 조합 pip install "synaptic-memory[sqlite,korean,vector,mcp]" +# LangChain retriever 예제를 실행할 때 추가 +pip install "synaptic-memory[langchain]" + # 또는 Postgres / Kuzu / Qdrant / MinIO 까지 전부 pip install "synaptic-memory[all]" ``` @@ -210,13 +213,14 @@ asyncio.run(main()) |------|---------------|---------------| | GraphRAG 계열 (MS GraphRAG, Cognee, Graphiti) | LLM으로 엔티티 + 관계 + 커뮤니티 요약 추출 | 서사형 코퍼스에서 recall 최상. 대신 문서 추가마다 LLM 토큰 비용 | | LightRAG 계열 | LLM 호출을 쿼리 시점으로 지연 | 인덱스 비용 낮음. 대신 쿼리마다 비용 | -| **Synaptic** | **없음.** 구조·통계 시그널만 (FK, NEXT_CHUNK, phrase DF 허브, MENTIONS) | 비용 0 + 결정론적. 단, 새로운 관계를 스스로 합성하지 않음 | +| **Synaptic 기본 경로** | **없음.** 구조·통계 시그널만 (FK, NEXT_CHUNK, phrase DF 허브, MENTIONS) | 비용 0 + 결정론적. 명시적 cross-reference를 LLM 없이 엣지화 | -인덱싱에 LLM을 쓰지 않습니다. 그래프는 지식 저장소가 아니라 검색 인덱스입니다. -LLM이 합성한 요약이 필요하면 그래프 위에 별도 에이전트 레이어로 쌓으세요 — -Synaptic은 primitive를 제공하고, 합성 여부는 사용자가 선택합니다. +기본 인덱싱은 LLM-free입니다. 그래프는 지식 저장소가 아니라 검색 인덱스입니다. +문서에 명시된 cross-reference는 LLM 없이 `REFERENCES` 엣지로 만들 수 있습니다. +OpenIE를 opt-in하면 bounded/revertible 방식으로 LLM 기반 semantic relation을 +추가할 수 있지만, 기본 deterministic 경로에는 포함되지 않습니다. -> **v0.28+**: `graph.search()`는 하나의 경로만 사용합니다. BM25 + HNSW + +> **현재 API**: `graph.search()`는 하나의 경로만 사용합니다. BM25 + HNSW + > PPR + cross-encoder + MMR 기반 EvidenceSearch 파이프라인입니다. > 예전 `engine=` 스위치는 제거되었으므로 예제는 > `graph.search("질문")`처럼 바로 호출하면 됩니다. @@ -230,7 +234,7 @@ Synaptic은 primitive를 제공하고, 합성 여부는 사용자가 선택합 |------|------| | `deep_search` | **추천.** 검색 → 확장 → 문서 읽기를 한 번에 | | `compare_search` | 복합 질문 자동 분해 + 병렬 검색 | -| `knowledge_search` | 핵심 의미 검색 (v0.14.2+에서 EvidenceSearch 경유) | +| `knowledge_search` | EvidenceSearch 기반 핵심 의미 검색 | | `agent_search` | FTS + 벡터 하이브리드 + intent routing | | `expand` | 1-hop 그래프 이웃 탐색 | | `get_document` | 쿼리 관련 청크만 선별한 문서 전문 | @@ -243,8 +247,9 @@ Synaptic은 primitive를 제공하고, 합성 여부는 사용자가 선택합 | `filter_nodes` | 속성 필터 (>=, <=, contains) — `{total, showing}` 반환으로 카운팅 정확 | | `aggregate_nodes` | GROUP BY + COUNT/SUM/AVG/MAX/MIN + WHERE 사전 필터 | | `join_related` | FK 기반 관련 레코드 조회 — RELATED 엣지 순회 (O(degree)) | +| `top_nodes` | “가장 X한”, “top N”, “최대/최소”, “최근” 질의를 단일 호출로 처리 | -### 인제스트 / CDC 도구 (v0.14.0+) +### 인제스트 / CDC 도구 대화 중에 Claude가 새 자료를 배울 수 있도록 하는 도구. | 도구 | 용도 | @@ -255,7 +260,7 @@ Synaptic은 primitive를 제공하고, 합성 여부는 사용자가 선택합 | `knowledge_ingest_path` | 로컬 CSV/JSONL/TXT 파일 단건 인제스트 | | `knowledge_remove` | 단건 노드 삭제 (엣지 cascade) | | `knowledge_sync_from_database` | CDC 증분 동기화 | -| `knowledge_backfill` | 누락된 임베딩·phrase 허브 복구 (v0.14.4+) | +| `knowledge_backfill` | 누락된 임베딩·phrase 허브 복구 | ### 탐색 도구 | 도구 | 용도 | @@ -301,88 +306,64 @@ graph.reranker_weights = RerankerWeights( await graph.reinforce([node_id], success=True) # 이 결과가 도움 됨 → 다음엔 상위로 ``` ---- +### Memory operating layer -## 벤치마크 +검색을 항상 stateful하게 만들지 않고도 관찰할 수 있습니다. -아래 숫자는 버전 태그가 붙은 역사적 스냅샷입니다. 정확한 재현은 링크된 -스크립트/보고서를 기준으로 하고, 지속적으로 정리되는 결과 로그는 -[`docs/comparison/synaptic_results.md`](docs/comparison/synaptic_results.md)를 보세요. +```python +from synaptic import FeedbackSignal, MemoryScope -### 재현 가능한 임베더-프리 베이스라인 (노트북 2초) +scope = MemoryScope(workspace_id="docs", user_id="alice") +result = await graph.search("환불 예외", record=True, scope=scope) -```bash -pip install "synaptic-memory[korean]" -python examples/benchmark_allganize.py +await graph.record_feedback( + event_id=result.event_id, + signal=FeedbackSignal.EXPLICIT_POSITIVE, + success=True, + scope=scope, +) + +health = await graph.memory_health(scope=scope) +signals = await graph.scan_memory_signals(scope=scope) ``` -결정론적 출력 (v0.16.0): +event, feedback, provenance, health signal은 그래프 metadata로 저장됩니다. +`Node.content`에 섞지 않고, LLM prompt에 원본 metadata 전체를 자동으로 밀어 넣지도 않습니다. -``` -Dataset Corpus Queries MRR R@10 Hit Time --------------------------------------------------------------------------------- -Allganize RAG-ko 200 200 0.947 1.000 200/200 9.3s -Allganize RAG-Eval 300 300 0.911 0.950 285/300 5.9s -``` +--- -임베더·reranker 없이 **EvidenceSearch 파이프라인(BM25 + PPR + MMR)** 만으로 -낸 결과입니다. 전체 소스: -[`examples/benchmark_allganize.py`](examples/benchmark_allganize.py). -원본: [allganize/RAG-Evaluation-Dataset-KO](https://huggingface.co/datasets/allganize/RAG-Evaluation-Dataset-KO). +## 벤치마크와 보고서 -> **v0.16.0 개선 누적**. v0.15.1의 query-mode Kiwi + v0.16.0의 engine default -> 전환으로 Korean 공개 벤치마크 전반에서 MRR이 **+0.22~+0.33 상승**. 영어 -> HotPotQA-24도 **+0.148**. 자세한 ablation: -> [`examples/ablation/run_ablation.py`](examples/ablation/run_ablation.py). +루트 README는 현재 설치 경로와 공개 API를 기준으로 유지합니다. 상세 수치는 +오래된 측정값이 현재 API 계약처럼 보이지 않도록 버전이 붙은 보고서에 둡니다. -### 영어 multi-hop 표준 벤치마크 (500q subset, v0.16.0) +빠른 로컬 smoke: ```bash -pip install "synaptic-memory[eval]" -python examples/ablation/download_benchmarks.py -python examples/ablation/run_tier1_benchmarks.py --subset 500 +synaptic-quickstart --json ``` -| 데이터셋 | 소스 | Docs | MRR@10 | R@5 | Hit@10 | -|---------|------|------|--------|-----|--------| -| HotPotQA dev (distractor) | HuggingFace | 66,635 | **0.784** | 0.585 | 459/500 (91.8%) | -| 2WikiMultihopQA dev | HuggingFace | 56,687 | **0.795** | 0.501 | 456/500 (91.2%) | -| MuSiQue-Ans dev | HuggingFace | 21,100 | 0.590 | 0.379 | 381/500 (76.2%) | +가벼운 한국어 FTS 벤치마크: -HippoRAG2 등 선행 연구와의 해석은 -[docs/comparison/synaptic_results.md](docs/comparison/synaptic_results.md#tier-15--english-multi-hop-standard-benchmarks-v0160) -참고. 직접 비교 지표가 아니므로 엄격한 head-to-head는 조심하지만, -**MuSiQue R@5 0.379 vs HippoRAG2 0.747 격차**는 embedder 도입 시 좁힐 여지. - -### 전체 파이프라인 (임베더 + reranker, pre-v0.16.0 측정) - -아래 수치는 **v0.16.0 engine flip 이전** 측정값으로, **EvidenceSearch + 임베더 -(Ollama `qwen3-embedding:4b`) + cross-encoder reranker(TEI `bge-reranker-v2-m3`)** -조합이었습니다. 재현에는 GPU 환경이 필요합니다. 전체 harness: -[`eval/run_all.py`](eval/run_all.py). v0.16.1에서 재측정 예정. - -### 단일 검색 - -| 데이터셋 | 유형 | 노드 | MRR | Hit | -|---------|------|------|-----|-----| -| KRRA Easy | 한국어 문서 (비공개) | 19,720 | **0.967** | 20/20 | -| KRRA Hard | 한국어 문서 (비공개) | 19,720 | **1.000** | 15/15 | -| X2BEE Easy | PostgreSQL 이커머스 (비공개) | 19,843 | **1.000** | 20/20 | -| assort Easy | 패션 CSV (비공개) | 13,909 | **0.867** | 13/15 | -| HotPotQA-24 | 영어 multi-hop (공개 서브셋) | 226 | **0.964** | 24/24 | +```bash +pip install "synaptic-memory[korean]" +python examples/benchmark_allganize.py +``` -> HotPotQA-24는 24문항 서브셋. 전체 HotPotQA-dev(7,405q) 전체 실행은 -> v0.16.1에서 예정 (PPR O(corpus) 최적화 후). +선택 패키지/API key가 준비된 경우 competitor harness: -### 멀티턴 에이전트 (GPT-4o-mini, 최대 5턴) +```bash +python examples/benchmark_vs_competitors/run_comparison.py --only synaptic +``` -| 데이터셋 | 결과 | -|---------|------| -| KRRA Hard agent | 10-13/15 (67-87%) | -| **X2BEE Hard agent** | **17/19 (89%)** | -| **assort Hard agent** | **12/15 (80%)** | +참고 보고서: -필터 / 집계 / FK 조인 / 카운팅 같은 정형 데이터 질의가 그래프 기반 도구로 end-to-end 동작합니다. +| 보고서 | 내용 | +|--------|------| +| [docs/comparison/synaptic_results.md](docs/comparison/synaptic_results.md) | 재현 가능한 Synaptic 벤치마크 결과와 provenance | +| [docs/REPORT-rag-vs-synaptic.md](docs/REPORT-rag-vs-synaptic.md) | 금융 법령 multi-hop 검색에서 RAG와 synaptic-memory 비교 | +| [docs/REPORT-memory-operating-layer-eval.md](docs/REPORT-memory-operating-layer-eval.md) | memory operating layer 평가와 health/reporting gate | +| [examples/benchmark_vs_competitors/README.md](examples/benchmark_vs_competitors/README.md) | competitor adapter 공정성 주의사항 | --- diff --git a/README.md b/README.md index fe3b145..a3f7de1 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Synaptic Memory -**Zero API calls at index time. Zero infra. Zero lock-in.** +**Default path: zero API calls at index time. Zero infra. Zero lock-in.** A knowledge graph + MCP tool server for LLM agents, with hybrid retrieval, CDC-based live database sync, and Korean FTS built in. @@ -86,7 +86,7 @@ MCP tools → LLM agent explores via graph-aware multi-turn tool use ``` **Two jobs, nothing else:** -1. **Build the graph well** — cheap extraction, no LLM at index time +1. **Build the graph well** — cheap deterministic extraction by default 2. **Give the LLM good tools** — the agent decides what to search --- @@ -94,9 +94,12 @@ MCP tools → LLM agent explores via graph-aware multi-turn tool use ## Install ```bash -# Recommended — covers every example in this README +# Recommended local graph + MCP setup pip install "synaptic-memory[sqlite,korean,vector,mcp]" +# Add this for the LangChain retriever example +pip install "synaptic-memory[langchain]" + # Or everything, including Postgres / Kuzu / Qdrant / MinIO pip install "synaptic-memory[all]" ``` @@ -112,6 +115,7 @@ pip install synaptic-memory[vector] # + usearch HNSW index pip install synaptic-memory[mcp] # + MCP server for Claude pip install synaptic-memory[embedding] # + aiohttp for embedding APIs pip install synaptic-memory[reranker] # + flashrank cross-encoder +pip install synaptic-memory[langchain] # + LangChain retriever adapter pip install synaptic-memory[postgresql] # + asyncpg + pgvector pip install synaptic-memory[docs] # + xgen-doc2chunk (PDF/DOCX/PPTX/XLSX/HWP) ``` @@ -250,17 +254,17 @@ goes from **32% (standard dense RAG)** / **31% (HippoRAG2)** to |----------|-----------------|-----------| | GraphRAG-style (MS GraphRAG, Cognee, Graphiti) | LLM extracts entities + relations + community summaries | Highest recall on narrative corpora, but every new document costs LLM tokens | | LightRAG-style | LLM deferred to query time | Less index cost, but each query pays | -| **Synaptic** | **None.** Structural + statistical signals (FK, NEXT_CHUNK, phrase DF hubs, MENTIONS) + rule-based REFERENCES edges | Cheapest, deterministic; extracts explicit cross-references but won't *LLM-synthesize* fuzzy semantic relations | +| **Synaptic default** | **None.** Structural + statistical signals (FK, NEXT_CHUNK, phrase DF hubs, MENTIONS) + rule-based REFERENCES edges | Cheapest, deterministic; extracts explicit cross-references without LLM calls | -No LLM at indexing. The graph is a search index, not a knowledge base. +By default, indexing is LLM-free. The graph is a search index, not a knowledge base. Cross-references that documents state explicitly (statute article citations, clause codes) are turned into `REFERENCES` edges with -zero LLM (see [Option D](#option-d-relation-ontology-multi-hop)). What -Synaptic does *not* do is LLM-synthesize fuzzy semantic relations or -community summaries — if you need those, layer them with your own -agent; Synaptic gives you the primitives. +zero LLM (see [Option D](#option-d-relation-ontology-multi-hop)). +If you opt into OpenIE, Synaptic can add bounded, revertible +LLM-extracted semantic relations; that is not part of the default +deterministic path. -> **v0.28+**: `graph.search()` has one retrieval path: the hybrid +> **Current API**: `graph.search()` has one retrieval path: the hybrid > EvidenceSearch pipeline (BM25 + HNSW + PPR + cross-encoder + MMR). > The old `engine=` switch was removed, so examples should call > `graph.search("question")` directly. @@ -274,7 +278,7 @@ agent; Synaptic gives you the primitives. |------|---------| | `deep_search` | **Recommended.** Search → expand → read documents in ONE call | | `compare_search` | Auto-decompose multi-topic queries, search in parallel | -| `knowledge_search` | Core semantic search (routes through EvidenceSearch in v0.14.2+) | +| `knowledge_search` | Core semantic search through EvidenceSearch | | `agent_search` | FTS + vector hybrid search with intent routing | | `expand` | 1-hop graph neighbours | | `get_document` | Full document with query-relevant chunks | @@ -287,13 +291,13 @@ agent; Synaptic gives you the primitives. | `filter_nodes` | Property filter (>=, <=, contains) — returns `{total, showing}` for accurate counting | | `aggregate_nodes` | GROUP BY + COUNT/SUM/AVG/MAX/MIN with optional WHERE pre-filter | | `join_related` | FK-based related record lookup — walks RELATED edges (O(degree)) | -| `top_nodes` | **Single-call top-N ranking** — "가장 X한" / "top N" / "최대/최소" / "최근" questions without composing aggregate_nodes. Each row carries `sort_value` for chaining into join_related / filter_nodes(from_ids=...). v0.18-β2+. | +| `top_nodes` | **Single-call top-N ranking** — "가장 X한" / "top N" / "최대/최소" / "최근" questions without composing aggregate_nodes. Each row carries `sort_value` for chaining into join_related / filter_nodes(from_ids=...). | All four structured tools emit `hints` on 0-result returns (alternate operator, dropped WHERE, fuzzy column match) so the agent's next turn gets a concrete corrective action instead of a retry loop. -### Ingest / CDC tools (v0.14.0+) +### Ingest / CDC tools Mid-conversation ingestion so Claude can teach itself new material without leaving the chat. | Tool | Purpose | @@ -304,7 +308,7 @@ Mid-conversation ingestion so Claude can teach itself new material without leavi | `knowledge_ingest_path` | Ingest a CSV / JSONL / text file from the local filesystem | | `knowledge_remove` | Delete a single node with edge cascade | | `knowledge_sync_from_database` | Incremental sync from a live database (CDC) | -| `knowledge_backfill` | Repair graphs missing embeddings or phrase hubs (v0.14.4+) | +| `knowledge_backfill` | Repair graphs missing embeddings or phrase hubs | ### Navigation tools | Tool | Purpose | @@ -351,209 +355,66 @@ graph.reranker_weights = RerankerWeights( await graph.reinforce([node_id], success=True) # this result helped → lift it next time ``` ---- - -## Benchmarks +### Memory operating layer -The numbers below are version-tagged historical snapshots. Use the linked -scripts/reports for exact reproduction, and prefer -[`docs/comparison/synaptic_results.md`](docs/comparison/synaptic_results.md) -for a continuously curated result log. +Retrieval can be observed without making every search stateful: -### Reproducible FTS-only baseline (< 2 seconds on a laptop) +```python +from synaptic import FeedbackSignal, MemoryScope -```bash -pip install "synaptic-memory[korean]" -python examples/benchmark_allganize.py -``` +scope = MemoryScope(workspace_id="docs", user_id="alice") +result = await graph.search("refund exception", record=True, scope=scope) -Output (deterministic, v0.16.0): +await graph.record_feedback( + event_id=result.event_id, + signal=FeedbackSignal.EXPLICIT_POSITIVE, + success=True, + scope=scope, +) +health = await graph.memory_health(scope=scope) +signals = await graph.scan_memory_signals(scope=scope) ``` -Dataset Corpus Queries MRR R@10 Hit Time --------------------------------------------------------------------------------- -Allganize RAG-ko 200 200 0.947 1.000 200/200 9.3s -Allganize RAG-Eval 300 300 0.911 0.950 285/300 5.9s -``` - -This is the **embedder-free baseline** (EvidenceSearch pipeline: BM25 + -PPR + MMR, no vector index, no cross-encoder). Full source: -[`examples/benchmark_allganize.py`](examples/benchmark_allganize.py). -Data source: [allganize/RAG-Evaluation-Dataset-KO](https://huggingface.co/datasets/allganize/RAG-Evaluation-Dataset-KO). - -> **v0.16.0 — engine default flipped to `"evidence"`.** Combined with -> the v0.15.1 query-mode Kiwi improvement, FTS-only Korean retrieval -> moved from **Allganize RAG-ko MRR 0.621 (v0.15.0) → 0.947 -> (v0.16.0)** without any embedder or reranker. English (HotPotQA-24) -> 0.727 → 0.875. Full ablation: -> [`examples/ablation/run_ablation.py`](examples/ablation/run_ablation.py). -> Reproducibility under streaming ingest sharpens in lockstep: top-1 -> rank invariance rose from 54.5 % → **100 %**, bit-wise top-10 -> identical from 51.5 % → **96 %**, with MRR drift exactly zero. -### RAG vs HippoRAG2 vs synaptic-memory — multi-hop retrieval (v0.25) +Events, feedback, provenance, and health signals are stored as graph metadata. +They are not appended to `Node.content`, and they are not automatically dumped +into LLM prompts. -RAG (chunk → top-k → one LLM answer), HippoRAG2 (NeurIPS'24 LLM-entity -graph + PPR), and the synaptic-memory agent measured head-to-head on -the same corpus, ground truth, and strict scoring (`finreg` — 4,417 -Korean financial-statute articles, 120 multi-hop queries): - -``` -System multi-hop solved ---------------------------------------------------- -RAG (FTS / BM25) 0/120 (0%) -RAG (dense, bge-m3) 39/120 (32%) -HippoRAG2 (LLM graph + PPR) 37/120 (31%) -synaptic-memory (agent) 88/120 (73%) -``` +--- -A statute article that cites another article ("제30조에 따라 …") is a -**multi-hop** query: the cited provision shares no query vocabulary, so -single-shot retrieval structurally cannot reach it. Notably, **HippoRAG2 -ties plain dense RAG (31% ≈ 32%)** — an LLM-extracted *entity* graph -cannot capture an exact *structural* citation. synaptic-memory turns -cross-references into `REFERENCES` graph edges -([`StructuralReferenceLinker`](src/synaptic/extensions/structural_reference_linker.py), -LLM-free, auto-derived from the corpus) and the agent follows them — -2.3× the nearest competitor. Full report, model notes, and limits: -[`docs/REPORT-rag-vs-synaptic.md`](docs/REPORT-rag-vs-synaptic.md). +## Benchmarks And Reports -### Head-to-head vs Mem0 / Cognee / HippoRAG2 +The root README stays current with the install path and public API. Detailed +numbers are versioned in reports so old measurements do not look like the +current API contract. -A runnable harness (BEIR-style corpus, same MRR / R@10 scoring code -for every system): +Run a quick local smoke: ```bash -# Synaptic only (no API keys, ~2 s) -python examples/benchmark_vs_competitors/run_comparison.py --only synaptic - -# Adapters for Mem0, Cognee, HippoRAG2 ship in-tree — add them to -# --only when the respective packages are installed and API keys are set -python examples/benchmark_vs_competitors/run_comparison.py --only synaptic,mem0 --subset 10 +synaptic-quickstart --json ``` -See [examples/benchmark_vs_competitors/README.md](examples/benchmark_vs_competitors/README.md) -for fairness caveats. Competitor self-reported numbers (Mem0 LoCoMo -91.6, HippoRAG2 MuSiQue F1 51.9, the Zep 84→58 correction incident, -etc.) are catalogued with sources in -[docs/comparison/published_numbers.md](docs/comparison/published_numbers.md). - -### Embedder-free full-dataset summary (v0.16.0) - -Run via `python examples/ablation/run_ablation.py`: - -| Dataset | Lang | Queries | MRR | Hit @ 10 | -|---------|------|---------|-----|----------| -| Allganize RAG-ko | ko | 200 | **0.947** | 200/200 | -| Allganize RAG-Eval | ko | 300 | **0.911** | 285/300 | -| AutoRAG KO | ko | 114 | **0.906** | 114/114 | -| PublicHealthQA KO | ko | 77 | **0.546** | 64/77 | -| HotPotQA-24 EN | en | 24 | **0.875** | 24/24 | - -### English multi-hop standard benchmarks (v0.16.0, subset) - -Run via: +Run the lightweight Korean FTS benchmark: ```bash -pip install "synaptic-memory[eval]" # adds `datasets` for HuggingFace download -python examples/ablation/download_benchmarks.py -python examples/ablation/run_tier1_benchmarks.py --subset 500 +pip install "synaptic-memory[korean]" +python examples/benchmark_allganize.py ``` -Adds HotPotQA-dev (66 k corpus), MuSiQue-Ans-dev (21 k), and -2WikiMultihopQA-dev (57 k) — the three retrieval corpora the -HippoRAG / GraphRAG line of research uses for head-to-head. -Numbers go in -[docs/comparison/synaptic_results.md](docs/comparison/synaptic_results.md#tier-15--english-multi-hop-standard-benchmarks-v0160). - -### Full pipeline (BGE-M3 + cross-encoder) — v0.17.1 - -Measured 2026-04-19 on an H100, `BAAI/bge-m3` + `BAAI/bge-reranker-v2-m3` -loaded in-process via `transformers` (no external TEI / Ollama -endpoint needed): +Run the competitor harness when optional packages/API keys are available: ```bash -python eval/run_all.py --quick --local-bge +python examples/benchmark_vs_competitors/run_comparison.py --only synaptic ``` -#### 14-bench single-shot (5 public + 9 private) - -| Dataset | Lang | Queries | FTS-only | Full pipeline | Δ | -|---------|------|--------:|---------:|--------------:|---:| -| HotPotQA-24 | en | 24 | 0.875 | **0.979** | +0.104 | -| Allganize RAG-ko | ko | 200 | 0.947 | **0.983** | +0.036 | -| Allganize RAG-Eval | ko | 300 | 0.911 | **0.955** | +0.044 | -| PublicHealthQA | ko | 77 | 0.547 | **0.748** | +0.201 | -| AutoRAG KO | ko | 114 | **0.906** | 0.806 | −0.100 ⚠ | -| KRRA Easy | ko | 20 | 0.967 | **0.975** | +0.008 | -| KRRA Hard | ko | 40 | 0.583 | **0.589** | +0.006 | -| KRRA Conv | ko | 30 | 0.146 | **0.166** | +0.020 | -| assort Easy | ko | 15 | 0.760 | **0.856** | +0.096 | -| assort Hard | ko | 40 | 0.000 | 0.000 | 0 | -| assort Conv | ko | 30 | 0.425 | **0.472** | +0.047 | -| X2BEE Easy | en | 20 | 1.000 | 1.000 | 0 | -| X2BEE Hard | en/ko | 20 | **0.379** | 0.368 | −0.011 | -| X2BEE Conv | en/ko | 30 | 0.167 | 0.164 | −0.003 | -| **Mean** | | | 0.615 | **0.647** | **+0.032 (+5.2pp)** | - -v0.17.1 is the **first release where mean Full-pipeline MRR exceeds -mean FTS-only MRR** across all 14 benches (v0.17.0 was net −1.1 %). -12/14 benches improve or hold; the 3 mild residual regressions -(X2BEE Hard / Conv, AutoRAG) are at or below single-query noise -except for AutoRAG where the structural reranker mismatch persists. - -**When the cross-encoder helps — and when it hurts.** The reranker -shines on paraphrase-heavy corpora (PublicHealthQA +20 pp, -HotPotQA +10 pp). On retrieval-style corpora where FTS ranking is -already near-optimal (AutoRAG, X2BEE Hard) it injects noise that -displaces the gold rank. v0.17.1's adaptive blend ( -`std/3` discriminator) and structured-row reranker skip recover -most of that — AutoRAG went −0.264 → −0.100 — but pure -**`reranker=None` is the only way to hit the FTS-only ceiling** on -those corpora. Diagnostic: [`examples/ablation/diagnose_autorag.py`](examples/ablation/diagnose_autorag.py). - -#### Multi-turn agent (Qwen3.5-27B via vLLM, 5 turns, LLM-judge) - -| Dataset | Single-shot | Agent solved | vs v0.13 (gpt-4o-mini) | -|---------|------------:|------------:|----------------------:| -| KRRA Hard | 0.589 | 30/39 (77 %) | 11/15 (73 %) +4 pp | -| assort Hard | 0.000 | 30/33 (91 %) | 13/15 (87 %) +4 pp | -| X2BEE Hard | 0.368 | **19/19 (100 %)** | 17/19 (89 %) +11 pp | -| KRRA Conv | 0.166 | 14/30 (47 %) | 21/30 (70 %) −23 pp | -| assort Conv | 0.472 | 22/24 (92 %) | 20/24 (83 %) +9 pp | -| X2BEE Conv | 0.164 | 25/27 (93 %) | 22/27 (81 %) +12 pp | -| **Mean** | | **140/172 = 81 %** | | - -**This is Synaptic's real number.** Single-shot retrieval is the -floor; the multi-turn agent (`deep_search` + `compare_search` + -graph-context injection) brings the same questions from 0–47 % -to 47–100 %. assort Hard 0/40 → 91 % under agent shows what the -graph + structured tools can do that no single-shot pipeline can. -5 of 6 benches beat the v0.13 GPT-4o-mini baseline (Qwen3.5-27B -upgrade). KRRA Conv regression (Qwen Korean conversational gap) -is the one open issue — v0.18 track. - -#### Known structural gap — MuSiQue (English multi-hop) - -MuSiQue-Ans-dev 500q full pipeline R@5 **0.453** vs HippoRAG2's -published **0.747** (−0.294). Three rounds of targeted fixes (LLM -query decomposition, inline phrase hub, DF-filtered entity linker) -all regressed the score — the gap is structural. Closing it -requires OpenIE triple extraction + query→triple dense linking. -That work now lives as an opt-in, bounded, revertible layer rather -than a default deterministic path. Synaptic's strength is Korean / structured-data RAG; -English Wikipedia multi-hop is honestly documented as a trade-off. -See the OpenIE and memory operating layer plans under `docs/PLAN-v0.30+`. - -### Multi-turn agent (GPT-4o-mini, 5 turns max) - -| Dataset | Result | -|---------|--------| -| KRRA Hard agent | 10-13/15 (67-87%) | -| **X2BEE Hard agent** | **17/19 (89%)** | -| **assort Hard agent** | **12/15 (80%)** | - -Structured data queries (filter / aggregate / FK join / count) work end-to-end through graph-aware tools. +Reference reports: + +| Report | What it covers | +|--------|----------------| +| [docs/comparison/synaptic_results.md](docs/comparison/synaptic_results.md) | Reproducible Synaptic benchmark results with provenance | +| [docs/REPORT-rag-vs-synaptic.md](docs/REPORT-rag-vs-synaptic.md) | RAG vs synaptic-memory on multi-hop financial-statute retrieval | +| [docs/REPORT-memory-operating-layer-eval.md](docs/REPORT-memory-operating-layer-eval.md) | Memory operating layer evaluation and health/reporting gates | +| [examples/benchmark_vs_competitors/README.md](examples/benchmark_vs_competitors/README.md) | Fairness caveats for competitor adapters | --- @@ -599,6 +460,8 @@ Agent tools → MCP server → LLM agent | `embedding` | aiohttp for embedding API calls | | `mcp` | MCP server for Claude Desktop/Code | | `sqlite` | aiosqlite backend | +| `langchain` | LangChain retriever adapter | +| `postgresql` | asyncpg + pgvector | | `docs` | xgen-doc2chunk for PDF/DOCX/PPTX/XLSX/HWP loading | ---