Skip to content

Document infrastructure integration options #141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SonAIengine merged 1 commit into from
Jul 3, 2026
Merged

Document infrastructure integration options #141

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 82 additions & 7 deletions README.ko.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,12 @@ MCP 도구 → LLM 에이전트가 그래프 기반 멀티턴으로 탐색
# 일반 로컬 그래프 + MCP 조합
pip install "synaptic-memory[sqlite,korean,vector,mcp]"

# 팀/프로덕션 그래프: PostgreSQL + pgvector
pip install "synaptic-memory[postgresql,embedding,reranker]"

# 스케일아웃 보조 구성: Kuzu 그래프 + Qdrant 벡터 + MinIO blob
pip install "synaptic-memory[scale]"

# LangChain retriever 예제를 실행할 때 추가
pip install "synaptic-memory[langchain]"

Expand All @@ -113,13 +119,71 @@ pip install synaptic-memory[embedding] # + 임베딩 API (aiohttp)
pip install synaptic-memory[reranker] # + flashrank cross-encoder
pip install synaptic-memory[langchain] # + LangChain retriever 어댑터
pip install synaptic-memory[postgresql] # + asyncpg + pgvector
pip install synaptic-memory[mysql] # + aiomysql DB 인제스트
pip install synaptic-memory[oracle] # + oracledb DB 인제스트
pip install synaptic-memory[mssql] # + aioodbc DB 인제스트
pip install synaptic-memory[kuzu] # + 임베디드 property graph 백엔드
pip install synaptic-memory[qdrant] # + Qdrant 벡터 helper
pip install synaptic-memory[minio] # + MinIO/S3 호환 blob helper
pip install synaptic-memory[scale] # + Kuzu + Qdrant + MinIO + aiohttp
pip install synaptic-memory[docs] # + PDF/DOCX/PPTX/XLSX/HWP 로더
```

</details>

---

## 인프라 연계

기본 one-liner는 로컬 SQLite 그래프를 만듭니다. 이미 쓰는 운영 인프라에
붙일 때는 backend를 직접 만들고 연결한 뒤 `from_data()`, `from_chunks()`,
`from_database()`에 넘기면 됩니다.

```python
from synaptic import SynapticGraph
from synaptic.backends.postgresql import PostgreSQLBackend

backend = PostgreSQLBackend("postgresql://user:pass@host:5432/synaptic")
await backend.connect()

graph = await SynapticGraph.from_data("./docs/", backend=backend, preset="rag")
```

현재 backend 역할:

| 경로 | 설치 | 데이터 담당 | 적합한 상황 |
|------|------|-------------|-------------|
| 로컬 앱/노트북 | `sqlite,korean,vector` | SQLite FTS5 + 로컬 usearch HNSW | 빠른 도입, 데모, 작은 서비스 |
| 팀 서비스 | `postgresql,embedding,reranker` | PostgreSQL + pgvector + pg_trgm | 공유 그래프, 백업, SQL 운영 |
| 그래프 중심 임베디드 | `kuzu,korean,embedding` | Kuzu property graph | 로컬 graph traversal / Cypher workflow |
| 스케일아웃 조합 | `scale` | Kuzu 등 graph store + Qdrant + MinIO | graph/vector/blob 책임 분리 |

Qdrant와 MinIO는 단독 그래프 저장소가 아니라 helper service입니다.
`CompositeBackend`를 통해 사용합니다. graph storage는 node/edge를 갖고,
Qdrant는 ANN vector search를 담당하며, MinIO/S3 호환 저장소는 큰
`Node.content`를 외부 blob으로 분리합니다.

```python
from synaptic.backends.composite import CompositeBackend
from synaptic.backends.kuzu import KuzuBackend
from synaptic.backends.minio_store import MinIOBackend
from synaptic.backends.qdrant import QdrantBackend

backend = CompositeBackend(
KuzuBackend("synaptic.kuzu"),
vector=QdrantBackend("http://localhost:6333", collection="synaptic"),
blob=MinIOBackend("localhost:9000", bucket="synaptic"),
)
await backend.connect()

graph = await SynapticGraph.from_data("./docs/", backend=backend, preset="scale")
```

라이브러리는 backend contract와 retrieval layer를 제공합니다. 다만 수 TB급
운영 코퍼스에서는 별도 운영 레이어도 같이 설계해야 합니다. 예를 들면 durable
ingestion queue, parser/OCR worker, 외부 lexical index, tenant/ACL filter,
index lag 모니터링, 각 저장소별 backup/restore가 필요합니다.

## 빠른 시작

### 방법 A: 2줄 (가장 쉬움)
Expand Down Expand Up @@ -390,13 +454,15 @@ StorageBackend (Protocol)

## 백엔드

| 백엔드 | 벡터 검색 | 규모 | 용도 |
|--------|----------|------|------|
| `MemoryBackend` | cosine | ~1만 | 테스트 |
| `SqliteGraphBackend` | **usearch HNSW** | ~10만 | **기본 권장** |
| `KuzuBackend` | HNSW | ~1천만 | 그래프 중심 |
| `PostgreSQLBackend` | pgvector | ~100만 | 프로덕션 |
| `CompositeBackend` | Qdrant | 무제한 | 스케일아웃 |
| 백엔드 | 설치 옵션 | 역할 | 용도 |
|--------|-----------|------|------|
| `MemoryBackend` | core | 인프로세스 그래프 | 테스트와 예제 |
| `SqliteGraphBackend` | `sqlite`, `vector` | 로컬 그래프 + FTS5 + usearch HNSW | 기본 로컬/임베디드 배포 |
| `KuzuBackend` | `kuzu` | 임베디드 property graph + Cypher | 그래프 중심 로컬 workflow |
| `PostgreSQLBackend` | `postgresql` | durable graph + pgvector + pg_trgm | 공유 프로덕션 서비스 |
| `QdrantBackend` | `qdrant` | vector-only helper | `CompositeBackend` 뒤 ANN search |
| `MinIOBackend` | `minio` | blob-only helper | `CompositeBackend` 뒤 큰 content offload |
| `CompositeBackend` | `scale` | graph + vector + blob store 라우터 | 스케일아웃 조합 |

---

Expand All @@ -412,6 +478,15 @@ StorageBackend (Protocol)
| `mcp` | Claude Desktop/Code MCP 서버 |
| `langchain` | LangChain retriever 어댑터 |
| `postgresql` | asyncpg + pgvector |
| `mysql` | aiomysql DB 인제스트 |
| `oracle` | oracledb DB 인제스트 |
| `mssql` | aioodbc DB 인제스트 |
| `kuzu` | 임베디드 Kuzu graph 백엔드 |
| `qdrant` | Qdrant vector helper |
| `minio` | MinIO/S3 호환 blob helper |
| `scale` | Kuzu + Qdrant + MinIO + aiohttp |
| `rag` | spaCy + aiohttp endpoint helper |
| `all` | 주요 DB, vector, MCP, 한국어, reranker 옵션 묶음 |
| `docs` | PDF/DOCX/PPTX/XLSX/HWP 문서 로더 (xgen-doc2chunk) |

---
Expand Down
88 changes: 81 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,12 @@ MCP tools → LLM agent explores via graph-aware multi-turn tool use
# Recommended local graph + MCP setup
pip install "synaptic-memory[sqlite,korean,vector,mcp]"

# Team / production graph on PostgreSQL + pgvector
pip install "synaptic-memory[postgresql,embedding,reranker]"

# Scale-out helpers: Kuzu graph + Qdrant vector + MinIO blob storage
pip install "synaptic-memory[scale]"

# Add this for the LangChain retriever example
pip install "synaptic-memory[langchain]"

Expand All @@ -117,13 +123,70 @@ pip install synaptic-memory[embedding] # + aiohttp for embedding APIs
pip install synaptic-memory[reranker] # + flashrank cross-encoder
pip install synaptic-memory[langchain] # + LangChain retriever adapter
pip install synaptic-memory[postgresql] # + asyncpg + pgvector
pip install synaptic-memory[mysql] # + aiomysql DB ingest
pip install synaptic-memory[oracle] # + oracledb DB ingest
pip install synaptic-memory[mssql] # + aioodbc DB ingest
pip install synaptic-memory[kuzu] # + embedded property graph backend
pip install synaptic-memory[qdrant] # + Qdrant vector helper
pip install synaptic-memory[minio] # + MinIO/S3-compatible blob helper
pip install synaptic-memory[scale] # + Kuzu + Qdrant + MinIO + aiohttp
pip install synaptic-memory[docs] # + xgen-doc2chunk (PDF/DOCX/PPTX/XLSX/HWP)
```

</details>

---

## Infrastructure Integration

The default one-liner creates a local SQLite graph. For existing
infrastructure, create the backend yourself, connect it, and pass it to
`from_data()`, `from_chunks()`, or `from_database()`.

```python
from synaptic import SynapticGraph
from synaptic.backends.postgresql import PostgreSQLBackend

backend = PostgreSQLBackend("postgresql://user:pass@host:5432/synaptic")
await backend.connect()

graph = await SynapticGraph.from_data("./docs/", backend=backend, preset="rag")
```

Current backend roles:

| Path | Install | What owns the data | When to use |
|------|---------|--------------------|-------------|
| Local app / laptop | `sqlite,korean,vector` | SQLite FTS5 + local usearch HNSW | fastest adoption, demos, small services |
| Team service | `postgresql,embedding,reranker` | PostgreSQL + pgvector + pg_trgm | durable shared graph, backups, SQL ops |
| Graph-heavy embedded | `kuzu,korean,embedding` | Kuzu property graph | local graph traversal / Cypher workflows |
| Scale-out composition | `scale` | Kuzu or another graph store + Qdrant + MinIO | separate graph, vector, and blob responsibilities |

Qdrant and MinIO are helper services, not full graph stores. Use them through
`CompositeBackend`: graph storage keeps nodes/edges, Qdrant handles ANN vector
search, and MinIO/S3-compatible storage offloads large `Node.content`.

```python
from synaptic.backends.composite import CompositeBackend
from synaptic.backends.kuzu import KuzuBackend
from synaptic.backends.minio_store import MinIOBackend
from synaptic.backends.qdrant import QdrantBackend

backend = CompositeBackend(
KuzuBackend("synaptic.kuzu"),
vector=QdrantBackend("http://localhost:6333", collection="synaptic"),
blob=MinIOBackend("localhost:9000", bucket="synaptic"),
)
await backend.connect()

graph = await SynapticGraph.from_data("./docs/", backend=backend, preset="scale")
```

The library gives you the backend contracts and the retrieval layer. For
multi-terabyte production corpora, plan the surrounding operating layer too:
durable ingestion queues, parser/OCR workers, external lexical indexes,
tenant/ACL filters, index-lag monitoring, and backup/restore for each store.

## Quick Start

### Option A: Two lines (easiest)
Expand Down Expand Up @@ -441,13 +504,15 @@ Agent tools → MCP server → LLM agent

## Backends

| Backend | Vector Search | Scale | Use Case |
|---------|--------------|-------|----------|
| `MemoryBackend` | cosine | ~10K | Testing |
| `SqliteGraphBackend` | **usearch HNSW** | ~100K | **Default** |
| `KuzuBackend` | HNSW | ~10M | Graph-heavy |
| `PostgreSQLBackend` | pgvector | ~1M | Production |
| `CompositeBackend` | Qdrant | Unlimited | Scale-out |
| Backend | Install extra | Role | Use case |
|---------|---------------|------|----------|
| `MemoryBackend` | core | in-process graph | tests and examples |
| `SqliteGraphBackend` | `sqlite`, `vector` | local graph + FTS5 + usearch HNSW | default local/embedded deployment |
| `KuzuBackend` | `kuzu` | embedded property graph + Cypher | graph-heavy local workflows |
| `PostgreSQLBackend` | `postgresql` | durable graph + pgvector + pg_trgm | shared production service |
| `QdrantBackend` | `qdrant` | vector-only helper | ANN search behind `CompositeBackend` |
| `MinIOBackend` | `minio` | blob-only helper | large content offload behind `CompositeBackend` |
| `CompositeBackend` | `scale` | router over graph + vector + blob stores | scale-out composition |

---

Expand All @@ -462,6 +527,15 @@ Agent tools → MCP server → LLM agent
| `sqlite` | aiosqlite backend |
| `langchain` | LangChain retriever adapter |
| `postgresql` | asyncpg + pgvector |
| `mysql` | aiomysql database ingest |
| `oracle` | oracledb database ingest |
| `mssql` | aioodbc database ingest |
| `kuzu` | embedded Kuzu graph backend |
| `qdrant` | Qdrant vector helper |
| `minio` | MinIO/S3-compatible blob helper |
| `scale` | Kuzu + Qdrant + MinIO + aiohttp |
| `rag` | spaCy + aiohttp endpoint helpers |
| `all` | common database, vector, MCP, Korean, reranker extras |
| `docs` | xgen-doc2chunk for PDF/DOCX/PPTX/XLSX/HWP loading |

---
Expand Down
Loading