Update docs; add contributing standards
- docs/overview.md: rewrite for current architecture (src/ layout, split JS/CSS modules, credentials/models/functions/ui config categories, correct test fixture targets) - docs/contributing.md: new — documentation philosophy and style guide - AGENTS.md: add rule to follow docs/contributing.md
This commit is contained in:
157
docs/overview.md
157
docs/overview.md
@@ -6,53 +6,76 @@ AI plugins identify spine text; archive plugins supply bibliographic metadata.
|
||||
|
||||
## Stack
|
||||
- **Server**: FastAPI + SQLite (no ORM), Python 3.11+, Poetry (`poetry run serve`)
|
||||
- **Frontend**: Single-file vanilla JS SPA (`static/index.html`)
|
||||
- **Frontend**: Vanilla JS SPA — `static/index.html` + `static/css/` + `static/js/`; no build step
|
||||
- **AI**: OpenAI-compatible API (OpenRouter, OpenAI, etc.) via `openai` library
|
||||
- **Images**: Stored uncompressed in `data/images/`; Pillow used server-side for crops and AI prep
|
||||
|
||||
## Directory Layout
|
||||
```
|
||||
app.py # FastAPI routes only
|
||||
storage.py # DB schema/helpers, settings loading, photo file I/O
|
||||
logic.py # Image processing, boundary helpers, plugin runners, batch pipeline
|
||||
scripts.py # Poetry console entry points: fmt, presubmit
|
||||
plugins/
|
||||
__init__.py # Registry: load_plugins(), get_manifest(), get_plugin()
|
||||
rate_limiter.py # Thread-safe per-domain rate limiter (one global instance)
|
||||
ai_compat/
|
||||
__init__.py # Exports the four AI plugin classes
|
||||
_client.py # Internal: AIClient (openai wrapper, JSON extractor)
|
||||
boundary_detector_shelves.py # BoundaryDetectorShelvesPlugin
|
||||
boundary_detector_books.py # BoundaryDetectorBooksPlugin
|
||||
text_recognizer.py # TextRecognizerPlugin
|
||||
book_identifier.py # BookIdentifierPlugin
|
||||
archives/
|
||||
openlibrary.py # OpenLibrary JSON API
|
||||
rsl.py # RSL AJAX JSON API
|
||||
html_scraper.py # Config-driven HTML scraper (rusneb, alib, shpl)
|
||||
sru_catalog.py # SRU XML catalog (nlr)
|
||||
telegram_bot.py # STUB (pending Telegram credentials)
|
||||
static/index.html # Full SPA (no build step)
|
||||
src/
|
||||
app.py # FastAPI app, exception handlers
|
||||
api.py # All routes (APIRouter)
|
||||
db.py # All SQL; connection() / transaction() context managers
|
||||
files.py # Image file I/O; DATA_DIR, IMAGES_DIR
|
||||
config.py # Config loading and typed AppConfig
|
||||
models.py # Typed dataclasses / mashumaro decoders
|
||||
errors.py # Domain exceptions (NotFoundError, BadRequestError subtypes)
|
||||
logic/
|
||||
__init__.py # dispatch_plugin() orchestrator + re-exports
|
||||
boundaries.py # Boundary math, shelf/spine crop sources, boundary detector runner
|
||||
identification.py # Status computation, text recognizer, book identifier runners
|
||||
archive.py # Archive searcher runner (sync + background)
|
||||
batch.py # Batch pipeline, process_book_sync
|
||||
images.py # crop_save, prep_img_b64, serve_crop
|
||||
plugins/
|
||||
__init__.py # Registry: load_plugins(), get_plugin(), get_manifest()
|
||||
rate_limiter.py # Thread-safe per-domain rate limiter
|
||||
ai_compat/ # AI plugin implementations
|
||||
archives/ # Archive plugin implementations
|
||||
scripts/
|
||||
presubmit.py # Poetry console entry points: fmt, presubmit
|
||||
static/
|
||||
index.html # HTML shell + CSS/JS imports (load order matters)
|
||||
css/ # base, layout, tree, forms, overlays
|
||||
js/ # state → helpers → api → canvas-boundary → tree-render →
|
||||
# detail-render → canvas-crop → editing → photo → events → init
|
||||
config/
|
||||
providers.default.yaml # Provider credentials (placeholder api_key)
|
||||
prompts.default.yaml # Default prompt templates
|
||||
plugins.default.yaml # Default plugin configurations
|
||||
ui.default.yaml # Default UI settings
|
||||
providers.user.yaml # ← create this with your real api_key (gitignored)
|
||||
*.user.yaml # Optional overrides for other categories (gitignored)
|
||||
data/ # Runtime: books.db + images/
|
||||
docs/overview.md # This file
|
||||
credentials.default.yaml # API endpoints and keys (override in credentials.user.yaml)
|
||||
models.default.yaml # Model selection and prompts per AI function
|
||||
functions.default.yaml # Plugin definitions and per-plugin settings
|
||||
ui.default.yaml # UI display settings
|
||||
*.user.yaml # Gitignored overrides — create these with real values
|
||||
data/ # Runtime: books.db + images/ (gitignored)
|
||||
tests/
|
||||
*.py # Python tests (pytest)
|
||||
js/pure-functions.test.js # JS tests (node:test)
|
||||
docs/
|
||||
overview.md # This file
|
||||
contributing.md # Documentation and contribution standards
|
||||
```
|
||||
|
||||
## Layer Architecture
|
||||
Unidirectional: `api` → `logic` → `db` / `files`. No layer may import from a layer above it.
|
||||
|
||||
- **api**: HTTP parsing, entity existence checks via `db.connection()`, calls logic, returns HTTP responses. Owns HTTPException and status codes.
|
||||
- **logic**: Business operations, no HTTP/FastAPI imports. Raises domain exceptions from `errors.py` for expected failures.
|
||||
- **db / files**: SQL and file I/O only. Returns typed dataclasses or None. Never raises domain exceptions.
|
||||
|
||||
## Configuration System
|
||||
Config is loaded from `config/*.default.yaml` merged with `config/*.user.yaml` overrides.
|
||||
Deep merge: dicts are merged recursively; lists in user files replace default lists entirely.
|
||||
Config loaded from `config/*.default.yaml` merged with `config/*.user.yaml`. Deep merge: dicts recursive, lists replaced. Typed via `mashumaro BasicDecoder[AppConfig]`.
|
||||
|
||||
Categories: `providers`, `prompts`, `plugins`, `ui` — each loaded from its own pair of files.
|
||||
Categories:
|
||||
|
||||
Minimal setup — create `config/providers.user.yaml`:
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `credentials` | `base_url` + `api_key` per endpoint; no model or prompt |
|
||||
| `models` | `credentials` ref + `model` string + optional `extra_body` + `prompt` |
|
||||
| `functions` | Plugin definitions; dict key = plugin_id (unique across all categories) |
|
||||
| `ui` | Frontend display settings |
|
||||
|
||||
Minimal setup — create `config/credentials.user.yaml`:
|
||||
```yaml
|
||||
providers:
|
||||
credentials:
|
||||
openrouter:
|
||||
api_key: "sk-or-your-actual-key"
|
||||
```
|
||||
@@ -62,28 +85,26 @@ providers:
|
||||
### Categories
|
||||
| Category | Input | Output | DB field |
|
||||
|----------|-------|--------|----------|
|
||||
| `boundary_detector` (`target=shelves`) | cabinet image | `{boundaries:[…], confidence:N}` | `cabinets.ai_shelf_boundaries` |
|
||||
| `boundary_detector` (`target=books`) | shelf image | `{boundaries:[…]}` | `shelves.ai_book_boundaries` |
|
||||
| `text_recognizer` | spine image | `{raw_text, title, author, …}` | `books.raw_text` + `candidates` |
|
||||
| `book_identifier` | raw_text | `{title, author, …, confidence}` | `books.ai_*` + `candidates` |
|
||||
| `archive_searcher` | query string | `[{source, title, author, …}, …]` | `books.candidates` |
|
||||
| `boundary_detectors` (`target=shelves`) | cabinet image | `{boundaries:[…], confidence:N}` | `cabinets.ai_shelf_boundaries` |
|
||||
| `boundary_detectors` (`target=books`) | shelf image | `{boundaries:[…]}` | `shelves.ai_book_boundaries` |
|
||||
| `text_recognizers` | spine image | `{raw_text, title, author, …}` | `books.raw_text` + `candidates` |
|
||||
| `book_identifiers` | raw_text | `{title, author, …, confidence}` | `books.ai_*` + `candidates` |
|
||||
| `archive_searchers` | query string | `[{source, title, author, …}, …]` | `books.candidates` |
|
||||
|
||||
### Universal plugin endpoint
|
||||
```
|
||||
POST /api/{entity_type}/{entity_id}/plugin/{plugin_id}
|
||||
```
|
||||
Routes to the correct runner function in `logic.py` based on plugin category.
|
||||
Routes to the correct runner via `dispatch_plugin()` in `logic/__init__.py`.
|
||||
|
||||
### AI Plugin Configuration
|
||||
- **Providers** (`config/providers.*.yaml`): connection credentials only — `base_url`, `api_key`.
|
||||
- **Per-plugin** (`config/plugins.*.yaml`): `provider`, `model`, optionally `max_image_px` (default 1600), `confidence_threshold` (default 0.8).
|
||||
- `OUTPUT_FORMAT` is a **hardcoded class constant** in each plugin class — not user-configurable.
|
||||
It is substituted into the prompt template as `${OUTPUT_FORMAT}` by `AIClient.call()`.
|
||||
- `credentials` file: connection only — `base_url`, `api_key`.
|
||||
- `models` file: `credentials` ref, `model` string, `prompt` text, optional `extra_body`.
|
||||
- `functions` file: per-plugin settings — `model`, `max_image_px` (default 1600), `confidence_threshold` (default 0.8), `auto_queue`, `rate_limit_seconds`, `timeout`.
|
||||
- `OUTPUT_FORMAT` is a hardcoded class constant in each plugin — not user-configurable; injected into the prompt as `${OUTPUT_FORMAT}` by `AIClient`.
|
||||
|
||||
### Archive Plugin Interface
|
||||
All archive plugins implement `search(query: str) -> list[CandidateRecord]`.
|
||||
`CandidateRecord`: TypedDict with `{source, title, author, year, isbn, publisher}`.
|
||||
Uses shared `RATE_LIMITER` singleton for per-domain throttling.
|
||||
### Archive plugins
|
||||
All implement `search(query: str) -> list[CandidateRecord]`. Use shared `RATE_LIMITER` singleton for per-domain throttling.
|
||||
|
||||
### Auto-queue
|
||||
- After `text_recognizer` completes → fires all `archive_searchers` with `auto_queue: true` in background thread pool.
|
||||
@@ -93,7 +114,7 @@ Uses shared `RATE_LIMITER` singleton for per-domain throttling.
|
||||
| Table | Notable columns |
|
||||
|-------|-----------------|
|
||||
| `cabinets` | `shelf_boundaries` (JSON `[…]`), `ai_shelf_boundaries` (JSON `{pluginId:[…]}`) |
|
||||
| `shelves` | `book_boundaries`, `ai_book_boundaries` (same format) |
|
||||
| `shelves` | `book_boundaries`, `ai_book_boundaries` (same format), `photo_filename` (optional override) |
|
||||
| `books` | `raw_text`, `ai_title/author/year/isbn/publisher`, `candidates` (JSON `[{source,…}]`), `identification_status` |
|
||||
|
||||
`identification_status`: `unidentified` → `ai_identified` → `user_approved`.
|
||||
@@ -102,30 +123,36 @@ Uses shared `RATE_LIMITER` singleton for per-domain throttling.
|
||||
N interior boundaries → N+1 segments. `full = [0] + boundaries + [1]`. Segment K spans `full[K]..full[K+1]`.
|
||||
- User boundaries: `shelf_boundaries` / `book_boundaries` (editable via canvas drag)
|
||||
- AI suggestions: `ai_shelf_boundaries` / `ai_book_boundaries` (JSON object `{pluginId: [fractions]}`)
|
||||
- Shelf K image = cabinet photo cropped to `(0, y_start, 1, y_end)` unless override photo exists
|
||||
- Shelf K image = cabinet photo cropped to `(0, y_start, 1, y_end)` unless shelf has override photo
|
||||
- Book K spine = shelf image cropped to `(x_start, *, x_end, *)` with composed crop if cabinet-based
|
||||
|
||||
## Frontend JS
|
||||
No ES modules, no bundler. All files use global scope; load order in `index.html` is the dependency order. State lives in `state.js` (`S`, `_plugins`, `_bnd`, `_photoQueue`, etc.). Events delegated via `#app` in `events.js`.
|
||||
|
||||
## Tooling
|
||||
```
|
||||
poetry run serve # start uvicorn on :8000
|
||||
poetry run fmt # black (in-place)
|
||||
poetry run presubmit # black --check + flake8 + pyright + pytest ← run before finishing any task
|
||||
poetry run presubmit # black --check + flake8 + pyright + pytest + JS tests
|
||||
npm install # install ESLint + Prettier (requires network; enables JS lint/fmt in presubmit)
|
||||
npm run lint # ESLint on static/js/
|
||||
npm run fmt # Prettier on static/js/
|
||||
```
|
||||
Line length: 120. Type checking: pyright strict mode. Pytest fixtures with `yield` use `Iterator[T]` return type.
|
||||
Tests in `tests/`; use `monkeypatch` on `storage.DB_PATH` / `storage.DATA_DIR` for temp-DB fixtures.
|
||||
Line length: 120. Pyright strict mode. Pytest fixtures with `yield` return `Iterator[T]`.
|
||||
Test fixtures: monkeypatch `db.DB_PATH` / `files.DATA_DIR` / `files.IMAGES_DIR`.
|
||||
|
||||
## Key API Endpoints
|
||||
```
|
||||
GET /api/config # UI config + plugin manifest
|
||||
GET /api/tree # full nested tree
|
||||
GET /api/config # UI config + plugin manifest
|
||||
GET /api/tree # full nested tree
|
||||
POST /api/{entity_type}/{entity_id}/plugin/{plugin_id} # universal plugin runner
|
||||
PATCH /api/cabinets/{id}/boundaries # update shelf boundary list
|
||||
PATCH /api/shelves/{id}/boundaries # update book boundary list
|
||||
GET /api/shelves/{id}/image # shelf image (override or cabinet crop)
|
||||
GET /api/books/{id}/spine # book spine crop
|
||||
POST /api/books/{id}/process # run full auto-queue pipeline (single book)
|
||||
POST /api/batch # start batch processing
|
||||
GET /api/batch/status
|
||||
POST /api/books/{id}/dismiss-field # dismiss a candidate suggestion
|
||||
PATCH /api/{kind}/reorder # SortableJS drag reorder
|
||||
PATCH /api/cabinets/{id}/boundaries # update shelf boundary list
|
||||
PATCH /api/shelves/{id}/boundaries # update book boundary list
|
||||
GET /api/shelves/{id}/image # shelf image (override or cabinet crop)
|
||||
GET /api/books/{id}/spine # book spine crop
|
||||
POST /api/books/{id}/process # full auto-queue pipeline (single book)
|
||||
POST /api/batch / GET /api/batch/status # batch processing
|
||||
POST /api/books/{id}/dismiss-field # dismiss a candidate suggestion
|
||||
PATCH /api/{kind}/reorder # drag-to-reorder
|
||||
POST /api/cabinets/{id}/crop / POST /api/shelves/{id}/crop # permanent crop
|
||||
```
|
||||
|
||||
Reference in New Issue
Block a user