# Text Handling Service Agent Guide Base URL: `https://text-handling.aisloppy.com` ## Purpose Public API for extracting text, parsing transcript structure, chunking content, and generating summaries. ## Shared Shell - The semantic header is app-owned at `[data-platform-header]`, enhanced by `https://fairystack.com/header-toolkit.js`, and declared in `fairystack.json`. - Runtime navigation and host auth policy come from Platform 2040 `GET /api/app-shell/config`; this dependency is intentionally visible in app source and is never injected by nginx. - The app's own `/api/auth-mode` remains authoritative for backend enforcement. Public deployment uses AuthReturn; the network-isolated Finance deployment uses trusted-network mode. ## Features - Extracts readable text from PDFs, URLs, raw HTML, and transcripts. - Uses BrightWrapper-assisted DOM outline selection for URL/HTML primary-content extraction and returns extraction metadata. - Parses transcript speakers and timestamps into structured segments. - Reads PDF outline data when native bookmarks are present. - Splits large inputs into chunks for downstream processing. - Produces structured summaries for chunks, full documents, and URLs. - Supports queued batch structured-generation jobs with poll and cancel endpoints. - Supports queued source-mapped evidence synthesis: structured extraction across every chunk followed by a structured merge. ## Sub-services and Reference Implementations - **Narrative/book summarization:** overview, brief, and detailed modes with excerpts; reference UI: `https://genlibrary.com/book/1984`. - **Source importance heatmap:** full-text reconstruction from scored, source-mapped chunks. - **Structural extraction:** primary readable content and document structure from HTML, URL, PDF, and transcript inputs. - **Source-mapped chunking:** stable chunk IDs, exact character ranges, structural boundaries, and optional overlap. ## Endpoints - `GET /api/health` - Health status. - `GET /api/auth-mode` - Returns deployment policy: `authreturn` publicly or `trusted-network` when authentication is disabled on a network-isolated instance. - `GET /agent-guide.md` - This guide. - `POST /api/auth/signup` - Body: `{ "email": "...", "password": "..." }` - Returns: proxied signup response for this service account namespace. - `POST /api/auth/api-key` - Body: `{ "email": "...", "password": "...", "force?": true }` - Returns: proxied API-key creation response for this service account namespace. - `GET /api/v1/summarize/config` - Returns: `{ "chunk_system_prompt", "chunk_prompt_template", "chunk_result_schema", "default_excerpt_count" }` - `POST /api/v1/extract/pdf-upload` - Content-Type: `multipart/form-data` - Fields: `file` (PDF binary, required), `title` (optional), `include_images` (optional, currently false only) - Returns: `{ "title", "text", "char_count" }` - `POST /api/v1/extract/pdf-structure-upload` - Content-Type: `multipart/form-data` - Fields: `file` (PDF binary, required), `title` (optional) - Returns: `{ "title", "total_pages", "document_metadata", "outline_source", "has_native_outline", "parts", "top_level_parts", "warnings" }` - `document_metadata` is the PDF's own information dictionary: `doc_title`, `author`, `subject`, `producer`, `creator`, `created`, `modified`, `encrypted`. Absent fields are `null`, never `""`. - `POST /api/v1/extract/pdf` (compatibility mode) - Body: `{ "pdf_base64": "", "title": "Optional", "include_images": false }` - or `{ "pdf_path": "", "include_images": false }` - Returns: `{ "title", "text", "char_count", "images?" }` - `POST /api/v1/extract/pdf-structure` - Body: `{ "pdf_base64": "", "title": "Optional" }` - or `{ "pdf_path": "" }` - Returns: `{ "title", "total_pages", "document_metadata", "outline_source", "has_native_outline", "parts", "top_level_parts", "warnings" }` - `document_metadata` is the PDF's own information dictionary: `doc_title`, `author`, `subject`, `producer`, `creator`, `created`, `modified`, `encrypted`. Absent fields are `null`, never `""`. - `POST /api/v1/extract/url` - Body: `{ "url": "https://..." }` - Auth: requires `X-API-Key` - Returns: `{ "title", "text", "char_count", "source_type": "article", "extraction_metadata": {...} }` - `POST /api/v1/detect/url` - Body: `{ "url": "https://..." }` - Auth: requires `X-API-Key` - Returns: `{ "source_type": "pdf|epub|html|web", "source_url", "resolved_url" }` using the service-owned verified fetch path. - `POST /api/v1/fetch/url` - Body: `{ "url": "https://..." }` - Auth: requires `X-API-Key` - Returns: `{ "html", "source_url", "resolved_url" }` using the service-owned verified fetch path. - `POST /api/v1/fetch/file` - Body: `{ "url": "https://..." }` - Auth: requires `X-API-Key` - Streams the source bytes with the upstream content type plus `X-Source-Resolved-URL` and `X-Source-Fetch-Path` response headers. - The direct source stream is capped at 250 MiB and every upstream call has a 120-second socket timeout. - `POST /api/v1/extract/url/robust` - Body: `{ "url": "https://..." }` - Auth: requires `X-API-Key` - Starts the canonical service-owned URL extraction task. It handles X/Twitter posts, PDFs, plain text, and HTML, then uses a reader API and Browser Driver when ordinary page extraction fails. - Returns `202` with `{ "task_id", "status", "poll_url", "source" }` - `GET /api/task//extract_url` - Auth: requires `X-API-Key` - Polls a robust URL extraction task and returns `{ "status": "completed", "result": { "title", "text", "char_count", "source_type", "extraction_metadata" } }` when done. - `POST /api/v1/extract/html` - Body: `{ "html": "...", "title?": "...", "source_url?": "https://..." }` - Returns: `{ "title", "text", "char_count", "source_type": "article", "extraction_metadata": {...} }` - `POST /api/v1/extract/url/agent` - Body: `{ "url": "https://...", "reason?": "why the normal extractor needs help" }` - Auth: requires `X-API-Key` - Returns `202` with `{ "task_id", "status", "poll_url", "source" }` - `GET /api/task//agent_extract` - Auth: requires `X-API-Key` - Polls a rendered-browser extraction task and returns `{ "status": "completed", "result": { "title", "text", "char_count", "source_type", "extraction_metadata" } }` when done. - `POST /api/v1/extract/transcript` - Body: `{ "text": "...", "title?": "...", "source_url?": "...", "speaker_names?": [] }` - Returns: `{ "title", "source_type", "text", "char_count", "speakers", "segment_count", "segments" }` - `POST /api/v1/chunk` - Body: `{ "text": "...", "chunk_size": 4000, "overlap": 500, "include_metadata?": true }` - Default return: `{ "chunks": ["..."], "count": 12 }`. With metadata, each chunk includes `text`, stable `chunk_id`, `source_start`, `source_end`, structural unit bounds, and boundary types. - `POST /api/v1/summarize/chunk` - Body: `{ "chunk_text", "chunk_number", "total_chunks", "prompt_template?", "system_prompt?", "project_name", "priorities?": {"speed","quality","cost"} }` - Auth: requires `X-API-Key` - Returns: `{ "result": {"title","summary","summary_original_language","most_interesting","excerpt","excerpt_translation"}, "model": "..." }`; `summary_original_language` contains one source-language translation per English summary bullet for non-English source text, otherwise `[]`. `excerpt_translation` contains a faithful English rendering of a non-English excerpt, otherwise `""`. - `POST /api/v1/summarize/compose` - Body: `{ "chunk_summaries": [{"title","summary","most_interesting","excerpt"}], "excerpt_count": 5 }` - Returns: `{ "summary", "summary_word_count", "chunk_summaries", "excerpt_sections" }` - `POST /api/v1/summarize/condense` - Body: `{ "title?", "chunk_summaries": [{"title","summary","excerpt?"}], "profile": "overview|standard", "project_name": "...", "priorities?": {"speed","quality","cost"} }` - Auth: requires `X-API-Key` - Returns: `{ "one_sentence", "quick_take", "why_it_matters", "key_themes", "notable_excerpts", "structure", "spoiler_level", "profile", "model?" }`; each notable excerpt is `{ "quote", "translation", "section_title" }`, with a faithful English `translation` for non-English quotes and an empty string for English quotes. - `POST /api/structured_llm_call/batch` - Body: queued structured-generation batch payload with `items`, `text_format_schema`, `project_name`, and optional `system_prompt`, `max_concurrent`, `speed`, `quality` - Auth: requires `X-API-Key` - Returns: `{ "task_id", "status", "poll_url", "item_count" }` - `GET /api/task//text_batch` - Auth: requires `X-API-Key` - Returns: queued batch status with `status`, `sub_tasks`, `results`, `progress`, and `error` - `POST /api/task//text_batch/cancel` - Auth: requires `X-API-Key` - Cancels a queued batch task - `POST /api/v1/evidence-synthesis` - Auth: requires `X-API-Key` - Starts a service-owned map/reduce job and returns `202` with `{ "task_id", "status", "stage", "poll_url", "deadline_at" }` before model work begins. - Required body: `{ "text", "question", "evidence_schema", "synthesis_schema", "project_name" }`. - Optional body: `{ "chunk_size", "overlap", "max_concurrent", "deadline_seconds", "idempotency_key", "map_prompt_template", "merge_prompt_template", "map_system_prompt", "merge_system_prompt", "priorities", "model" }`. - Map templates may use `{question}`, `{chunk_number}`, `{total_chunks}`, `{chunk_id}`, `{source_start}`, `{source_end}`, and `{chunk_text}`. Merge templates may use `{question}` and `{evidence_json}`. - Completed results contain `{ "synthesis", "evidence", "model", "fallback_from" }`; every evidence record includes its stable `chunk_id`, `source_start`, and `source_end`. - `GET /api/task//evidence-synthesis` - Auth: requires `X-API-Key` - Returns persisted `queued`, `running`, `completed`, `failed`, `cancelled`, or `timed_out` state with stage, timestamps, heartbeat, deadline, progress, concrete error, and completed result. - `POST /api/task//evidence-synthesis/cancel` - Auth: requires `X-API-Key` - Cancels owned mapping work and persists a terminal `cancelled` state. - `POST /api/v1/summarize/text` - Body: `{ "text", "title?", "chunk_size": 10000, "overlap": 0, "max_workers": 4, "project_name": "..." }` - Auth: requires `X-API-Key` - Returns: excerpt-aware summary markdown, structured chunk summaries with stable `chunk_id`, `source_start`, and `source_end`, excerpt sections, word stats, and model list. - `POST /api/v1/summarize/document` - Body: `{ "text", "source_type": "article|book|transcript|notes", "title?", "chunk_size": 10000, "overlap": 0, "max_workers": 4, "project_name": "...", "profile?": "...", "speaker_names?": [] }` - Auth: requires `X-API-Key` - Returns: unified document summary with `chunk_summaries`, `document_summary`, and transcript metadata when `source_type="transcript"`. - `POST /api/v1/summarize/url` - Body: `{ "url", "chunk_size": 10000, "max_workers": 4, "project_name": "..." }` - Auth: requires `X-API-Key` - Returns: excerpt-aware summary markdown, structured chunk summaries, excerpt sections, word stats, and model list. ### Escalating fetch (all source-fetching endpoints) Every endpoint above fetches through one shared path, which sends the header set a real Chrome tab sends (`Sec-Fetch-*`, client hints, same-origin `Referer`) — not a bare User-Agent. When an origin *refuses* the request the fetch escalates; an ordinary response never does. A refusal is an empty `202`, a 401/403/429/451, a `Content-Length: 0` body, or a challenge interstitial (AWS WAF, Cloudflare, Imperva, DataDome, PerimeterX) recognised in the first bytes. | `fetch_path` / `X-Source-Fetch-Path` | Rung | |---|---| | `direct` | Ordinary request. Realistic headers alone clear most 403s. | | `residential` | Retried through the Oxylabs residential pool, sticky per host. Clears blocks keyed on datacentre IP reputation. | | `browser` | Browser Driver loads the origin so the JavaScript challenge executes, then re-fetches the file in-page with the challenge cookies. Capped at 25 MiB (the body returns as base64). | The whole ladder shares a 50-second budget so a synchronous caller stays inside nginx's 60s read timeout; a rung that cannot fit is skipped and named in the error. If every rung is refused, the origin's own status and message are returned — the failure is never disguised as an empty or partial document. ## Client Guidance - Pass `project_name` so requests can be attributed correctly. - Use the task endpoints for queued batch work and check the returned `status` field to determine whether the job is pending, running, completed, failed, or cancelled. - Set client-side timeouts that match your calling environment and handle non-2xx responses explicitly. ## Authentication - Costly endpoints require an app-scoped API key in the `X-API-Key` header. - If you do not already have an account in this app namespace, create one first: ```bash curl -X POST https://text-handling.aisloppy.com/api/auth/signup \ -H "Content-Type: application/json" \ -d '{"email":"YOUR_SERVICE_ACCOUNT_EMAIL","password":"YOUR_SERVICE_ACCOUNT_PASSWORD"}' ``` - Then create an API key for that account. If the account already exists, skip signup and go straight to this step: ```bash curl -X POST https://text-handling.aisloppy.com/api/auth/api-key \ -H "Content-Type: application/json" \ -d '{"email":"YOUR_SERVICE_ACCOUNT_EMAIL","password":"YOUR_SERVICE_ACCOUNT_PASSWORD"}' ``` - Send the returned key on protected requests as `X-API-Key: ar_...`. - If signup returns `400` with `"This email is already registered"`, reuse that account and call `/api/auth/api-key`. - If API-key creation returns `409` with an existing key prefix, do not force-rotate casually. Use the existing stored key when available. Add `"force": true` only when you intentionally want to revoke and replace the active key. Example protected call: ```bash curl -X POST https://text-handling.aisloppy.com/api/v1/extract/url \ -H "Content-Type: application/json" \ -H "X-API-Key: ar_..." \ -d '{"url":"https://example.com"}' ``` ## Required Client Env - `TEXT_HANDLING_SERVICE_URL=https://text-handling.aisloppy.com` ## Notes - Use `extract/pdf-upload` for cross-app calls to avoid base64 overhead and path-permission issues. - Use `extract/pdf-structure-upload` for cross-app PDF structure extraction. - `include_images` is currently supported only for server-local `pdf_path` mode in `extract/pdf`. - Record extraction defects in `tests/fixtures/pdf_extraction_problem_strings.json` before changing shared normalization behavior; each case preserves the observed output, expected output, source, and page for regression review. ### Source failures in robust extraction Reader responses that wrap an upstream HTTP error and browser navigation HTTP errors retain the source status (including 403 and 429). Blocking pages are rejected before article selection. Failed task logs preserve earlier fallback errors; HTTP 422 means the parser found too little content, not proof that the source lacks readable text. ### General proxy recovery The robust extraction task tries ScrapingBee after primary extraction fails or returns low confidence for any HTTP(S) source URL, before the reader/browser fallbacks. Successful primary extraction does not invoke the paid proxy. It uses Auto-Mode capped at 25 credits, a 10-second connection timeout and 120-second read timeout. Recovered HTML passes through the existing article selector; it is not accepted merely because HTTP succeeded. Successful extraction metadata includes `fetch_path: scrapingbee` and, when reported by the proxy, `proxy_cost_credits`. Proxy errors preserve diagnostics without exposing API keys. Runtime setup uses `deploy/install-scrapingbee.sh` to provision the existing SOPS-managed `scraping_bee.json` into the app's own read-only secret mount. See https://www.scrapingbee.com/documentation/ for the upstream API contract.