# Media Pipeline GoClaw supports channel attachments, Web UI uploads, a persistent media store, workspace uploads for tool access, inline image input to the model, and voice tools. ## Runtime Paths | Path | Purpose | |---|---| | `config.MediaDir()` | Persistent media-store objects served through `/media/`. | | `config.WorkspaceDir()/uploads` | Web chat uploads that tools can read by workspace-relative path. | | `config.CanvasDir()` | Canvas artifacts served through `/canvas/`. | ## HTTP Endpoints | Endpoint | Purpose | |---|---| | `POST /api/media/upload` | Save a multipart upload to the media store. | | `GET\|HEAD /media/` | Serve a media-store object (see below for which types are shown inline). | | `POST /api/upload-workspace` | Save a multipart upload to `workspace/uploads`. | | `GET /uploads/` | Serve workspace uploads for Web UI previews. | | `GET /api/canvas` | List generated canvas artifacts. | | `GET /canvas/` | Serve generated canvas artifacts. | | `GET\|POST\|DELETE /api/memory/wiki/attachments` | List, upload or delete a wiki page's attachments. The files are kept in the media store and served at `/media/`. | `/media/` serves images (except SVG), audio, video, PDF and plain text inline with their own type. Anything else, SVG, HTML, XML and JavaScript included, is sent as an `application/octet-stream` download. Every response carries `X-Content-Type-Options: nosniff` and `Content-Security-Policy: default-src 'none'; sandbox`. `/uploads/` and `/canvas/` decide by file extension, and send the CSP header only on the files they force to download: only `.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`, `.bmp`, `.ico`, `.pdf`, `.mp3`, `.wav`, `.ogg`, `.m4a`, `.flac`, `.mp4`, `.webm`, `.mov`, `.txt` and `.log` are shown inline; any other file, or one without an extension, is a download. `/media/`, `/uploads/`, and `/canvas/` are treated as user-data paths by the auth middleware. Static Web UI assets remain public, but these file-serving routes require normal gateway authentication. ## Web Chat Upload Flow 1. The browser uploads a file to `POST /api/upload-workspace`. 2. The gateway writes it under `workspace/uploads`. 3. The response returns `local_path`, `url`, `filename`, `mime_type`, and `size`. 4. The Web UI chat sends the message with the attachment metadata to `POST /api/chat` and reads the answer as a server-sent event stream. Other clients send the same metadata over the WebSocket at `/gateway`. 5. `gateway.buildContentParts()` (used by both) converts image attachments into inline image markers and references other files by workspace path. The model gets each image as an image: Gemini as an inline part, Claude as an image block, Ollama in the message's `images`, OpenAI and DeepSeek as a `data:` URL. A text-only model that refuses the image (DeepSeek's `deepseek-v4-pro`, a text model behind `ai.openai.base_url`, an Ollama model without vision) gets the message with a note that one was attached (the request is sent again without it). Retrieval, hooks and the usage record read the message without the image data. 6. The session store persists UI-safe attachment metadata in `session.ExtraMeta.Attachments` so previews survive page refreshes. ## Attachment Metadata Persisted session metadata uses: ```go type AttachmentMeta struct { Filename string `json:"filename,omitempty"` MimeType string `json:"mime_type,omitempty"` LocalPath string `json:"local_path,omitempty"` MediaID string `json:"media_id,omitempty"` MediaURL string `json:"media_url,omitempty"` URL string `json:"url,omitempty"` Size int64 `json:"size,omitempty"` } ``` This metadata is for UI rendering only. LLM prompt semantics remain unchanged: the agent receives previous history separately from the current user content. ## Image Input For image uploads, `buildContentParts()` embeds: ```text __GCIMG__:: ``` This keeps the public `ProcessStream(ctx, content string, history)` signature unchanged. The agent loop decodes the marker, and each adapter sends the images: | Provider | The image reaches the model as | |---|---| | Gemini | `genai.Part{InlineData: ...}` (`gemini_stream.go`) | | Claude | an `image` block (`claude_messages.go`); a type Claude does not take is replaced by the note | | Ollama | the message's `images` (`ollama_stream.go`) | | OpenAI, DeepSeek | an `image_url` part with a `data:` URL (`openai_stream.go`) | A model that refuses an image (a text-only model behind a provider that takes them) is asked again without it, with the same note. The message text that retrieval, hooks and the usage record read has the image lines removed (`splitInlineImages`). Non-image files are sent as path references, for example: ```text [Attached file: report.pdf (application/pdf) — use file_read/pdf_reader with path "uploads/report.pdf"] ``` ## Channel Attachments Messaging channels download attachments into the message in their own way: - **Discord**: text-like files (by extension or MIME type) are inlined into the message, up to 256 KB each. Other files are saved to `workspace/uploads/` and named in the message, with a hint for the tool to read them. Files over 10 MB are not downloaded. - **Microsoft Teams**: text-like files up to 256 KB are inlined; other files up to 50 MB are saved to `workspace/uploads/`. - **Facebook Messenger** and **Telegram**: files up to 50 MB are saved to `workspace/uploads/` (Telegram's Bot API itself stops at 20 MB). ## Voice Tools Voice goes through two tools: | Tool | Purpose | |---|---| | `voice_input` | Transcribe an audio file using the configured STT provider. Files over 25 MB are refused. | | `voice_output` | Generate speech into a file using the configured TTS provider. | `voice.stt_provider` and `voice.tts_provider` each take `openai` (the default, with `ai.openai.api_key`), `gemini` (with `ai.gemini.api_key`) or `chimege` (with the `voice.chimege.*` tokens). ElevenLabs is not implemented: the `voice.elevenlabs.*` keys are read by nothing. The older `vocal_speak` / `vocal_listen` tools (host speaker and microphone) and the `desktop_*` tools are disabled: their source (`voice.go`, `desktop.go`) is commented out, so they are not registered. ## Implementation Files | File | Responsibility | |---|---| | `internal/gateway/media.go` | Media upload handlers, `/media/` serving, `buildContentParts()`. | | `internal/gateway/handler_chat.go` | Web UI chat (`POST /api/chat`, SSE) with attachments. | | `internal/gateway/protocol.go` | WebSocket attachment payload shape. | | `internal/gateway/attachments.go` | Conversion from WebSocket attachments to session metadata. | | `internal/session/session.go` | Persistent session history and attachment metadata. | | `internal/agent/inline_images.go` | Inline image markers split from the message; each provider's image form. | | `internal/agent/gemini_stream.go` | Gemini inline image parts. | | `web/js/views/chat.js` | Web chat upload, drag/drop, previews, and history rendering. | ## Gaps - Advanced image processing such as crop, rotation, and format conversion is not implemented. - Cleanup is partial. Canvas artifacts and per-turn `out-*` directories are removed after 48 hours (checked hourly). Media-store objects a session referenced are deleted when that session is deleted, cleared or reset. Files in `workspace/uploads/` are never removed, and wiki attachments stay until they are deleted from their page. - Non-image files are available to tools by path, but the main LLM request does not embed their bytes directly.