🏠 Home ← Docs Index
Open Beta 📖 Interactive Reader Raw .md llms.txt

Media Pipeline

GoClaw supports channel attachments, Web UI uploads, a persistent media store, workspace uploads for tool access, inline image input to the model, and voice tools.

Runtime Paths

Path Purpose
config.MediaDir() Persistent media-store objects served through /media/<id>.
config.WorkspaceDir()/uploads Web chat uploads that tools can read by workspace-relative path.
config.CanvasDir() Canvas artifacts served through /canvas/<file>.

HTTP Endpoints

Endpoint Purpose
POST /api/media/upload Save a multipart upload to the media store.
GET|HEAD /media/<id> Serve a media-store object (see below for which types are shown inline).
POST /api/upload-workspace Save a multipart upload to workspace/uploads.
GET /uploads/<file> Serve workspace uploads for Web UI previews.
GET /api/canvas List generated canvas artifacts.
GET /canvas/<file> Serve generated canvas artifacts.
GET|POST|DELETE /api/memory/wiki/attachments List, upload or delete a wiki page's attachments. The files are kept in the media store and served at /media/<id>.

/media/<id> serves images (except SVG), audio, video, PDF and plain text inline with their own type. Anything else, SVG, HTML, XML and JavaScript included, is sent as an application/octet-stream download. Every response carries X-Content-Type-Options: nosniff and Content-Security-Policy: default-src 'none'; sandbox. /uploads/ and /canvas/ decide by file extension, and send the CSP header only on the files they force to download: only .png, .jpg, .jpeg, .gif, .webp, .bmp, .ico, .pdf, .mp3, .wav, .ogg, .m4a, .flac, .mp4, .webm, .mov, .txt and .log are shown inline; any other file, or one without an extension, is a download.

/media/, /uploads/, and /canvas/ are treated as user-data paths by the auth middleware. Static Web UI assets remain public, but these file-serving routes require normal gateway authentication.

Web Chat Upload Flow

  1. The browser uploads a file to POST /api/upload-workspace.
  2. The gateway writes it under workspace/uploads.
  3. The response returns local_path, url, filename, mime_type, and size.
  4. The Web UI chat sends the message with the attachment metadata to POST /api/chat and reads the answer as a server-sent event stream. Other clients send the same metadata over the WebSocket at /gateway.
  5. gateway.buildContentParts() (used by both) converts image attachments into inline image markers and references other files by workspace path. The model gets each image as an image: Gemini as an inline part, Claude as an image block, Ollama in the message's images, OpenAI and DeepSeek as a data: URL. A text-only model that refuses the image (DeepSeek's deepseek-v4-pro, a text model behind ai.openai.base_url, an Ollama model without vision) gets the message with a note that one was attached (the request is sent again without it). Retrieval, hooks and the usage record read the message without the image data.
  6. The session store persists UI-safe attachment metadata in session.ExtraMeta.Attachments so previews survive page refreshes.

Attachment Metadata

Persisted session metadata uses:

type AttachmentMeta struct {
    Filename  string `json:"filename,omitempty"`
    MimeType  string `json:"mime_type,omitempty"`
    LocalPath string `json:"local_path,omitempty"`
    MediaID   string `json:"media_id,omitempty"`
    MediaURL  string `json:"media_url,omitempty"`
    URL       string `json:"url,omitempty"`
    Size      int64  `json:"size,omitempty"`
}

This metadata is for UI rendering only. LLM prompt semantics remain unchanged: the agent receives previous history separately from the current user content.

Image Input

For image uploads, buildContentParts() embeds:

__GCIMG__:<mime_type>:<base64_data>

This keeps the public ProcessStream(ctx, content string, history) signature unchanged. The agent loop decodes the marker, and each adapter sends the images:

Provider The image reaches the model as
Gemini genai.Part{InlineData: ...} (gemini_stream.go)
Claude an image block (claude_messages.go); a type Claude does not take is replaced by the note
Ollama the message's images (ollama_stream.go)
OpenAI, DeepSeek an image_url part with a data: URL (openai_stream.go)

A model that refuses an image (a text-only model behind a provider that takes them) is asked again without it, with the same note. The message text that retrieval, hooks and the usage record read has the image lines removed (splitInlineImages).

Non-image files are sent as path references, for example:

[Attached file: report.pdf (application/pdf) — use file_read/pdf_reader with path "uploads/report.pdf"]

Channel Attachments

Messaging channels download attachments into the message in their own way:

Voice Tools

Voice goes through two tools:

Tool Purpose
voice_input Transcribe an audio file using the configured STT provider. Files over 25 MB are refused.
voice_output Generate speech into a file using the configured TTS provider.

voice.stt_provider and voice.tts_provider each take openai (the default, with ai.openai.api_key), gemini (with ai.gemini.api_key) or chimege (with the voice.chimege.* tokens). ElevenLabs is not implemented: the voice.elevenlabs.* keys are read by nothing.

The older vocal_speak / vocal_listen tools (host speaker and microphone) and the desktop_* tools are disabled: their source (voice.go, desktop.go) is commented out, so they are not registered.

Implementation Files

File Responsibility
internal/gateway/media.go Media upload handlers, /media/<id> serving, buildContentParts().
internal/gateway/handler_chat.go Web UI chat (POST /api/chat, SSE) with attachments.
internal/gateway/protocol.go WebSocket attachment payload shape.
internal/gateway/attachments.go Conversion from WebSocket attachments to session metadata.
internal/session/session.go Persistent session history and attachment metadata.
internal/agent/inline_images.go Inline image markers split from the message; each provider's image form.
internal/agent/gemini_stream.go Gemini inline image parts.
web/js/views/chat.js Web chat upload, drag/drop, previews, and history rendering.

Gaps