From 2707033efc9c09e663b92967535dba557b44b15e Mon Sep 17 00:00:00 2001
From: Cameron Pfiffer <cameron@pfiffer.org>
Date: Fri, 2 May 2025 09:21:03 -0700
Subject: [PATCH] Add Modal-based inference server documentation to README

- Introduced a new section detailing the setup and usage of a Modal-based inference server for running LLM inference in the cloud.
- Included step-by-step instructions for installing Modal, deploying the inference server, and using the OpenAI-compatible API.
- Documented available models and options for the client script to enhance user experience and accessibility.
---
 README.md | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/README.md b/README.md
index ee21f89..1938758 100644
--- a/README.md
+++ b/README.md
@@ -46,6 +46,69 @@ Comind creates a network of AI agents that collaborate to process information fl
 Early development phase with a reference implementation available for running your own Comind agent. See the [getting started guide](content/docs/getting-started/_index.md),
 though it is hard to use right now.
 
+## Running Inference with Modal
+
+Comind includes code for a Modal-based inference server for running LLM inference in the cloud without needing powerful local GPUs.
+
+### Setup
+
+1. Install Modal:
+   ```bash
+   pip install modal
+   ```
+
+2. Set up your Modal account:
+   ```bash
+   modal setup
+   ```
+
+3. Deploy the inference server:
+   ```bash
+   modal deploy modal_inference.py
+   ```
+
+This will create an OpenAI-compatible API endpoint running on Modal's infrastructure.
+
+### Using the API
+
+The API is compatible with the OpenAI Python client:
+
+```python
+from openai import OpenAI
+
+# Replace YOUR_WORKSPACE with your Modal workspace name
+client = OpenAI(
+    api_key="comind-api-key",  # Must match the API_KEY in modal_inference.py
+    base_url="https://YOUR_WORKSPACE--comind-vllm-inference-serve-phi4.modal.run/v1"
+)
+
+response = client.chat.completions.create(
+    model="microsoft/Phi-4",
+    messages=[{"role": "user", "content": "Hello, how are you?"}]
+)
+
+print(response.choices[0].message.content)
+```
+
+There's also a convenient client script (`modal_client.py`) for testing:
+
+```bash
+python modal_client.py --workspace YOUR_WORKSPACE --prompt "Tell me a joke"
+```
+
+Additional options:
+- `--model`: Choose model endpoint (phi4 or embeddings)
+- `--api-key`: Specify API key (must match the one in modal_inference.py)
+- `--stream`: Enable streaming responses
+
+### Available Models
+
+The current implementation supports:
+- Phi-4 (default)
+- Embeddings (mixedbread-ai/mxbai-embed-xsmall-v1)
+
+You can easily add more models by editing the `MODELS` dictionary in `modal_inference.py`.
+
 ## Resources
 
 - [Getting started guide](content/docs/getting-started/_index.md)
-- 
2.51.2