From 2add0c2ac98e6f598f66ed7013f2613bb7534c6c Mon Sep 17 00:00:00 2001
From: Cameron Pfiffer <cameron@pfiffer.org>
Date: Fri, 2 May 2025 09:25:05 -0700
Subject: [PATCH] add blog post about modal inference server

---
 content/blog/2025-05-01.md | 94 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 94 insertions(+)
 create mode 100644 content/blog/2025-05-01.md

diff --git a/content/blog/2025-05-01.md b/content/blog/2025-05-01.md
new file mode 100644
index 0000000..56a5514
--- /dev/null
+++ b/content/blog/2025-05-01.md
@@ -0,0 +1,94 @@
+---
+title: "Easy cloud compute for Comind"
+date: 2025-05-01T16:43:23-07:00
+description: "I added convenience tools for using cloud compute to power a Comind instance."
+draft: false
+---
+
+A big issue with Comind is that it requires a set of structured output features that are not supported by commercial providers, so you have to run the model yourself. Most people don't have a giant GPU like I do, so I wanted to provide a simple way to run the model in the cloud.
+
+I chose to use [Modal](https://modal.com), a straightforward and easy service to deploy a [vLLM](https://github.com/vllm-project/vllm) server. vLLM is probably the best inference server available, and has a tight integration with several structured output libraries. An additional benefit of using Modal is that it's easy to deploy a server that can be accessed by any client that supports the [OpenAI API](https://platform.openai.com/docs/api-reference/introduction).
+
+The deployment process is straightforward:
+
+```bash
+# Install and set up Modal
+pip install modal
+modal setup
+
+# Deploy the inference server
+modal deploy modal_inference.py
+```
+
+You'll then get a response like:
+
+```
+✓ Created objects.
+├── 🔨 Created mount /a/directory/comind/modal_inference.py
+├── 🔨 Created web function serve_model => https://YOUR_WORKSPACE_NAME--comind-vllm-inference-serve-model.modal.run
+├── 🔨 Created web function serve_phi4 => https://YOUR_WORKSPACE_NAME--comind-vllm-inference-serve-phi4.modal.run
+└── 🔨 Created web function embeddings => https://YOUR_WORKSPACE_NAME--comind-vllm-inference-embeddings.modal.run
+✓ App deployed in 1.121s! 🎉
+
+View Deployment: https://modal.com/apps/YOUR_WORKSPACE_NAME/main/deployed/comind-vllm-inference
+```
+
+Now, copy these URLs into the `.env` file for your Comind instance:
+
+```bash
+# LLM server info
+COMIND_LLM_SERVER_URL = https://YOUR_WORKSPACE_NAME--comind-vllm-inference-serve-model.modal.run/v1/
+COMIND_LLM_SERVER_API_KEY= "comind-api-key"
+
+# Embedding server info
+COMIND_EMBEDDING_SERVER_URL = https://YOUR_WORKSPACE_NAME--comind-vllm-inference-embeddings.modal.run/v1/
+COMIND_EMBEDDING_SERVER_API_KEY= "comind-api-key"
+```
+
+> [!NOTE]
+> The inference server uses the API key `comind-api-key` by default. You can change this in the `modal_client.py` script:
+> 
+> ```python
+> # Configuration options (can be modified as needed)
+> MINUTES = 60  # seconds
+> VLLM_PORT = 8000
+> API_KEY = "comind-api-key"  # Replace with a secret for production use
+> ```
+
+
+After deployment, you can access your models through the OpenAI client:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="comind-api-key",  # Set in modal_inference.py
+    base_url="https://YOUR_WORKSPACE--comind-vllm-inference-serve-phi4.modal.run/v1"
+)
+
+# vLLM only supports one model at a time, so we need to get the first one
+model_id = client.models.list().data[0].id
+
+response = client.chat.completions.create(
+    model=model_id,
+    messages=[{"role": "user", "content": "What is Comind?"}]
+)
+
+print(response.choices[0].message.content)
+```
+
+(it will not know what Comind is for sure)
+
+I've also created a simple client script that you can use to test the inference server:
+
+```bash
+python modal_client.py --workspace YOUR_WORKSPACE --prompt "Tell me about Comind"
+```
+
+It currently only supports Phi-4. PRs welcome to add more models! I did a weird job so please help.
+
+> [!NOTE]
+> Keep in mind that the server has a warmup time, so it may take a while for it to boot up.
+
+-- Cameron
+
-- 
2.51.2