From f8c04cbce3611875e7272bf36281bde71720c276 Mon Sep 17 00:00:00 2001 From: Cameron Pfiffer Date: Fri, 2 May 2025 23:29:33 -0700 Subject: [PATCH] Update modal_inference.py GPU configuration and revise blog post title and description - Changed GPU configuration for the "phi3-mini" model from "T4:1" to "A10G:1" for improved performance. - Revised blog post title to "Comind development update: Modal inference and conceptualization" and updated the description to reflect recent backend infrastructure and semantic processing advancements. --- content/blog/2025-05-01.md | 350 +++++++++++++++++-------------------- modal_inference.py | 2 +- 2 files changed, 165 insertions(+), 187 deletions(-) diff --git a/content/blog/2025-05-01.md b/content/blog/2025-05-01.md index 230569a..9541d41 100644 --- a/content/blog/2025-05-01.md +++ b/content/blog/2025-05-01.md @@ -1,7 +1,7 @@ --- -title: "Easy cloud compute for Comind" +title: "Comind development update: Modal inference and conceptualization" date: 2025-05-01T16:43:23-07:00 -description: "I added convenience tools for using cloud compute to power a Comind instance." +description: "Recent progress on Comind's backend infrastructure and semantic processing capabilities." draft: false --- @@ -11,279 +11,257 @@ draft: false > Hey -- I'm trying something different here. > > I vibe code Comind quite a bit. I'm going to have Claude write up our changes in this devlog. Claude and I are going to write this together, hence the use of "we" and "us". The tone is also occasionally stranger than I would write, but that's AI shit for you. +> +> So -- here goes. -## Cloud compute for Comind +## Running LLMs in the Cloud for Comind -A big issue with Comind is that it requires a set of structured output features that are not supported by commercial providers, so you have to run the model yourself. Most people don't have a giant GPU like I do, so I wanted to provide a simple way to run the model in the cloud. +One of the biggest challenges with Comind has been accessibility. Comind requires structured output features not supported by commercial providers, forcing users to run models themselves. This creates a barrier for anyone without access to powerful GPUs. -I chose to use [Modal](https://modal.com), a straightforward and easy service to deploy a [vLLM](https://github.com/vllm-project/vllm) server. vLLM is probably the best inference server available, and has a tight integration with several structured output libraries. An additional benefit of using Modal is that it's easy to deploy a server that can be accessed by any client that supports the [OpenAI API](https://platform.openai.com/docs/api-reference/introduction). +Our goal was simple: create a way for Comind users to run inference in the cloud without needing local GPU hardware. -## Updated Modal Deployment Process +**Cameron** IMPORTANT: The Modal implementation described below is still in early development and not yet ready for production use. There are several configuration issues that need to be resolved. Please consider this a preview of future functionality rather than something you should be running right now. -After addressing some compatibility issues with Modal, here's the updated deployment process: +## The Modal + vLLM Solution -### Option 1: Quick Interactive Setup (Recommended) +We chose [Modal](https://modal.com) as our cloud platform for several reasons: -Use our deployment helper script that will guide you through the process: +1. It provides easy access to GPU resources +2. It has a straightforward deployment model +3. It integrates well with [vLLM](https://github.com/vllm-project/vllm) - one of the best inference servers available + +This combination gives us an OpenAI-compatible API that works with the existing Comind infrastructure while providing the structured output capabilities we need. + +## What We've Built + +We've created three core components: + +1. `modal_inference.py` - The main inference server supporting multiple models +2. `modal_client.py` - A test client for validating the deployment +3. `modal_deploy.py` - A helper script that simplifies deployment and management + +The system supports several models with different resource requirements: + +- **Phi-4** (A10G GPU) - Microsoft's flagship 4B parameter model +- **Hermes-3-Llama-3.1-8B** (A10G GPU) - NousResearch's 8B parameter model +- **Hermes-3-Llama-3.2-3B** (T4 GPU) - NousResearch's 3B parameter model +- **Phi-3-mini** (T4 GPU) - Smaller Microsoft model for less powerful GPUs +- **TinyLlama-1.1B** (T4 GPU) - Ultra-lightweight model +- **Qwen3-0.6B** (T4 GPU) - RedHat's efficient 0.6B model +- **mxbai-embed-xsmall** (T4 GPU) - Efficient text embedding model + +**Cameron** However -- there's still some issues here, particularly with choosing GPUs. T4s are cheap but too old, and need some additional TLC to get them to work. Something something `--dtype=half` or something. I'll figure it out. + +## Deployment Options + +### Interactive Deployment (Recommended) + +The simplest approach is using our deployment helper: ```bash -# Deploy and setup interactively (recommended) +# Deploy with interactive setup python modal_deploy.py deploy -# Just create a secret without deploying +# Create a secret without deploying python modal_deploy.py create-secret -# Check the status of your deployment +# Check deployment status python modal_deploy.py status ``` -### Option 2: Manual Setup +This guides you through selecting models to deploy and handles container warming automatically. + +### Manual Deployment -If you prefer to set things up manually: +For those who prefer more control: -1. **Create a Secret in Modal** (optional but recommended): +1. **Create a Secret** (recommended): ```bash - # Create a secret via CLI modal secret create comind-api-key --value "your-api-key-here" - - # Or create it via the Modal web UI: - # Visit https://modal.com/secrets/create?secret_name=comind-api-key ``` -2. **Deploy your application**: +2. **Deploy the application**: ```bash modal deploy modal_inference.py ``` 3. **Keep containers warm**: ```bash - # Run this to reduce cold start times python modal_deploy.py warm ``` -### Troubleshooting +## Key Challenges We Solved -If you encounter deployment errors: +During development, we encountered several significant challenges: -1. **Secret not found**: You can either: - - Create the secret as shown above - - Continue without a secret (a default API key will be used) +### 1. Cold Start Times -2. **Deprecation warnings**: These are informational for future Modal updates and won't affect functionality currently. +Modal, like most serverless platforms, suffers from "cold start" delays when initializing new containers. This was particularly problematic for LLM inference where users expect quick responses. -3. **Authorization errors**: Make sure your client is using the same API key as your server: - ```python - client = OpenAI( - api_key="your-api-key-here", # Must match what you set in Modal - base_url="https://YOUR_WORKSPACE--comind-vllm-inference-serve-phi4.modal.run/v1" - ) - ``` - -Now, copy these URLs into the `.env` file for your Comind instance: - -```bash -# LLM server info -COMIND_LLM_SERVER_URL = https://YOUR_WORKSPACE_NAME--comind-vllm-inference-serve-model.modal.run/v1/ -COMIND_LLM_SERVER_API_KEY= "comind-api-key" - -# Embedding server info -COMIND_EMBEDDING_SERVER_URL = https://YOUR_WORKSPACE_NAME--comind-vllm-inference-embeddings.modal.run/v1/ -COMIND_EMBEDDING_SERVER_API_KEY= "comind-api-key" -``` - -> [!NOTE] -> The inference server uses the API key `comind-api-key` by default. You can change this in the `modal_client.py` script: -> -> ```python -> # Configuration options (can be modified as needed) -> MINUTES = 60 # seconds -> VLLM_PORT = 8000 -> API_KEY = "comind-api-key" # Replace with a secret for production use -> ``` - -After deployment, you can access your models through the OpenAI client: +We implemented several strategies to address this: ```python -from openai import OpenAI - -client = OpenAI( - api_key="comind-api-key", # Must match API_KEY in modal_inference.py - base_url="https://YOUR_WORKSPACE--comind-vllm-inference-serve-phi4.modal.run/v1" -) - -# vLLM only supports one model at a time, so we need to get the first one -model_id = client.models.list().data[0].id - -response = client.chat.completions.create( - model=model_id, - messages=[{"role": "user", "content": "What is Comind?"}] +@app.function( + # Other parameters... + min_containers=1, # Keep at least one container warm + buffer_containers=1, # Provision extra container when active + scaledown_window=10 * MINUTES, # Delay scaling down ) ``` -(it will not know what Comind is for sure) - -I've also created a simple client script that you can use to test the inference server: +The `modal_deploy.py` script also immediately warms containers after deployment: -```bash -python modal_client.py --workspace YOUR_WORKSPACE --prompt "Tell me about Comind" +```python +# Keep containers warm after deployment +serve_phi4.keep_warm(1) ``` -It currently only supports Phi-4. PRs welcome to add more models! I did a weird job so please help. - -> [!NOTE] -> Keep in mind that the server has a warmup time, so it may take a while for it to boot up. +### 2. Authentication & Security -## Solving Cold Start Problems - -One common issue with Modal and similar serverless platforms is cold start time - the delay when a new container needs to be initialized. For a quick fix: - -1. **Keep containers warm** by adding these parameters to your Modal functions: +Initially, we hardcoded API keys, which created inconsistencies between client and server configurations. We switched to Modal's secret management system: ```python -@app.function( - image=vllm_image, - gpu="A10G", - volumes={...}, - min_containers=1, # Keep at least one container warm at all times - buffer_containers=1, # Provision one extra container when active - scaledown_window=10 * MINUTES, # Wait longer before scaling down -) +# Create a secure API key +modal secret create comind-api-key --value "your-secure-key-here" ``` -2. **Update immediately after deployment** with: +Our code now checks for existing secrets and creates them when needed: ```python -if __name__ == "__main__": - if len(sys.argv) > 1 and sys.argv[1] == "deploy": - app.deploy() - # Keep containers warm after deployment - serve_phi4.keep_warm(1) +# Try to use a secret if it exists +try: + api_key_secret = modal.Secret.from_name("comind-api-key") + has_secret = True +except: + print("No 'comind-api-key' secret found. Using default API key.") + has_secret = False ``` -3. **Schedule warm container adjustments** based on time of day: - -```python -@app.function(schedule=modal.Cron("0 * * * *")) -def adjust_warm_containers(): - """Adjust warm containers based on time of day.""" - # During peak hours, keep more warm - serve_phi4.keep_warm(2) - # During off-peak, keep at least one - serve_phi4.keep_warm(1) -``` +### 3. Modal Secret Handling -## Authorization Errors +We discovered that Modal's Secret API works differently than expected. After some trial and error, we implemented the correct pattern: -If you're seeing authorization errors, make sure: +1. **Reference existing secrets**: + ```python + api_key_secret = modal.Secret.from_name("comind-api-key") + ``` -1. The API key in your client matches the server: +2. **Pass secrets to functions**: ```python - # In modal_inference.py - API_KEY = "comind-api-key" - - # In your client - client = OpenAI( - api_key="comind-api-key", # MUST MATCH - base_url="https://..." + @app.function( + # Other parameters... + secrets=[api_key_secret], ) ``` -2. The endpoint URL is correct and includes `/v1` at the end. +3. **Access via environment variables**: + ```python + def get_api_key(): + import os + return os.environ.get("api_key", "comind-api-key") + ``` -3. No typos in either the API key or URL. +This approach fixed the errors users were seeing like `AttributeError: 'Secret' object has no attribute 'get'`. -## Easier Deployment with modal_deploy.py +## Using the API -I've also created a deployment helper script that simplifies the process and solves the cold start issues automatically: +After deployment, you'll have OpenAI-compatible endpoints for each model: -```bash -# Deploy and keep containers warm in one step -python modal_deploy.py deploy +```python +from openai import OpenAI -# Just warm up existing containers anytime -python modal_deploy.py warm +client = OpenAI( + api_key="comind-api-key", # Must match the server configuration + base_url="https://YOUR_WORKSPACE--comind-vllm-inference-serve-phi4.modal.run/v1" +) -# Check the status of your deployments -python modal_deploy.py status +response = client.chat.completions.create( + model="microsoft/Phi-4", + messages=[{"role": "user", "content": "Hello, how are you?"}] +) ``` -This script automatically keeps containers warm after deployment and shows you your endpoint URLs based on your Modal workspace name. It's a much better experience than the manual deployment process. +For testing, you can use the included client script: -## Securing API Keys +```bash +python modal_client.py --workspace YOUR_WORKSPACE --prompt "Tell me about Comind" +``` + +## Configuring Comind -Instead of hardcoding API keys (which is never a good idea), the updated version uses Modal's built-in secret management: +To connect your Comind instance to the Modal servers, update your `.env` file: ```bash -# Create a secure API key (run this once) -modal secret create comind-api-key --value "your-secure-key-here" +# LLM server info +COMIND_LLM_SERVER_URL = https://YOUR_WORKSPACE_NAME--comind-vllm-inference-serve-model.modal.run/v1/ +COMIND_LLM_SERVER_API_KEY= "comind-api-key" + +# Embedding server info +COMIND_EMBEDDING_SERVER_URL = https://YOUR_WORKSPACE_NAME--comind-vllm-inference-embeddings.modal.run/v1/ +COMIND_EMBEDDING_SERVER_API_KEY= "comind-api-key" ``` -The deployment script automatically checks if this secret exists and creates it with a default value if needed. This provides three benefits: +## What conceptualization looks like -1. Your API key isn't stored in source code -2. You can rotate keys without changing code -3. The same key is consistently used across all services +Conceptualization is the process of Comind examining a piece of content and generating a list of concepts and relationships between them. -In your client code, you'll use this same key: +All agents on Comind can be "associated" with a sphere, meaning that they will take on a common system prompt describing what the sphere is supposed to do. -```python -client = OpenAI( - api_key="your-secure-key-here", # Same value from your Modal secret - base_url="https://YOUR_WORKSPACE--comind-vllm-inference-serve-phi4.modal.run/v1" -) -``` +This system prompt is called the "core perspective" of the sphere, and governs the behavior of the entire sphere. -This eliminates the "unauthorized" errors that happen when keys don't match between client and server. +You can define the core perspective however you want -- you could make a sphere that will continuously think about -## Handling Modal Secrets Properly +- Don Cheadle +- the Comind project +- the semantic web +- the void +- antiques (this one was funny) -> [!IMPORTANT] -> There's an important update regarding Modal secrets handling. If you're seeing errors like `AttributeError: 'Secret' object has no attribute 'get'` or `TypeError: _App.function() got an unexpected keyword argument 'env'`, the code has been updated to fix these issues. +**Cameron** I created a new sphere called "comind" that is responsible for understanding the Comind project. It is associated with the following system prompt: -Modal's Secret API works differently than we initially expected. Here's the correct way to use Modal secrets: +``` +Your role is to understand and build the comind network, including: -1. **Create a secret**: - ```python - # Create a secret from a dictionary - api_key_secret = modal.Secret.from_dict({"api_key": "comind-api-key"}) - - # Or reference an existing named secret - api_key_secret = modal.Secret.from_name("comind-api-key") - ``` +- introspect on your core functions +- design improvements or system +modifications +- comment/observe the network at a high +level +``` -2. **Pass the secret to your functions**: - ```python - @app.function( - image=vllm_image, - secrets=[api_key_secret], # Pass the secret as a list - # other parameters... - ) - def serve_phi4(): - # function code... - ``` +Defining the prompt this way gives the sphere a specific "voice" that it uses to understand the content. -3. **Access the secret in your function**: - ```python - def get_api_key(): - """Get the API key from the environment.""" - import os - # Modal automatically injects secret values as environment variables - return os.environ.get("api_key", "comind-api-key") - ``` +For example, the prompt above uses a lot of technical-sounding language like "core functions" and "system modifications". It also refers to the network as a whole, which implies some kind of higher-level goal or purpose. -The secret values are injected as environment variables in your container, so you access them with `os.environ`. This pattern is now implemented in all the Modal functions in our codebase. +Here's an example output: + +``` +conceptualizer - INFO - accessibility - PART_OF - The content discusses generating alt text for images, which is a key aspect of web accessibility. - 0.8 +conceptualizer - INFO - ai integration - SUPPORTS - The use of Google Gemini AI to generate alt text illustrates integration of AI in practical applications. - 0.75 +conceptualizer - INFO - firefox extension - INSTANCE_OF - The specific product discussed is a Firefox extension, representing a tangible instance of technology application. - 0.9 +conceptualizer - INFO - semantic web - PART_OF - Improving accessibility and integrating AI ties into broader semantic web goals by enhancing information usability. - 0.65 +conceptualizer - INFO - open source collaboration - PART_OF - Add-ons are typically open source, hinting at collaboration in development and improvement. - 0.6 +conceptualizer - INFO - network enhancement - CONTRADICTS - While not explicitly mentioned, enhancing tools like alt text generators can indirectly support network understanding by improving content accessibility. - 0.5 +conceptualizer - INFO - real time feedback - INSTANCE_OF - The author's real-time updates about version changes and the availability of the new version exemplify real-time feedback mechanisms. - 0.7 +``` -## Current Status and Known Issues +The "comind" sphere utilizes its resources to understand the Comind project, generating concepts that connect to the core perspective. Each line shows: -As of the latest update, there are still some ongoing issues with the Modal interface that we're actively working to resolve: +1. A concept identified in the content +2. The relationship type (PART_OF, SUPPORTS, INSTANCE_OF, etc.) +3. An explanation of how the concept relates to the content +4. A confidence score between 0 and 1 -1. **API Integration Issues**: Some users are experiencing inconsistent responses when connecting their Comind instance to the Modal-hosted inference server. We're investigating the root cause, which appears to be related to how the API endpoints handle certain request formats. +**Cameron** This is a very simple example, but it shows the potential of the system. More complex spheres could be made by writing a system prompt that describes the sphere's role in more detail. -2. **Container Warmup Reliability**: Despite the warmup mechanisms we've implemented, some users may still experience occasional cold start delays. We're fine-tuning the container management logic to improve reliability. +## Current Status and Next Steps -3. **Authentication Edge Cases**: In certain scenarios, authentication between the client and server may fail even with correctly configured API keys. We're working on more robust error handling to make these cases more diagnosable. +While the system is functional, we're still addressing some issues: -If you encounter any of these issues, please help us improve by reporting specific error messages and the steps to reproduce in the project's issue tracker. We're actively monitoring and addressing these concerns to make the cloud deployment experience as seamless as possible. +1. **API Integration Issues**: Some users experience inconsistent responses between Comind and the Modal-hosted servers +2. **Container Warmup Reliability**: Occasional cold start delays despite our mitigation strategies +3. **Authentication Edge Cases**: Rare authentication failures even with correct configurations --- Cameron +**Claude** We're actively working to improve these areas. +**Cameron** Thanks for reading, buddy. diff --git a/modal_inference.py b/modal_inference.py index 6494602..6a81cb3 100644 --- a/modal_inference.py +++ b/modal_inference.py @@ -65,7 +65,7 @@ MODELS = { "phi3-mini": { "name": "microsoft/Phi-3-mini-4k-instruct", "revision": None, - "gpu": "T4:1", # Smaller model can run on cheaper T4 GPU + "gpu": "A10G:1", # Smaller model can run on cheaper ? }, "tiny-llama": { "name": "TinyLlama/TinyLlama-1.1B-Chat-v1.0", -- 2.51.2