feat: add Cerebras and Fireworks AI LLM providers (#2113)
* fix: improve local model provider robustness and UX - Extract shared Docker URL rewriting and env conversion into BaseProvider to eliminate 4x duplicated code across Ollama and LMStudio - Add error handling and 5s timeouts to all model-listing fetches so one unreachable provider doesn't block the entire model list - Fix Ollama using createOllama() instead of mutating provider internals - Fix LLMManager singleton ignoring env updates on subsequent requests - Narrow cache key to only include provider-relevant env vars instead of the entire server environment - Fix 'as any' casts in LMStudio and OpenAILike by using shared convertEnvToRecord helper - Replace console.log/error with structured logger in OpenAILike - Fix typo: filteredStaticModesl -> filteredStaticModels in manager - Add connection status indicator (green/red dot) for local providers in the ModelSelector dropdown - Show helpful "is X running?" message when local provider has no models Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add Cerebras LLM provider - Add Cerebras provider with 8 models (Llama, GPT OSS, Qwen, ZAI GLM) - Integrate @ai-sdk/cerebras@0.2.16 for compatibility - Add CEREBRAS_API_KEY to environment configuration - Register provider in LLMManager registry Models included: - llama3.1-8b, llama-3.3-70b - gpt-oss-120b (reasoning) - qwen-3-32b, qwen-3-235b variants - zai-glm-4.6, zai-glm-4.7 (reasoning) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add Fireworks AI LLM provider - Add Fireworks provider with 6 popular models - Integrate @ai-sdk/fireworks@0.2.16 for compatibility - Add FIREWORKS_API_KEY to environment configuration - Register provider in LLMManager registry Models included: - Llama 3.1 variants (405B, 70B, 8B Instruct) - DeepSeek R1 (reasoning model) - Qwen 2.5 72B Instruct - FireFunction V2 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add coding-specific models to existing providers Enhanced providers with state-of-the-art coding models: **DeepSeek Provider:** + DeepSeek V3.2 (integrates thinking + tool-use) + DeepSeek V3.2-Speciale (high-compute variant, beats GPT-5) **Fireworks Provider:** + Qwen3-Coder 480B (262K context, best for coding) + Qwen3-Coder 30B (fast coding specialist) **Cerebras Provider:** + Qwen3-Coder 480B (2000 tokens/sec!) - Removed deprecated models (qwen-3-32b, llama-3.3-70b) Total new models: 4 Total coding models across all providers: 12+ Performance highlights: - Qwen3-Coder: State-of-the-art coding performance - DeepSeek V3.2: Integrates thinking directly into tool-use - ZAI GLM 4.6: 73.8% SWE-bench score - Ultra-fast inference: 2000 tok/s on Cerebras Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add dynamic model discovery to providers Implemented getDynamicModels() for automatic model discovery: **DeepSeek Provider:** - Fetches models from https://api.deepseek.com/models - Automatically discovers new models as DeepSeek adds them - Filters out static models to avoid duplicates **Cerebras Provider:** - Fetches models from https://api.cerebras.ai/v1/models - Auto-discovers new Cerebras models - Keeps UI up-to-date with latest offerings **Fireworks Provider:** - Fetches from https://api.fireworks.ai/v1/accounts/fireworks/models - Includes context_length from API response - Discovers new Qwen-Coder and other models automatically **Moonshot Provider:** - Fetches from https://api.moonshot.ai/v1/models - OpenAI-compatible endpoint - Auto-discovers new Kimi models Benefits: - ✅ No manual updates needed when providers add new models - ✅ Users always have access to latest models - ✅ Graceful fallback to static models if API fails - ✅ 5-second timeout prevents hanging - ✅ Caching system built into BaseProvider Technical details: - Uses BaseProvider's built-in caching system - Cache invalidates when API keys change - Failed API calls fallback to static models - All endpoints have 5-second timeout protection Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add Z.AI provider with GLM models and JWT authentication Merged changes from PR #2069 to add Z.AI provider: - Added GLM-4.6 (200K), GLM-4.5 (128K), and GLM-4.5 Flash models - Implemented secure JWT token generation with HMAC-SHA256 signing - Added dynamic model discovery from Z.AI API - Included proper error handling and token validation - GLM-4.6 achieves 73.8% on SWE-bench coding benchmarks Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Stijnus
@@ -3,6 +3,7 @@ import { useEffect, useState, useRef, useMemo, useCallback } from 'react';
|
||||
import type { KeyboardEvent } from 'react';
|
||||
import type { ModelInfo } from '~/lib/modules/llm/types';
|
||||
import { classNames } from '~/utils/classNames';
|
||||
import { LOCAL_PROVIDERS } from '~/lib/stores/settings';
|
||||
|
||||
// Fuzzy search utilities
|
||||
const levenshteinDistance = (str1: string, str2: string): number => {
|
||||
@@ -130,6 +131,32 @@ export const ModelSelector = ({
|
||||
const providerDropdownRef = useRef<HTMLDivElement>(null);
|
||||
const [showFreeModelsOnly, setShowFreeModelsOnly] = useState(false);
|
||||
|
||||
type ConnectionStatus = 'unknown' | 'connected' | 'disconnected';
|
||||
|
||||
const [localProviderStatus, setLocalProviderStatus] = useState<Record<string, ConnectionStatus>>({});
|
||||
|
||||
// Check connectivity of local providers when provider list changes
|
||||
useEffect(() => {
|
||||
const checkLocalProviders = async () => {
|
||||
const statuses: Record<string, 'connected' | 'disconnected'> = {};
|
||||
|
||||
for (const p of providerList) {
|
||||
if (!LOCAL_PROVIDERS.includes(p.name)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// If the provider has models loaded, it's connected
|
||||
const hasModels = modelList.some((m) => m.provider === p.name);
|
||||
|
||||
statuses[p.name] = hasModels ? 'connected' : 'disconnected';
|
||||
}
|
||||
|
||||
setLocalProviderStatus(statuses);
|
||||
};
|
||||
|
||||
checkLocalProviders();
|
||||
}, [providerList, modelList]);
|
||||
|
||||
// Debounce search queries
|
||||
useEffect(() => {
|
||||
const timer = setTimeout(() => {
|
||||
@@ -440,7 +467,28 @@ export const ModelSelector = ({
|
||||
tabIndex={0}
|
||||
>
|
||||
<div className="flex items-center justify-between">
|
||||
<div className="truncate">{provider?.name || 'Select provider'}</div>
|
||||
<div className="flex items-center gap-2 truncate">
|
||||
{provider?.name && LOCAL_PROVIDERS.includes(provider.name) && (
|
||||
<span
|
||||
className={classNames(
|
||||
'inline-block w-2 h-2 rounded-full flex-shrink-0',
|
||||
localProviderStatus[provider.name] === 'connected'
|
||||
? 'bg-green-500'
|
||||
: localProviderStatus[provider.name] === 'disconnected'
|
||||
? 'bg-red-400'
|
||||
: 'bg-bolt-elements-textTertiary',
|
||||
)}
|
||||
title={
|
||||
localProviderStatus[provider.name] === 'connected'
|
||||
? `${provider.name} is running`
|
||||
: localProviderStatus[provider.name] === 'disconnected'
|
||||
? `${provider.name} is not reachable`
|
||||
: 'Checking...'
|
||||
}
|
||||
/>
|
||||
)}
|
||||
{provider?.name || 'Select provider'}
|
||||
</div>
|
||||
<div
|
||||
className={classNames(
|
||||
'i-ph:caret-down w-4 h-4 text-bolt-elements-textSecondary opacity-75',
|
||||
@@ -559,11 +607,25 @@ export const ModelSelector = ({
|
||||
}}
|
||||
tabIndex={focusedProviderIndex === index ? 0 : -1}
|
||||
>
|
||||
<div
|
||||
dangerouslySetInnerHTML={{
|
||||
__html: (providerOption as any).highlightedName || providerOption.name,
|
||||
}}
|
||||
/>
|
||||
<div className="flex items-center gap-2">
|
||||
{LOCAL_PROVIDERS.includes(providerOption.name) && (
|
||||
<span
|
||||
className={classNames(
|
||||
'inline-block w-2 h-2 rounded-full flex-shrink-0',
|
||||
localProviderStatus[providerOption.name] === 'connected'
|
||||
? 'bg-green-500'
|
||||
: localProviderStatus[providerOption.name] === 'disconnected'
|
||||
? 'bg-red-400'
|
||||
: 'bg-bolt-elements-textTertiary',
|
||||
)}
|
||||
/>
|
||||
)}
|
||||
<span
|
||||
dangerouslySetInnerHTML={{
|
||||
__html: (providerOption as any).highlightedName || providerOption.name,
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
))
|
||||
)}
|
||||
@@ -717,8 +779,17 @@ export const ModelSelector = ({
|
||||
? `No models match "${debouncedModelSearchQuery}"${showFreeModelsOnly ? ' (free only)' : ''}`
|
||||
: showFreeModelsOnly
|
||||
? 'No free models available'
|
||||
: 'No models available'}
|
||||
: provider?.name && LOCAL_PROVIDERS.includes(provider.name)
|
||||
? `No models found — is ${provider.name} running?`
|
||||
: 'No models available'}
|
||||
</div>
|
||||
{!debouncedModelSearchQuery && provider?.name && LOCAL_PROVIDERS.includes(provider.name) && (
|
||||
<div className="text-xs text-bolt-elements-textTertiary mt-1">
|
||||
Make sure {provider.name} is running and has at least one model loaded.
|
||||
{provider.name === 'Ollama' && ' Try: ollama pull llama3.2'}
|
||||
{provider.name === 'LMStudio' && ' Load a model in LM Studio first.'}
|
||||
</div>
|
||||
)}
|
||||
{debouncedModelSearchQuery && (
|
||||
<div className="text-xs text-bolt-elements-textTertiary">
|
||||
Try searching for model names, context sizes (e.g., "128k", "1M"), or capabilities
|
||||
|
||||
Reference in New Issue
Block a user