feat: add Cerebras and Fireworks AI LLM providers (#2113)
* fix: improve local model provider robustness and UX - Extract shared Docker URL rewriting and env conversion into BaseProvider to eliminate 4x duplicated code across Ollama and LMStudio - Add error handling and 5s timeouts to all model-listing fetches so one unreachable provider doesn't block the entire model list - Fix Ollama using createOllama() instead of mutating provider internals - Fix LLMManager singleton ignoring env updates on subsequent requests - Narrow cache key to only include provider-relevant env vars instead of the entire server environment - Fix 'as any' casts in LMStudio and OpenAILike by using shared convertEnvToRecord helper - Replace console.log/error with structured logger in OpenAILike - Fix typo: filteredStaticModesl -> filteredStaticModels in manager - Add connection status indicator (green/red dot) for local providers in the ModelSelector dropdown - Show helpful "is X running?" message when local provider has no models Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add Cerebras LLM provider - Add Cerebras provider with 8 models (Llama, GPT OSS, Qwen, ZAI GLM) - Integrate @ai-sdk/cerebras@0.2.16 for compatibility - Add CEREBRAS_API_KEY to environment configuration - Register provider in LLMManager registry Models included: - llama3.1-8b, llama-3.3-70b - gpt-oss-120b (reasoning) - qwen-3-32b, qwen-3-235b variants - zai-glm-4.6, zai-glm-4.7 (reasoning) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add Fireworks AI LLM provider - Add Fireworks provider with 6 popular models - Integrate @ai-sdk/fireworks@0.2.16 for compatibility - Add FIREWORKS_API_KEY to environment configuration - Register provider in LLMManager registry Models included: - Llama 3.1 variants (405B, 70B, 8B Instruct) - DeepSeek R1 (reasoning model) - Qwen 2.5 72B Instruct - FireFunction V2 Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add coding-specific models to existing providers Enhanced providers with state-of-the-art coding models: **DeepSeek Provider:** + DeepSeek V3.2 (integrates thinking + tool-use) + DeepSeek V3.2-Speciale (high-compute variant, beats GPT-5) **Fireworks Provider:** + Qwen3-Coder 480B (262K context, best for coding) + Qwen3-Coder 30B (fast coding specialist) **Cerebras Provider:** + Qwen3-Coder 480B (2000 tokens/sec!) - Removed deprecated models (qwen-3-32b, llama-3.3-70b) Total new models: 4 Total coding models across all providers: 12+ Performance highlights: - Qwen3-Coder: State-of-the-art coding performance - DeepSeek V3.2: Integrates thinking directly into tool-use - ZAI GLM 4.6: 73.8% SWE-bench score - Ultra-fast inference: 2000 tok/s on Cerebras Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add dynamic model discovery to providers Implemented getDynamicModels() for automatic model discovery: **DeepSeek Provider:** - Fetches models from https://api.deepseek.com/models - Automatically discovers new models as DeepSeek adds them - Filters out static models to avoid duplicates **Cerebras Provider:** - Fetches models from https://api.cerebras.ai/v1/models - Auto-discovers new Cerebras models - Keeps UI up-to-date with latest offerings **Fireworks Provider:** - Fetches from https://api.fireworks.ai/v1/accounts/fireworks/models - Includes context_length from API response - Discovers new Qwen-Coder and other models automatically **Moonshot Provider:** - Fetches from https://api.moonshot.ai/v1/models - OpenAI-compatible endpoint - Auto-discovers new Kimi models Benefits: - ✅ No manual updates needed when providers add new models - ✅ Users always have access to latest models - ✅ Graceful fallback to static models if API fails - ✅ 5-second timeout prevents hanging - ✅ Caching system built into BaseProvider Technical details: - Uses BaseProvider's built-in caching system - Cache invalidates when API keys change - Failed API calls fallback to static models - All endpoints have 5-second timeout protection Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * feat: add Z.AI provider with GLM models and JWT authentication Merged changes from PR #2069 to add Z.AI provider: - Added GLM-4.6 (200K), GLM-4.5 (128K), and GLM-4.5 Flash models - Implemented secure JWT token generation with HMAC-SHA256 signing - Added dynamic model discovery from Z.AI API - Included proper error handling and token validation - GLM-4.6 achieves 73.8% on SWE-bench coding benchmarks Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Stijnus
@@ -2,7 +2,7 @@ import { BaseProvider } from '~/lib/modules/llm/base-provider';
|
||||
import type { ModelInfo } from '~/lib/modules/llm/types';
|
||||
import type { IProviderSetting } from '~/types/model';
|
||||
import type { LanguageModelV1 } from 'ai';
|
||||
import { ollama } from 'ollama-ai-provider';
|
||||
import { createOllama } from 'ollama-ai-provider';
|
||||
import { logger } from '~/utils/logger';
|
||||
|
||||
interface OllamaModelDetails {
|
||||
@@ -39,31 +39,17 @@ export default class OllamaProvider extends BaseProvider {
|
||||
|
||||
staticModels: ModelInfo[] = [];
|
||||
|
||||
private _convertEnvToRecord(env?: Env): Record<string, string> {
|
||||
if (!env) {
|
||||
return {};
|
||||
}
|
||||
|
||||
// Convert Env to a plain object with string values
|
||||
return Object.entries(env).reduce(
|
||||
(acc, [key, value]) => {
|
||||
acc[key] = String(value);
|
||||
return acc;
|
||||
},
|
||||
{} as Record<string, string>,
|
||||
);
|
||||
}
|
||||
|
||||
getDefaultNumCtx(serverEnv?: Env): number {
|
||||
const envRecord = this._convertEnvToRecord(serverEnv);
|
||||
const envRecord = this.convertEnvToRecord(serverEnv);
|
||||
|
||||
return envRecord.DEFAULT_NUM_CTX ? parseInt(envRecord.DEFAULT_NUM_CTX, 10) : 32768;
|
||||
}
|
||||
|
||||
async getDynamicModels(
|
||||
private _resolveBaseUrl(
|
||||
apiKeys?: Record<string, string>,
|
||||
settings?: IProviderSetting,
|
||||
serverEnv: Record<string, string> = {},
|
||||
): Promise<ModelInfo[]> {
|
||||
serverEnv?: Record<string, string>,
|
||||
): string {
|
||||
let { baseUrl } = this.getProviderBaseUrlAndKey({
|
||||
apiKeys,
|
||||
providerSettings: settings,
|
||||
@@ -73,31 +59,55 @@ export default class OllamaProvider extends BaseProvider {
|
||||
});
|
||||
|
||||
if (!baseUrl) {
|
||||
throw new Error('No baseUrl found for OLLAMA provider');
|
||||
throw new Error('No baseUrl found for Ollama provider');
|
||||
}
|
||||
|
||||
if (typeof window === 'undefined') {
|
||||
/*
|
||||
* Running in Server
|
||||
* Backend: Check if we're running in Docker
|
||||
*/
|
||||
const isDocker = process?.env?.RUNNING_IN_DOCKER === 'true' || serverEnv?.RUNNING_IN_DOCKER === 'true';
|
||||
baseUrl = this.resolveDockerUrl(baseUrl, serverEnv);
|
||||
|
||||
baseUrl = isDocker ? baseUrl.replace('localhost', 'host.docker.internal') : baseUrl;
|
||||
baseUrl = isDocker ? baseUrl.replace('127.0.0.1', 'host.docker.internal') : baseUrl;
|
||||
return baseUrl;
|
||||
}
|
||||
|
||||
async getDynamicModels(
|
||||
apiKeys?: Record<string, string>,
|
||||
settings?: IProviderSetting,
|
||||
serverEnv: Record<string, string> = {},
|
||||
): Promise<ModelInfo[]> {
|
||||
const baseUrl = this._resolveBaseUrl(apiKeys, settings, serverEnv);
|
||||
|
||||
try {
|
||||
const response = await fetch(`${baseUrl}/api/tags`, {
|
||||
signal: this.createTimeoutSignal(),
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
|
||||
}
|
||||
|
||||
const data = (await response.json()) as OllamaApiResponse;
|
||||
|
||||
return data.models.map((model: OllamaModel) => ({
|
||||
name: model.name,
|
||||
label: `${model.name} (${model.details.parameter_size})`,
|
||||
provider: this.name,
|
||||
maxTokenAllowed: 8000,
|
||||
}));
|
||||
} catch (error) {
|
||||
if (error instanceof DOMException && error.name === 'TimeoutError') {
|
||||
logger.warn('Ollama model fetch timed out — is Ollama running?');
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
if (error instanceof TypeError && error.message.includes('fetch')) {
|
||||
logger.warn(`Ollama not reachable at ${baseUrl} — is Ollama running?`);
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
logger.error('Error fetching Ollama models:', error);
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
const response = await fetch(`${baseUrl}/api/tags`);
|
||||
const data = (await response.json()) as OllamaApiResponse;
|
||||
|
||||
// console.log({ ollamamodels: data.models });
|
||||
|
||||
return data.models.map((model: OllamaModel) => ({
|
||||
name: model.name,
|
||||
label: `${model.name} (${model.details.parameter_size})`,
|
||||
provider: this.name,
|
||||
maxTokenAllowed: 8000,
|
||||
}));
|
||||
}
|
||||
|
||||
getModelInstance: (options: {
|
||||
@@ -107,33 +117,18 @@ export default class OllamaProvider extends BaseProvider {
|
||||
providerSettings?: Record<string, IProviderSetting>;
|
||||
}) => LanguageModelV1 = (options) => {
|
||||
const { apiKeys, providerSettings, serverEnv, model } = options;
|
||||
const envRecord = this._convertEnvToRecord(serverEnv);
|
||||
const envRecord = this.convertEnvToRecord(serverEnv);
|
||||
|
||||
let { baseUrl } = this.getProviderBaseUrlAndKey({
|
||||
apiKeys,
|
||||
providerSettings: providerSettings?.[this.name],
|
||||
serverEnv: envRecord,
|
||||
defaultBaseUrlKey: 'OLLAMA_API_BASE_URL',
|
||||
defaultApiTokenKey: '',
|
||||
});
|
||||
|
||||
// Backend: Check if we're running in Docker
|
||||
if (!baseUrl) {
|
||||
throw new Error('No baseUrl found for OLLAMA provider');
|
||||
}
|
||||
|
||||
const isDocker = process?.env?.RUNNING_IN_DOCKER === 'true' || envRecord.RUNNING_IN_DOCKER === 'true';
|
||||
baseUrl = isDocker ? baseUrl.replace('localhost', 'host.docker.internal') : baseUrl;
|
||||
baseUrl = isDocker ? baseUrl.replace('127.0.0.1', 'host.docker.internal') : baseUrl;
|
||||
const baseUrl = this._resolveBaseUrl(apiKeys, providerSettings?.[this.name], envRecord);
|
||||
|
||||
logger.debug('Ollama Base Url used: ', baseUrl);
|
||||
|
||||
const ollamaInstance = ollama(model, {
|
||||
const ollamaProvider = createOllama({
|
||||
baseURL: `${baseUrl}/api`,
|
||||
});
|
||||
|
||||
return ollamaProvider(model, {
|
||||
numCtx: this.getDefaultNumCtx(serverEnv),
|
||||
}) as LanguageModelV1 & { config: any };
|
||||
|
||||
ollamaInstance.config.baseURL = `${baseUrl}/api`;
|
||||
|
||||
return ollamaInstance;
|
||||
});
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user