2.1.0

BoxLang AI Module v2.1.0 Release Notes - Multi-Tenant Tracking, OpenSearch Vector Memory, Bedrock Streaming, and Provider Configuration

This release focuses on enterprise features including multi-tenant usage tracking, OpenSearch vector memory support, AWS Bedrock streaming, and enhanced provider configuration capabilities.

🎉 Major Features

Multi-Tenant Usage Tracking

Provider-agnostic request tagging for per-tenant billing and usage tracking.

Track Usage by Tenant:

// Tag requests with tenant information
result = aiChat(
    messages: "Generate report",
    options: {
        tenantId: "customer-123",
        usageMetadata: {
            costCenter: "marketing",
            projectId: "campaign-2024",
            userId: "[email protected]"
        }
    }
)

// Works with all providers: OpenAI, Bedrock, Ollama, DeepSeek, etc.

Intercept Token Usage Events:

// Register interceptor for billing
BoxRegisterInterceptor( {
    class: "TenantBillingInterceptor",
    points: {
        onAITokenCount: function( event, data ) {
            var usage = data.usage // { prompt_tokens, completion_tokens, total_tokens }
            var tenantId = data.tenantId
            var metadata = data.usageMetadata

            // Record usage for billing
            queryExecute(
                "INSERT INTO tenant_usage (tenant_id, tokens, cost_center, timestamp) VALUES (:tid, :tokens, :cc, :ts)",
                {
                    tid: tenantId,
                    tokens: usage.total_tokens,
                    cc: metadata.costCenter,
                    ts: now()
                }
            )
        }
    }
} )

Predefined Provider Configuration

Define multiple providers with default parameters in module configuration.

Module Configuration:

// In ModuleConfig.bx or application settings
settings = {
    providers: {
        "openai": {
            params: {
                model: "gpt-4"
            },
            options: {
                apiKey: "my-openai-api-key"
            }
        },
        "ollama": {
            params: {
                model: "qwen2.5:0.5b-instruct"
            },
            options: {
                baseUrl: "http://my-ollama-server:11434/"
            }
        }
    }
}

Using Predefined Providers:

// Use predefined configuration
model = aiModel( provider: "openai" ) // Automatically uses gpt-4 and API key

// Seed services with additional options
model = aiModel(
    provider: "openai",
    options: { logRequestToConsole: true }
)

Provider-Specific Options

Generic providerOptions struct for provider-specific configuration.

AWS Bedrock Inference Profiles:

result = aiChat(
    provider: "bedrock",
    messages: "Analyze this data",
    options: {
        providerOptions: {
            inferenceProfileArn: "arn:aws:bedrock:us-east-1:123456789012:inference-profile/my-profile"
        }
    }
)

Access Provider Options in Custom Services:

// In custom service implementation
function invoke( required request ) {
    var profileArn = arguments.request.getProviderOption(
        key: "inferenceProfileArn",
        defaultValue: ""
    )

    if ( profileArn.len() ) {
        // Use inference profile
    }
}

OpenSearch Vector Memory Provider

Full integration with OpenSearch k-NN for semantic search and RAG applications.

Create OpenSearch Memory:

// Basic configuration
memory = aiMemory(
    provider: "opensearch",
    options: {
        url: "https://search-domain.us-east-1.es.amazonaws.com",
        indexName: "conversation-memory",
        embeddingProvider: "openai",
        vectorDimension: 1536
    }
)

// AWS OpenSearch with authentication
memory = aiMemory(
    provider: "opensearch",
    options: {
        url: "https://search-domain.us-east-1.es.amazonaws.com",
        indexName: "knowledge-base",
        embeddingProvider: "openai",
        vectorDimension: 1536,
        awsRegion: "us-east-1", // Enables SigV4 authentication
        username: "admin",
        password: "MyPassword123!"
    }
)

// Custom HNSW configuration
memory = aiMemory(
    provider: "opensearch",
    options: {
        url: "http://localhost:9200",
        indexName: "vectors",
        embeddingProvider: "ollama",
        vectorDimension: 768,
        spaceType: "cosinesimilarity", // or "l2", "innerproduct"
        hnswM: 16,
        hnswEfConstruction: 100,
        hnswEfSearch: 100
    }
)

Multi-Tenant Isolation:

// Store with tenant context
memory.add(
    messages: messages,
    userId: "user-123",
    conversationId: "conv-456"
)

// Query with tenant filtering
results = memory.query(
    query: "What did we discuss about pricing?",
    userId: "user-123",
    conversationId: "conv-456",
    limit: 5
)

OpenAI-Compatible Embedding Support

Use custom OpenAI-compatible embedding services with vector memory providers.

Custom Embedding Endpoint:

// Use Ollama for embeddings
memory = aiMemory(
    provider: "pinecone",
    options: {
        embeddingProvider: "openai",
        embeddingOptions: {
            baseURL: "http://localhost:11434/v1" // Ollama endpoint
        }
    }
)

// Use LM Studio for embeddings
memory = aiMemory(
    provider: "qdrant",
    options: {
        embeddingProvider: "openai",
        embeddingOptions: {
            baseURL: "http://localhost:1234/v1" // LM Studio endpoint
        }
    }
)

AWS Bedrock Streaming Support

Full streaming support for AWS Bedrock provider with all model families.

Stream Bedrock Responses:

// Stream with Claude on Bedrock
aiChatStream(
    provider: "bedrock",
    messages: "Write a long story",
    params: { model: "anthropic.claude-3-sonnet-20240229-v1:0" },
    onChunk: function( chunk ) {
        writeOutput( chunk.content )
    }
)

// Supported model families: Claude, Titan, Llama, Mistral

Custom Ollama Base URLs

Configure custom base URLs for Ollama chat and embeddings endpoints.

Custom Ollama Server:

// Configure in module settings
settings = {
    providers: {
        "ollama": {
            options: {
                baseUrl: "http://my-ollama-server:11434/"
            }
        }
    }
}

// Or pass directly to BIF
result = aiChat(
    provider: "ollama",
    messages: "Hello!",
    options: { baseUrl: "http://custom-server:11434/" }
)

🎁 New Features

• New Event: onMissingAiProvider - Handle cases where requested provider is not found

• aiModel() Options: New options struct parameter to seed services with configuration

• Request Merge Control: AiBaseRequest.mergeServiceParams() and mergeServiceHeaders() now accept override boolean argument

• Ollama Embeddings: Support for nomic-embed-text model for local embeddings

🔧 Enhancements

• Provider Defaults: All AI provider services inherit default chat and embedding parameters from IAiService interface

• Service Configuration: IAiService.configure() now accepts generic options argument instead of just apiKey

• Request Clarity: AiRequest class renamed to AiChatRequest for clarity and multi-modality support

• Error Handling: Added more AiError exception handling for service JSON errors

• Docker Retry Logic: Increased Model Runner retry time to 5 seconds with 10 max retries for large model loading

🐛 Bug Fixes

• Event Names: Corrected chat request event names to onAIChatRequest, onAIChatRequestCreate, and onAIChatResponse

• Headers Passthrough: aiChat, aiChatStream now correctly pass headers to AiChatRequest

• Request Building: BIFs now use aiChatRequest() to build requests instead of manual construction

• MCP Spec Compliance: Prompts now return arguments key instead of args per MCP specification

• Model Setting: AiChatRequest now correctly sets model from params

• API Key Passthrough: API key now properly passed to service in aiChat(), aiChatStream() BIFs

• Character Function: Fixed typo from chr() to char() in SSE formatting

• Model Retrieval: AiModel.getModel() now correctly returns model name when using predefined providers

• OpenSearch URL: Fixed parameter conflict by using requestUrl for HTTP requests instead of url

📚 Documentation Updates

• OpenSearch vector memory provider documentation

• Multi-tenant usage tracking examples

• Provider-specific options documentation

• Custom embedding endpoint configuration

• Bedrock streaming examples

🚀 Upgrade Notes

This release is backward compatible. New features are opt-in:

1. Multi-tenant tracking - Add tenantId and usageMetadata options to requests

2. Predefined providers - Configure providers in module settings for cleaner code

3. OpenSearch - Use for scalable vector search in production

4. Custom embeddings - Point to self-hosted embedding services

5. Bedrock streaming - Enable real-time streaming for Bedrock models

🙏 Thank You

Thank you to all contributors and users who continue to make BoxLang AI better!