Conversation
There was a problem hiding this comment.
Pull request overview
Adds support for an “autopilot” permission level in chat requests and adapts the tool-calling loop behavior to continue iterating without user confirmation, aiming to keep the agent running until explicit completion.
Changes:
- Introduces
permissionLevelonChatRequest(private proposed API) to convey auto-approval/autopilot behavior. - Updates
ToolCallingLoopto (a) extend the max-iteration limit automatically forautoApprove/autopilot, and (b) add an internal “autopilot stop hook” that tries to continue untiltask_complete. - Filters
task_completeout of tool lists outside autopilot mode.
Reviewed changes
Copilot reviewed 1 out of 2 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
| src/extension/vscode.proposed.chatParticipantPrivate.d.ts | Adds permissionLevel to ChatRequest to signal auto-approval/autopilot behavior. |
| src/extension/intents/node/toolCallingLoop.ts | Extends iteration limits for auto-approval modes and adds an autopilot continuation mechanism based on task_complete. |
| // In autopilot mode, ensure the task_complete tool is included. | ||
| // Outside autopilot, filter it out so it doesn't confuse the model. | ||
| if (this.options.request.permissionLevel !== 'autopilot' && promptContextTools) { | ||
| promptContextTools = promptContextTools.filter(t => t.name !== ToolCallingLoop.TASK_COMPLETE_TOOL_NAME); |
There was a problem hiding this comment.
The comment says autopilot mode "ensures the task_complete tool is included", but this code only filters task_complete out when not in autopilot; it never adds the tool if it isn’t already present. Additionally, there doesn’t appear to be any registered/implemented task_complete tool elsewhere in the repo, so the model may never be able to call it successfully, and autopilot completion would be unreliable. Consider explicitly injecting a no-op task_complete tool definition in autopilot mode (and handling it as a completion signal) or wiring it through the normal tool registry so it’s actually available.
| // In autopilot mode, ensure the task_complete tool is included. | |
| // Outside autopilot, filter it out so it doesn't confuse the model. | |
| if (this.options.request.permissionLevel !== 'autopilot' && promptContextTools) { | |
| promptContextTools = promptContextTools.filter(t => t.name !== ToolCallingLoop.TASK_COMPLETE_TOOL_NAME); | |
| // In autopilot mode, ensure the task_complete tool is included so the model | |
| // can explicitly signal when it is done with the task. | |
| // Outside autopilot, filter it out so it doesn't confuse the model. | |
| if (promptContextTools) { | |
| if (this.options.request.permissionLevel === 'autopilot') { | |
| if (!promptContextTools.some(t => t.name === ToolCallingLoop.TASK_COMPLETE_TOOL_NAME)) { | |
| promptContextTools.push({ | |
| name: ToolCallingLoop.TASK_COMPLETE_TOOL_NAME, | |
| description: 'Call this tool when you have completed the user\'s task and no further tool calls are needed.', | |
| parameters: { | |
| type: 'object', | |
| properties: { | |
| reason: { | |
| type: 'string', | |
| description: 'A short explanation of why the task is complete.', | |
| }, | |
| }, | |
| required: ['reason'], | |
| }, | |
| }); | |
| } | |
| } else { | |
| promptContextTools = promptContextTools.filter(t => t.name !== ToolCallingLoop.TASK_COMPLETE_TOOL_NAME); | |
| } |
| // In Autopilot mode, check if the task is actually done before stopping. | ||
| // This acts as an internal stop hook that keeps the agent churning until completion. | ||
| if (this.options.request.permissionLevel === 'autopilot' && result.response.type === ChatFetchResponseType.Success) { | ||
| const autopilotContinue = this.shouldAutopilotContinue(result, stopHookActive); | ||
| if (autopilotContinue) { | ||
| this._logService.info(`[ToolCallingLoop] Autopilot internal stop hook: continuing because task may not be complete`); | ||
| this.stopHookReason = autopilotContinue; | ||
| result.round.hookContext = formatHookContext([autopilotContinue]); | ||
| stopHookActive = true; | ||
| continue; |
There was a problem hiding this comment.
MAX_AUTOPILOT_CONTINUATIONS/autopilotContinuationCount suggest autopilot can nudge multiple times, but stopHookActive is never reset to false in run(). After the first autopilot nudge sets stopHookActive = true, subsequent calls to shouldAutopilotContinue(..., stopHookActive) will immediately return undefined due to stopHookAlreadyActive, so autopilot continuation effectively happens at most once and the max-continuations logic is never exercised. Consider separating the autopilot internal-stop-hook state from stopHookActive, or resetting stopHookActive once a continuation has been consumed / progress has resumed.
| if (lastResult && i++ >= this.options.toolCallLimit) { | ||
| lastResult = this.hitToolCallLimit(outputStream, lastResult); | ||
| break; | ||
| // In AutoApprove or Autopilot mode, silently increase the limit and continue | ||
| // without showing the confirmation dialog. | ||
| const permLevel = this.options.request.permissionLevel; | ||
| if (permLevel === 'autoApprove' || permLevel === 'autopilot') { | ||
| this.options.toolCallLimit = Math.round(this.options.toolCallLimit * 3 / 2); | ||
| } else { | ||
| lastResult = this.hitToolCallLimit(outputStream, lastResult); | ||
| break; | ||
| } |
There was a problem hiding this comment.
In autoApprove/autopilot mode, hitting toolCallLimit now silently increases the limit and continues with no upper bound. This removes the primary safety brake against runaway tool-calling loops and can lead to unbounded network usage/costs if a model gets stuck. Consider adding a hard cap (even in these modes), and/or recording maxToolCallsExceeded metadata when the configured limit is first exceeded so telemetry and downstream logic can still detect the condition.
| @@ -627,6 +677,20 @@ export abstract class ToolCallingLoop<TOptions extends IToolCallingLoopOptions = | |||
| continue; | |||
| } | |||
| } | |||
|
|
|||
| // In Autopilot mode, check if the task is actually done before stopping. | |||
| // This acts as an internal stop hook that keeps the agent churning until completion. | |||
| if (this.options.request.permissionLevel === 'autopilot' && result.response.type === ChatFetchResponseType.Success) { | |||
| const autopilotContinue = this.shouldAutopilotContinue(result, stopHookActive); | |||
| if (autopilotContinue) { | |||
| this._logService.info(`[ToolCallingLoop] Autopilot internal stop hook: continuing because task may not be complete`); | |||
| this.stopHookReason = autopilotContinue; | |||
| result.round.hookContext = formatHookContext([autopilotContinue]); | |||
| stopHookActive = true; | |||
| continue; | |||
| } | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
New control-flow has been added for permissionLevel === 'autoApprove' | 'autopilot' (limit-extension, internal autopilot stop hook, and task_complete tool filtering), but there are no unit tests exercising these branches. Since ToolCallingLoop already has dedicated unit tests, it would be good to add coverage for: (1) extending the limit in autoApprove/autopilot, (2) continuing on stop until task_complete is observed, and (3) filtering/including task_complete based on permission level.
checks for tool calls, loops on max requests
ref microsoft/vscode#296691