chore: backport Coder Agents system-instruction improvements to 2.37 (#29237, #29281, #29401, #29579) - #29654

… retrying an exclusive tool (#29281)

When the model batches `advisor` (an exclusive tool) with other tool
calls, every call in the batch is rejected and the `advisor` result only
says to retry with `advisor` alone. That nudges the model toward
re-calling `advisor` immediately, even when the sibling calls were there
to gather context for it.

Extend the message so the model is told to run the other tools first
when it still needs their output, then retry with only the exclusive
tool call.

Generated by Coder Agents on behalf of @ibetitsmike.

(cherry picked from commit 0362e53)

(cherry picked from commit eb800dc)

Chatd's guidance discouraged workspace creation even when the chat
lacked the tools or context needed to finish a request. Make a suitable
workspace the fallback when missing tools, skills, MCPs, or context
block progress, while preferring existing capabilities when sufficient.

Align the system prompt, workspace awareness, planning guidance, tool
description, regression tests, and user documentation. Preserve
template-selection guardrails and delegated-agent restrictions.

**Open review finding:** workspace readiness can precede initial
MCP/context discovery. The wording now acknowledges this limitation; a
runtime synchronization fix is pending a scope decision.

<details>
<summary>Implementation and verification context</summary>

- Coding requests authorize necessary workspace setup without a separate
workspace request; self-contained examples remain workspace-free when
existing capabilities suffice.
- Detached root chats create a workspace; attached stopped workspaces
use start_workspace. Persisted awareness describes the initial state
rather than claiming the chat remains detached.
- Delegated chats report workspace needs to their parent. Provisioning
does not unlock workspace MCP tools in Plan Mode.
- Use actually exposed capabilities; do not assume readiness means all
skills, MCP tools, and context have loaded, or recreate workspaces to
retry discovery.
- Prompt, workspace-creation, template-selection, setup-failure, Plan
Mode MCP boundary, and API prompt-composition tests pass. The creation
E2E now verifies published instructions, skills, and MCP tools appear in
request two with no extra user turn; repeated and race runs pass. This
does not establish live-model compliance or discovery readiness.
- Full make pre-commit passes, including generation, formatting, lint,
and the slim build. All CI checks pass on f4f6133 (23 passed, 8
skipped), and Codex reports no major issues on that commit.
- Codex review iterations corrected prompt/tool conflicts. The alleged
immutable-per-turn tool catalog was disproved by the per-task generation
lifecycle. The narrower initial-discovery race remains open: no existing
chatd-visible signal distinguishes pending discovery from completed
empty or failed discovery.

</details>

Generated by Coder Agents on behalf of @ibetitsmike.

Backport note: the TestCreateWorkspaceTool_EndToEnd rewrite from the
original commit is omitted. It depends on DisableWorkspaceAgentContextSync
(#28522) and on same-turn workspace-context publication, neither of which
is on release/2.37. The 2.37 version of that test still passes with the
new guidance; only the workspace-awareness assertions are updated.
(cherry picked from commit cc8c766)