Finding 1
Lower-stack fault sources do not map one-to-one to agent-visible faults or final outcomes.
RESEARCH ARTIFACT
When Messages Go Wrong
Understanding Communication Faults in LLM-Based Multi-Agent Systems
Code and experimental artifacts: GitHub repository ↗
Abstract
Large language model-based multi-agent systems (LLM-MAS) rely on message exchange to coordinate agents and interact with remote tools. Communication anomalies can leave messages processable while changing the information consumed downstream. We present a large-scale empirical study of communication faults in LLM-MAS, combining a cross-layer fault taxonomy with controlled fault injection and execution tracing across 100 tasks, five domains, three workflow topologies and three deployed model configurations. Communication faults can remain silent: 20.7% of failures with complete detection records are silent, rising to 76.2% for partial-content faults. Flat retains 53.0% task success under faults, compared with 3.2–3.3% for Sequential and Hierarchical, with its advantage concentrated where an alternative information path remains available. Execution-side and evidence protection recover complementary failures; evidence recovery remains effective only while trustworthy information is reachable through an unaffected path.
Research Questions and Findings
Four research questions connect fault manifestation, context sensitivity, fault tolerance and protection mechanisms.
RQ1 · Manifestation and propagation
How do communication faults become visible to agents and propagate to MAS consequences and final task outcomes?
Finding 2
Silent failures concentrate in faults that preserve a processable message.
RQ2 · Sensitivity
How do injection position and task context shape the impact of communication faults?
Finding 3
Fault impact depends on whether the affected information remains recoverable.
Finding 4
The most harmful communication fault changes with task context.
RQ3 · Fault tolerance
How do topologies and model configurations affect outcomes under communication faults?
Finding 5
Flat improves fault tolerance when its additional path still carries the required evidence.
Finding 6
Changing the LLM provides limited recovery from communication faults.
RQ4 · Protection mechanisms
Which mechanisms protect execution and information dependencies against communication faults?
Finding 7
Effective protection must match the failed dependency and retain access to trustworthy evidence.
Supplemental Materials
Methods, complete experimental results and archived evidence are available on the following pages.
- Methodology — taxonomy, injection boundaries, operators, evaluation and reproduction protocol.
- Experimental Details — motivation case, figures, quantitative findings and case explanations.
- — archived messages, exposure records and task outcomes.
- — filter and inspect records associated with each finding.
- — archive coverage, task numbering and interpretation limits.
Research questions & findings
Four research questions · Seven findingsSelect a run to inspect its evidence.
Scope & provenance
Frozen archives · source digests · explicit limitsResearch findings and evidence
The study addresses four research questions and reports seven findings. Aggregate statistics summarize the study results; individual archived runs provide case-level evidence.
The replay catalog covers the common matrix, source-mapping fixtures and historical protection collections. Position-extension and later protection analyses are distributed separately with their own denominators.
What this artifact reproduces
This website replays archived evidence. It does not rerun an agent, execute a fault, or contact a model provider. Source identifiers link every record to an archived input file.
Task numbering
Task 001–100 identifies the 100 main-study tasks across models, topologies and repetitions. Original benchmark IDs remain visible. Shopping uses 001–030, Admin 031–050, Reddit 051–065, SWE-bench 066–085, and TheAgentCompany 086–100. The Reddit extension reuses these IDs; the separate protection study uses RQ4 Task IDs, and lower-stack tests use Fixture IDs. These numbers identify tasks or fixtures, not individual executions.
Download fixed task-ID mapping ↓
Reading the outcomes
Composite success follows the archived adapter endpoint; it can require an accepted evidence contract as well as a correct answer. Handling response is a recorded flag, not proof of a complete recovery chain.
Repeated executions are not controlled model sampling seeds. A missing consequence label is not proof that no propagation occurred.
Evidence availability
All verified records are indexed. Detailed exports cover representative condition/outcome groups, named case studies, all RQ4 runs, and fixture measurements. Other records expose only catalog fields and are labeled summary only.
URLs, credentials, email addresses and private deployment paths are redacted. Event lists preserve source order. Separate fault logs are not assigned to conversation steps without recorded links.
Collections
Important boundaries
Source-to-symptom fixtures have no MAS success score. Frozen-evidence SWE/TAC trials do not represent a fresh end-to-end benchmark execution for every injected fault. RQ4 is a separate Shopping mechanism study with finite exposure budgets.
Download manifest and SHA-256 digests ↓