What we measure, and what we refuse to measure

WebArena: A Realistic Web Environment for Building Autonomous Agents ICLR 2024

Borrowed Realistic resettable websites with task-specific final-state validators, rather than a judge scoring a transcript.

Not adopted Its site set is generic and English-first. MICA extends the same validator idea to localized consumer markets and to surfaces that have no browsable equivalent.

VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks ACL 2024

Borrowed Rendered-state tasks and visual evidence: what the user can actually see is part of the task, not an implementation detail.

Not adopted MICA does not assume DOM-only access anywhere in the method, so a system that only reads markup is not privileged by the design.

WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? ICML 2024

Borrowed Parameterized task templates validated against backend state, so one task definition yields many checkable instances.

Not adopted Its domain is enterprise knowledge work on one platform. MICA's unit is a household consumer journey across independent local services.

OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments NeurIPS 2024

Borrowed Snapshot and reset discipline, and execution-based evaluators that read computer state instead of trusting a claim of success.

Not adopted A resettable desktop cannot reset a third-party merchant, a bank or a carrier. MICA's parity conditions do that work instead.

AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents arXiv 2024 / ICLR 2025

Borrowed Parameterized mobile tasks with executable app-state checks, which is the closest published analogue to app-enclosed consumer journeys.

Not adopted Its apps are installed and controlled by the harness. MICA's markets include signed-in channels the harness will never own.

Mind2Web: Towards a Generalist Agent for the Web NeurIPS 2023

Borrowed Breadth: a task taxonomy spanning many websites and domains, and an explicit test of generalization to unseen sites.

Not adopted Offline action matching against recorded traces is not proof of final state. MICA does not accept step agreement as success.

WebLINX: Real-World Website Navigation with Multi-Turn Dialogue ICML 2024

Borrowed Multi-turn conversational traces and website-disjoint splits, which is how MICA thinks about a user who changes their mind mid-journey.

Not adopted Offline trace metrics stay diagnostic in MICA. They never become the primary success criterion for a task.

AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks? 2024

Borrowed Long-horizon realistic tasks, and the practice of reporting the effort a task actually cost alongside whether it succeeded.

Not adopted Its outcomes are largely information answers. MICA's unit is a persistent state change in a real consumer service.

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents NeurIPS 2022

Borrowed Scalable consumer shopping goals expressed as constraint satisfaction over an instruction, which MICA reuses in its shopping family contracts.

Not adopted A single simulated store is narrower than a market. MICA's shopping tasks cross payment, identity and delivery rails that a single store hides.

WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models ACL 2024 / arXiv v1 25 January 2024

Borrowed Operation on live websites rather than a frozen replica: 643 tasks across 15 real sites, which is the published precedent for evaluating against services the harness does not own.

Not adopted There is no systematic per-country localized suite, and success is judged in part by a GPT-4V reading of screenshots and agent output rather than by authoritative market-side completion. MICA requires a receipt, readback or handoff record instead.

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents ACL 2024 / arXiv v1 26 July 2024

Borrowed Everyday consumer domains spanning multiple interacting apps and people, checked by state-based tests rather than by output matching.

Not adopted Its apps are a controllable simulated world authored for the benchmark, not 6 independently localized real-life markets with their own payment, identity and messaging rails.

tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains 2024

Borrowed Stateful user, agent and tool interaction judged against domain policy and backend state, plus repeated-trial reliability rather than one lucky run.

Not adopted It covers two simulated service domains, retail and airline, in a single language. They are not 6 country-localized real consumer markets, and MICA's policies are the authorization conventions those markets actually apply.

API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs EMNLP 2023

Borrowed The decomposition of tool use into deciding a tool is needed, retrieving it, forming arguments, executing and reading the response.

Not adopted A well-formed call is not a completed errand. MICA scores the final state, and uses call diagnostics only to explain it.

Berkeley Function-Calling Leaderboard: Structured function-call evaluation, including non-use, multi-call and multi-turn categories Evolving leaderboard; cite with the access date

Borrowed Call normalization before comparison, and the insight that correctly declining to call a tool is a measurable behaviour.

Not adopted It is a live, versionless surface. MICA cites it as a dated evolving resource and never treats a snapshot of it as a fixed standard.

AgentBench: Evaluating LLMs as Agents ICLR 2024

Borrowed Heterogeneous interactive environments reported per environment, so a strength in one setting cannot be laundered into a general claim.

Not adopted MICA does not aggregate unlike rewards into one figure. Family and market means are taken only over commensurable per-task scores.

GAIA: A Benchmark for General AI Assistants ICLR 2024

Borrowed Real-world multimodal tool tasks with unambiguous answers, and a hidden split held back from the public set.

Not adopted A correct final answer is not a persistent state change. MICA requires an authoritative receipt or readback, not a string match.

SWE-bench: Can Language Models Resolve Real-World GitHub Issues? ICLR 2024

Borrowed Success as positive obligations plus regression and invariant checks, with a per-instance artifact that makes a claim auditable.

Not adopted Its invariants are test suites the harness owns. MICA has to express the same idea as prohibited-state checks on services it does not own.

HELM: Holistic Evaluation of Language Models TMLR 2023

Borrowed The separation of scenario, system, metric and run, and the discipline of multidimensional disclosure instead of one headline number.

Not adopted HELM does not justify a single universal product score. MICA's per-task product is its own decision and is defended as such.

MLPerf Inference: MLPerf Inference Benchmark ISCA 2020

Borrowed Latency is comparable only under a fixed workload, a fixed quality target and a declared system boundary. MICA fixes all three before timing anything.

Not adopted Its workloads are deterministic and hardware-bounded. A consumer market changes under the runner, so MICA reports speed with the conditions attached.

OECD / JRC: Handbook on Constructing Composite Indicators: Methodology and User Guide 2008

Borrowed Normalization, weighting and compensability are substantive decisions that must be documented and tested, not formatting choices.

Not adopted MICA uses this handbook to criticize its own product and to require sensitivity analysis. It is not an endorsement of MICA's formula.

Keeney & Raiffa: Decisions with Multiple Objectives: Preferences and Value Tradeoffs Cambridge University Press edition

Borrowed The conditions under which a multiplicative form is a legitimate utility function: stated preference structure and specific independence assumptions.

Not adopted MICA has not established those assumptions, so its per-task product is a scoring policy and is not claimed to be a formal utility.

Fleming & Wallace: How Not to Lie with Statistics: The Correct Way to Summarize Benchmark Results CACM 1986

Borrowed Geometric means are the right summary for commensurable positive ratios, which is exactly the shape of a normalized speed or cost component.

Not adopted It does not license multiplying heterogeneous metrics in general, and MICA does not cite it as if it did.

Sokolova & Lapalme: A Systematic Analysis of Performance Measures for Classification Tasks Information Processing & Management, 2009

Borrowed The macro versus micro distinction: whether you average over items or over groups is a weighting decision with visible consequences.

Not adopted The original setting is classification metrics. MICA applies only the weighting concept, to tasks within a family and families within a market.

Agarwal et al.: Deep Reinforcement Learning at the Edge of the Statistical Precipice NeurIPS 2021

Borrowed Few-run point estimates mislead. Report repeated runs, uncertainty intervals and performance distributions rather than a single mean.

Not adopted Its estimators assume continuous returns. MICA's outcomes are binary and hierarchical, so the adaptation has to be justified, not copied.

Kakao Developers: App Official documentation · accessed 8 August 2026

Borrowed The documentation establishes that use of the platform begins with a registered application holding app keys and per-product configuration, so the registered app is the unit access is granted against.

Not adopted It does not establish that registration grants every capability. What a registered app may actually do is set per product and per scope, so MICA does not infer execution rights from registration.

Kakao Developers: Kakao Talk Message · REST API Official documentation · accessed 8 August 2026

Borrowed The documentation establishes that sending a message depends on a user access token, a consented scope, a supported message template and a permitted recipient relationship, which is a materially different condition from read-oriented local search.

Not adopted It does not establish that messaging is unavailable, and MICA does not read it that way. It establishes that the send path is conditioned, which is why declared coverage is never treated as executable coverage.

LINE Developers: Creating a channel Official documentation · accessed 8 August 2026

Borrowed The documentation establishes a provider and channel structure as the control plane: a channel is created for a specific product and carries its own credentials.

Not adopted Creating a channel is not blanket product approval, and this page does not say it is. Individual products keep their own eligibility and review conditions.

LINE Developers: Sending messages Official documentation · accessed 8 August 2026

Borrowed The documentation establishes that a send depends on an Official Account channel and its access token, on the recipient relationship, and on which send method applies, rather than on a generic outbound call.

Not adopted It does not establish anything about non-messaging surfaces, and MICA does not generalize from it to payment, commerce or account operations on the same platform.

LINE Developers: LINE MINI App development flow Official documentation · accessed 8 August 2026

Borrowed The documentation establishes a lifecycle in which development and testing precede a review step before release, so reaching production is a gated transition rather than a deployment.

Not adopted It does not establish uniform worldwide availability. Regional availability and per-region conditions can differ, so MICA records executability per market rather than per platform.

WeChat Mini Program: Release Official documentation · accessed 8 August 2026

Borrowed The documentation establishes an account-controlled runtime bound to an AppID, where a build is uploaded, submitted for review and only then released to users.

Not adopted The English page is not a complete statement of current conditions. It can lag the Chinese documentation and the admin console, so MICA cites it for the shape of the control and verifies specifics per market edition.

Grab Developer: Grab Developer Documentation Official documentation · accessed 8 August 2026

Borrowed The documentation root establishes that access is organized around a developer application and product-specific onboarding, with some production capabilities conditioned by country or partner status.

Not adopted It does not establish which specific capability is open in which market on a given date. MICA cites the documentation root rather than a deep link because product pages move, and it verifies specifics per market edition.

Brown et al. (GPT-3): Language Models are Few-Shot Learners NeurIPS 2020

Borrowed Benchmark overlap audits and clean-subset comparison as a routine, published part of a result rather than an afterthought.

Not adopted Overlap auditing is not a proof of cleanliness. MICA states residual contamination risk rather than declaring a set uncontaminated.

Magar & Schwartz: Data Contamination: From Memorization to Exploitation ACL 2022

Borrowed Exposure, memorization and score exploitation are three different things, and only the third necessarily corrupts a comparison.

Not adopted MICA cannot measure exploitation directly today, so it controls exposure and records provenance instead of claiming exploitation is absent.

Dynabench: Rethinking Benchmarking in NLP NAACL 2021

Borrowed Human and model in the loop challenge collection, which is how MICA intends to keep a rotating challenge set adversarial.

Not adopted Adversarial collection skews toward edge cases. MICA keeps a stable longitudinal core so the index does not drift into a corner-case suite.

LiveBench: A Challenging, Contamination-Free LLM Benchmark 2024 / ICLR 2025

Borrowed Regular refresh from recent material with objectively verifiable answers, which limits how long an exposed item stays worth exploiting.

Not adopted MICA will not claim contamination freedom. Refresh reduces risk; it does not eliminate it, and the wiki says so.

Pineau et al.: Improving Reproducibility in Machine Learning Research (Reproducibility Program Report) JMLR 2021

Borrowed A reproducibility checklist attached to every published result: what was run, on what, how many times, and what was reported.

Not adopted Full artifact release is impossible where a trace contains account state. MICA publishes redacted traces and keeps originals access-controlled.

Datasheets for Datasets: Documentation of motivation, composition, collection, uses, distribution and maintenance CACM 2021

Borrowed The documentation frame MICA applies to each edition of the task catalogue, one datasheet section per axis.

Not adopted A datasheet describes a dataset. MICA's catalogue is a set of executable contracts, so it also documents environment and authorization conditions.

MEGA: Multilingual Evaluation of Generative AI EMNLP 2023

Borrowed Per-language reporting as a requirement, because an average across languages hides exactly the market where a system fails.

Not adopted Multilingual coverage is not cultural localization. MICA treats language as one input to locality, not as locality itself.

BLEnD: A Benchmark for LLMs on Everyday Knowledge in Diverse Cultures and Languages NeurIPS 2024

Borrowed Everyday cultural knowledge collected locally rather than inferred, which is the standard MICA holds its task authors to.

Not adopted A country is not homogeneous. MICA records the region, channel and register a task assumes rather than labelling it with a flag.

M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models AAAI 2024

Borrowed Locally sourced material behaves differently from translated English material, which is why MICA requires local authorship rather than translation.

Not adopted Exam items have fixed answers. A consumer errand's correct outcome depends on live local state, so MICA validates final state instead.

Data Statements for NLP: Toward Mitigating System Bias and Enabling Better Science TACL 2018

Borrowed Documenting language variety, collection context and the situation of the people who authored and reviewed the material.

Not adopted MICA records reviewer context at the level of role and market competence only. No individual is identifiable from a published statement.

NIST AI RMF 1.0: Artificial Intelligence Risk Management Framework NIST, 2023

Borrowed The GOVERN, MAP, MEASURE and MANAGE structure, and its demand for documented ownership, validity criteria and ongoing monitoring.

Not adopted MICA claims no NIST alignment. A conformance claim would require a published mapping, and MICA has not produced one.

The Benchmark Lottery: How benchmark, task and metric choice can change the conclusion 2021

Borrowed The warning that a ranking can be an artifact of the suite. MICA publishes task-level results so a conclusion can be recomputed differently.

Not adopted The lesson is not that ranking is impossible. It is that a ranking without sensitivity analysis is not evidence, which MICA accepts as a requirement.