FC Portugal: Robotic Soccer Team

Brief and project description

The Algorithm Cards Evaluation (ACE) framework is an international research initiative that aims to establish a standard, user-centric paradigm for the validation of model documentation tools in Responsible AI. Led by a collaborative research team of computer science and human-computer interaction experts, the work addresses the critical gap in how the AI ecosystem evaluates transparency artifacts. Its goal is to develop a structured methodology and empirical testing framework that measures how well distinct user personas understand, trust, and leverage algorithm cards, creating a virtuous circle between AI developers, compliance auditors, and end-users. The Faculty of Engineering of the University of Porto, through LIACC's ongoing work in Responsible AI technologies, aligns closely with the algorithmic transparency principles explored in this foundational study.

The motivation behind this evaluation framework is straightforward: as algorithm cards become the standard tool to document fairness, accountability, and limitations, the industry lacks proof that these cards actually achieve their purpose for human readers. Making AI "transparent" is meaningless if the documentation remains opaque to the stakeholders who rely on it. Ensuring documentation effectiveness means verifying that design choices are clear, intuitive, and actionable, fostering public trust and robust regulatory compliance.

Within this broader effort, the research contribution focuses on empirical user studies, quantitative comprehension metrics, and qualitative feedback loops as a basis for building optimized algorithm card designs. The methodology standardizes evaluation parameters across varied technical and non-technical demographics, tracking how layout and data representation affect decision-making. By combining human-centered design with rigorous statistical testing, the work aims to make artifact evaluation more reproducible, objective, and resistant to the design blind spots that purely technical benchmarks ignore.

The expected outcome is a set of validated evaluation methods, user-study protocols, and design recommendations that help organizations assess whether their transparency tools behave effectively before public release — turning abstract transparency ideals into concrete, measurable, and user-tested documentation criteria.

Period: March 2023 – April 2023