Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation
Published in arXiv preprint, 2026
Recommended citation: Haocheng Yang, Licheng Pan, Yiming Dong, Xiaoxi Li, Yuan Lu, Zhenhao Chen, Jiajun Fan, Shuai Shao, Chunyuan Zheng, Hao Wang. "Not All Criteria Matter: Importance-Aware Criteria Weighting for Rubric-Based LLM Evaluation." arXiv preprint arXiv:2610.19915, 2026. https://arxiv.org/abs/2610.19915
Importance-Aware Criteria Weighting (ICW) weights the criteria of rubric-based LLM evaluation by importance instead of uniformly: an LLM makes pairwise importance comparisons, inconsistent ones are checked and revised, and the weights come from the principal eigenvector of the comparison matrix. It outperforms uniform weighting and competing weighting methods.