A frozen Jev pipeline reports the lowest ten-corpus valence-arousal error in a SemEval table

This topic was created 29 days ago, and the information it contains may have evolved or changed since then.

Yiqun Zhang and coauthors report the lowest ten-corpus valence-arousal error among full-coverage systems, using frozen Jev and 488 fitted coefficients.

arXiv:2609.35293, submitted September 28, 2026, breaks aspect-based sentiment into typed questions for a frozen Jev and then fits a small set of coefficients. The HTML we opened names the model jev-1.13.0. Code is at ZhangYiqun018/jev-dimabsa.

An October 1 post from @HEI links both. The license line on the paper is the arXiv non-exclusive license.

What was scored

SemEval-2026 Task III Track A has three tasks. Task 1 predicts valence and arousal, each on a scale from 1 to 9, for a given aspect. Tasks 2 and 3 extract an aspect, an opinion, and those two numbers, and Task 3 adds a category. The test set for Task 1 is 9,658 reviews and 16,186 aspect annotations, ten corpora, six languages.

Tasks 2 and 3 share 6,690 reviews and 14,262 triplets. The official scorer is used with Task 1 normalization disabled.

Jev is asked for Score, Choice, and Noul answers. No component generates text, and the backbone weights stay frozen. Task adaptation is 488 coefficients fit on CPU: 100 in the Task 1 ridge models, 332 in the Task 2 rerankers, 32 in the pair-level maps, and 24 in category fusion. The Task 2 reranker is fit on development labels.

The other fits use training labels. Test labels are used for scoring.

What Table 1 prints

Micro RMSE on Task 1 is 1.0645. The paper calls that the lowest aggregate among the 14 teams that report all ten corpora, 0.0018 under PAI at 1.0663. It also says the margin cannot be tested for significance without the other systems’ predictions.

Per corpus, the system beats the best published score on English laptop and Tatar restaurant, and trails on the other eight.

Task 2 continuous F1 is 52.09, between sixth and seventh of 12 complete-coverage participants. Task 3 is 44.06, between fifth and sixth of 9. Both are above the fine-tuned Llama-3.3-70B baseline, 46.40 and 38.62, and the GPT-OSS-120B baseline, 45.71 and 37.27. Llama’s Task 1 RMSE in that table is 2.5683. GPT-OSS-120B is 1.2362.

Takoyaki, a retrieval-and-rules system that does generate text, is about four points higher on both extraction tasks, 56.20 and 48.03. PAI’s extraction score on Task 2 is 57.73.

Where the error moves

Raw Jev scores, still with nine demonstrations, reach 2.0731 RMSE. The ridge calibration brings that to 1.0645. The paper says calibration removes about half of the error. Gold valence and arousal on the extracted pairs would add 4.46 continuous F1 to Task 2, so most of the remaining extraction error is which spans are chosen.

Averaged over corpora, 21.7% of gold pairs are never proposed and 26.8% are proposed and not selected. Adding a category drops this system by 8.03 continuous F1, from 52.09 to 44.06, which the paper marks as the smallest such drop among systems that report both tasks at or above this score.

The comparison is not a controlled swap of generation against decisions. The other systems use different backbones and different supervision. The paper says so in the caption’s neighborhood: the table places the results in context.

This site's reading

Editorial notes evaluating claims against primary sources, contextualizing findings alongside related implementations, and defining technical terms.

Verify

arXiv:2609.35293v1, submitted September 28, 2026. The HTML license line is the arXiv.org perpetual non-exclusive license. Authors are Yiqun Zhang, Peidong Wang, Zihan Wang, and Shi Feng, Northeastern University, China. We opened the HTML on October 2, 2026.

Table 1 prints 1.0645 RMSE, 52.09 continuous F1, and 44.06 continuous F1. The protocol section names jev-1.13.0. The coefficient split is 100, 332, 32, and 24. The test counts are 9,658 reviews and 16,186 aspect annotations for Task 1, and 6,690 reviews with 14,262 triplets for Tasks 2 and 3.

We did not open a LICENSE file in the GitHub repository. We did not rerun the scorer.

The October 1 post from @HEI, status 2105711847848435820, links the abstract and the repository. The attached image is the card saved for this page. The post text does not add a number beyond the title. Participant aggregates in Table 1 are reconstructed from Yu et al. 2026, Tables 6 to 8.

The paper says the 0.0018 gap under PAI cannot be tested without participant predictions. Task 2 is described as between sixth and seventh of 12 complete-coverage systems, and Task 3 between fifth and sixth of 9.

Compare

This is a sentiment benchmark, not a row on the JevBench page. The model is frozen jev-1.13.0. The 488 coefficients are fit on labels. Raw Task 1 error before that fit is 2.0731 RMSE, against 1.0645 after it.

Llama-3.3-70B in the same table is 2.5683, 46.40, and 38.62. GPT-OSS-120B is 1.2362, 45.71, and 37.27. Both of those baselines are 4-bit QLoRA. Takoyaki, which generates text and does not tune a backbone, is 56.20 and 48.03 on Tasks 2 and 3, about four points above this pipeline.

PAI's Task 1 aggregate is 1.0663, and its Task 2 score is 57.73.

Terms

continuous F1: The paper's extraction score. An exact span match contributes a weight that falls as the valence-arousal error grows, and the reported figure is 100 times the macro average over eight corpora.
Task 1: Given-aspect valence and arousal on a 1 to 9 scale. Ten corpora, six languages. The headline is micro RMSE over every annotation. Lower is better. The paper's number is 1.0645.
488 coefficients: The paper's count of fitted weights, 100 in the Task 1 ridge models, 332 in the Task 2 rerankers, 32 in the pair-level valence-arousal maps, and 24 in category fusion. They are fit on CPU. The Jev weights are not updated.