VENSES – A Linguistically-Based System for Semantic Evaluation

Abstract

Resumen:The system for semantic evaluation VENSES (Venice Semantic Evaluation System) is organized as a pipeline of two subsystems: the first is a reduced version of GETARUN, our system for Text Understanding. The output of the system is a flat list of head-dependent structures (HDS) with Grammatical Relations (GRs) and Semantic Roles (SRs) labels. The evaluation system is made up of two main modules: the first is a sequence of linguistic rule-based subcalls; the second is a quantitatively based measurement of input structures. VENSES measures semantic similarity which may range from identical linguistic items, to synonymous or just morphologically derivable. Both modules go through General Consistency checks which are targeted to high level semantic attributes like presence of modality, negation, and opacity operators, temporal and spatial location checks. Results in cws, accuracy and precision are homogenoues for both training and test corpus and fare higher than 60%.

Semantic Annotation for Textual Entailment Recognition

“…<Annotation Id="1834" Type="reference to" StartNode="11511" EndNode="11527"> <Feature><Name >construction id</Name><Value >2</Value></Feature> </Annotation> [11] 60.6% LCC [31] 73.75% LCC [32] 72.25% MITRE [4] 58.6 % DISCo [33] 63.88% "Al. I. Cuza" [18] 69.13%…”

Recognizing Textual Entailment with a Semantic Edit Distance Metric

“…Delmonte et al [6] introduced semantic-mismatch features such as: locations, discourse markers, quantifiers and antonyms. The entailment decision is based on applying rewards and penalties over the semantic-similarity and shallow scores.…”

“…The work was partially supported by the Mexican Government, specifically, by the Mexican National Council for Science and Technology (CONACYT) via scholarship reference 309261 to the first author and CONACYT 50206-H and SIP-IPN 20121823 projects; by Governments of India and Mexico under the CONACYT-DST India (CONACYT Table V COMPARISON WITH PREVIOUS WORKS OVER THE RTE TEST DATASETS Method RTE-1 (Accuracy) RTE-2 (Accuracy) RTE-3 (Accuracy) Roth and Sammons [4] --65.56% Burchardt and Frank [12], Burchardt et al [5] 54.6% 59.8% 62.62% Delmonte et al [6], Delmonte et al [7], Delmonte et al [13] 59 …”

Indonesian-Japanese CLIR using only limited resource

“…In lack of such a database, we are unable to provide precise performance data. The system has also been used for the RTE Challenge and performance was over 60% correct [11].…”

“…However, instances of those terms that only occurred in the fixed reference terms did not, in fact, refer to the citation of the paper, by definition of the ideal reference terms. For instance, one such extra occurrence of ibm is from a sentence following the citation that describes the exact model used in the current work: (11) According to the IBM models (Brown et al, 1993), the statistical word alignment model can be generally represented as in Equation (1) ... In this paper, we use a simplified IBM model 4 (Al-Onaizan et al, 1999), which ...…”