Focused Entailment Graphs for Open IE Propositions Omer Levy Ido Dagan Jacob Goldberger Bar-Ilan University, Israel Open IE Extracts propositions from text which makes aspirin relieve headaches. No supervision No pre-defined schema Whats missing in Open IE? Structure

Open IE does not consolidate natural language expressions relieve headache treat headache Adding Structure to Open IE Which structure? Build a graph of Open IE propositions and their semantic relations Adding Structure to Open IE Which structure? Build a graph of Open IE propositions and their entailment relations Why entailment? Merges paraphrases into mutual entailment cliques aspirin relieves headache aspirin treats headache

Organizes information hierarchically from specific to general aspirin relieves headache painkiller relieves headache aspirin, eliminate, headache aspirin, cure, headache coffee, help, headache drug, relieve, headache headache, control with, aspirin drug, treat, headache tea, soothe, headache analgesic, banish, headache headache, treat with, caffeine

headache, respond to, painkiller Original Open IE Output drug, relieve, headache drug, treat, headache headache, respond to, painkiller headache, treat with, caffeine analgesic, banish, headache headache, control with, aspirin tea, soothe, headache

aspirin, cure, headache aspirin, eliminate, headache coffee, help, headache Consolidated Open IE Output Semantic Applications Example: Structured Queries What relieves headaches? Semantic Applications Example: Structured Queries What relieves headaches?

drug, relieve, headache drug, treat, headache headache, respond to, painkiller headache, treat with, caffeine analgesic, banish, headache headache, control with, aspirin tea, soothe, headache aspirin, cure, headache aspirin, eliminate, headache

coffee, help, headache Structured Query: drug, relieve, headache drug, treat, headache headache, respond to, painkiller headache, treat with, caffeine analgesic, banish, headache headache, control with, aspirin

tea, soothe, headache aspirin, cure, headache aspirin, eliminate, headache coffee, help, headache Structured Query: drug painkiller caffeine

analgesic tea aspirin coffee Structured Query: Our Contributions Structuring Open IE with Proposition Entailment Graphs Dataset: 30 gold-standard graphs, 1.5 million entailment annotations Algorithm for constructing Focused Proposition Entailment Graphs Analysis: Predicate entailment is not quite what we thought Proposition Entailment Graphs

Related Work: Predicate Entailment Graphs Berant et al. (2010,2011,2012) We extend Berant et al.s work from predicates to propositions Focused Proposition Entailment Graphs Nodes: Open IE propositions Edges: Textual Entailment Focused Proposition Entailment Graphs Assumptions: Binary Propositions and Common Topic Binary Propositions

Focused on a common topic Focused Proposition Entailment Graphs Assumptions: Binary Propositions and Common Topic Binary Propositions Focused on a common topic drug, relieve, headache drug, treat, headache headache, respond to, painkiller headache, treat with, caffeine

analgesic, banish, headache headache, control with, aspirin tea, soothe, headache aspirin, cure, headache aspirin, eliminate, headache coffee, help, headache drug, relieve, headache drug, treat, headache headache, respond to, painkiller

headache, treat with, caffeine analgesic, banish, headache headache, control with, aspirin tea, soothe, headache aspirin, cure, headache aspirin, eliminate, headache coffee, help, headache Focused Proposition Entailment Graphs Edges: Textual Entailment

Proposition Entailment Simpler than sentence-level entailment More complicated than lexical entailment Enables investigation of inference phenomena in an isolated manner Constructing Proposition Entailment Graphs Task Definition: Given a set of propositions , find all their entailment edges. Dataset Dataset: High-Quality Open IE

Propositions Googles Syntactic N-grams Based on millions of books Filter for subject-verb-object Including prepositional objects and passive Result: 68 million high-quality propositions Dataset: Annotating Entailment Graphs Select 30 healthcare topics antibiotic, caffeine, insomnia, scurvy, Collect a set of propositions focused on each topic Manually clean noisy extractions

Retaining 200 propositions per graph (average) Efficiently annotate entailment 1.5 million entailment judgments Algorithm How do we recognize proposition entailment? . ? How do we recognize proposition entailment?

. Observation: propositions entail their lexical components entail How do we recognize proposition entailment? . Observation: propositions entail their lexical components entail How do we recognize proposition entailment? .

Proposition entailment is reduced to lexical entailment in context Lexical Entailment Lexical Entailment Features 1 Lexical Entailment (Logistic) 2

3 = ( ) Lexical Entailment Features WordNet Relations UMLS Distributional Similarity String Edit Distance Lexical Entailment Features

1 Lexical Entailment (Logistic) 2

3 = ( ) Supervision From Lexical to Proposition Entailment Lexical Entailment Features 1 Lexical Entailment (Logistic)

2 3 = ( ) Supervision From Lexical to Proposition

Entailment Predicate Entailment Features 1 Predicate Entailment (Logistic) 2

3 = ( ) Supervision Argument Entailment Features 1

Argument Entailment (Logistic) 2 3

= ( ) Supervision From Lexical to Proposition Entailment Predicate Entailment Features 1

Predicate Entailment (Logistic) 2 3 = ( ) Supervision

Proposition Entailment (Conjunction) Argument Entailment Features 1 Argument Entailment (Logistic) =

2 3 = ( ) Supervision Distant Supervision (WordNet)?

Predicate Entailment Features 1 Predicate Entailment (Logistic) 2

3 = ( ) WordNet Proposition Entailment (Conjunction) Following Snow (2005), Berant (2012) Argument Entailment Features

1 Argument Entailment (Logistic) = 2

3 = ( ) WordNet Direct Supervision (30 Annotated Graphs) Predicate Entailment Features

1 Predicate Entailment (Logistic) 2 Argument Entailment Features 3

= ( ) Proposition Entailment (Conjunction) 1 Argument Entailment (Logistic)

= Annotated Graphs 2 3

= ( ) Direct Supervision (30 Annotated Graphs) Predicate Entailment Features 1 2

Argument Entailment Features 3 1 Hidden Layer

Proposition Entailment (Conjunction) = Annotated Graphs 2

3 Flat Model Predicate Entailment Features 1 2

Argument Entailment Features 3 Proposition Entailment (Logistic) = ( Annotated Graphs

1 2 + ) 3

Compared Methods Component-Level Distant Supervision (WordNet) Predicates & Arguments Predicates Only Arguments Only Proposition-Level Direct Supervision (30 Annotated Graphs) Hierarchical (our method) Flat All methods used Berant et al.s Global Optimization method Results

Direct Supervision: Flat vs Hierarchical Hierarchal model performs better than flat model 70% Better to model predicate and argument entailment separately Performance (F1) 65% 60% 63.7%

55% 50% 61.6% Distant vs Direct Supervision 70% Although WordNet provides more training examples 65% Performance (F1) Direct supervision is better

60% 63.7% 55% 50% 61.6% 59.7% Predicate Entailment with Distant Supervision Ignoring predicates improves distant supervision baselines

70% 60% Performance (F1) 50% 40% 30% 59.7% 20% 10% 8.0% 0%

7.2% Are WordNet relations capturing real-world predicate entailments? Predicate Entailment vs WordNet Relations Over a predicate inference subset, how many predicate entailments are covered by WordNet? Positive 11.64% Negative 14.65%

Positive indicators synonyms, hypernyms, entailment None 73.70% Predicate Entailment vs WordNet Relations Over a predicate inference subset, how many predicate entailments are covered by WordNet? Positive 11.64% Negative 14.65%

Positive indicators synonyms, hypernyms, entailment Negative Indicators None 73.70% antonyms, hyponyms, cohyponyms Why isnt WordNet capturing predicate entailment? Predicate Entailment is ContextSensitive The words do not necessarily entail,

but the situations do. Predicate Entailment is ContextSensitive The words do not necessarily entail, but the situations do. Investigating Context-Sensitive Entailment Recent work on context-sensitive lexical inference e.g. (Melamud et al., 2013) Previous datasets Lexical substitution (McCarthy and Navigli, 2007) Predicate inference (Zeichner et al., 2012)

We offer a new dataset of real-world lexical entailments in context! Sample: synthetic vs naturally occurring Size: several thousands vs 1.5 million Conclusion Conclusion Structuring Open IE with Proposition Entailment Graphs Algorithm for constructing Focused Proposition Entailment Graphs Analysis: Predicate entailment is extremely context-sensitive Dataset: 1.5 million proposition entailment decisions Thank you for listening!