SemanticFind: Locating What You Want in a Patient Record, Not ...
SemanticFind: Locating What You Want in a Patient Record, Not Just What You Ask For John M. Prager, Jennifer J. Liang, Murthy V. Devarakonda IBM T.J. Watson Research Center AMIA Joint Summits, San Francisco CA, March 31 2017 Overview What is SemanticFind? What it is not How is SemanticFind different from Crtl/F? 13 match types 4 technologies Prototype User Interface Evaluation Conclusions What is SemanticFind? An application that aids the physician search a patients medical record for matches to terms of interest Extension of the familiar Ctrl/F find capability in document creation and reading applications Not limited to matching solely the entered
search term Can find matching content along a variety of different dimensions What it is Not Not an application designed for finding matching records (patients) amongst a large collection, i.e. clinical trial matching Can be done, just not optimized for this Not a presentation of a new user interface Prototype GUI is used in subsequent slides for demonstration of functionality Not a Question-Answering System So how is it different from Ctrl/F Search terms represent information needs, but 1. Information needs cannot usually be answered fully just by locating instances of them in text 2. Ambiguous intent behind the search term E.g. if the search term is a disease, user might be wondering
When/how it was first diagnosed What indicative labs for it were over time Are there counterindications Are there complications What treatments were prescribed for it Approach is to perform a variety of different searches simultaneously, and present the results organized by search type EHR systems have both structured information (e.g. tables of orders, diagnoses, lab results) and unstructured (e.g. progress and other notes, free text) Both kinds are searched Search terms are unrestricted: symptoms, diseases, medications, anatomical structures, any sequence of characters. Search is mediated by UMLS, so search terms that correspond to concepts in UMLS can be more fully explored Sample Screenshot: Pain in the abdomen SemanticFind Search Types (1) Trad. Search Search Type Explanation
Comments Literal Traditional find Ctrl-F Includes plurals Semantic Hypothetical Returns terms with same semantic meaning as search term Returns terms in negative context, incompatible with or opposite of the search term Returns terms that are not being asserted to apply to patient More general Returns terms that are broader than search term More specific
Returns terms that are narrower than search term Assoc. Procedures Returns procedures associated with search term Assoc. Tests Returns tests or findings associated with search term Assoc. Treatments Returns treatments associated with search term Assoc. Medications Returns medications related with this search Contradicted
Conceptual Search Associative Search Enables you to search for concepts without regard to clinical terminology represented in record or level of specificity Returns terms that are often related to the search term in the medical literature. IMPORTANT: Not necessarily related in this patient 7 SemanticFind Search Types (2) Search Type Ordered Medications Possible Med
Concerns Inferential Search Ordered Procedures Explanation Comments Returns medications from medication orders logically connected to search term. The medication often treats or prevents the condition (if searching for disease) Returns medications from medication orders that are known to be contraindicated with the search term Returns procedures from the procedure orders logically connected to the search term. The procedure often treats or prevents the condition (search term) Returns labs from lab orders that are logically connected to the search term. Searches through the Orders in
the patient record (structured data) to infer potential relations to the search term, based on known relationships in the medical literature. The explanation field shows the logical relation between the search term and the matched item IMPORTANT: Not necessarily related in this patient Ordered Labs The labs often aid in diagnosing or monitoring a condition (search term) 8 Technologies Used Literal Match As traditional search, but case-insensitive and disregards singular/plural Conceptual Match UMLS concepts and relations for Semantic, More General/Specific Our own Medical Concept Annotator, conceptually similar to cTAKES and MetaMap, but higher accuracy (paper in preparation)
Lab values and vitals mapped to indicated conditions K 6.1 gets annotated as hyperkalemia Parse- and Linguistic-principle-based transformations to catch semantically matching concepts/variants in UMLS Pain in the abdomen not a variant of abdominal pain NLP for Negation and Hypothetical Patient denies discomfort with the rash Ordered urine test to rule out arsenic poisoning Associative Match Uses Latent Semantic Analysis Finds terms in the record that occur in the same contexts in the literature as the search terms Useful for finding terms correlated with search term, but no named relation, e.g. sob wheezing Inferential Match Finds terms in the record that are related through curated relation chains to the search term Most useful for , , relations, e.g. Infection Lower respiratory tract infection Amoxicillin Augmentin 875 mg-125 mg tablet Semantic Search Examples of Search Types Search Term Example Matches
Type of Match Uterus Uterus, uteri Literal Shortness of breath Difficulty breathing, SOB, dyspnea Semantic High blood pressure hypertension, HTN, BP 150/100, elevated BP Semantic Normal blood pressure Hypertension, HTN, BP 90/60, low BP Contradicted Match
Tobacco Patient denies smoking Contradicted Match DM Risk of diabetes Hypothetical Match Hyperlipidemia Hypercholesterolemia More Specific PTSD Anxiety disorder More General Leg pain Pain in the lower limb
More Specific Pain in the lower limb Leg pain More General Insulin Sugar, Novolin, Metformin, Glucose Assoc. Medications Sertraline Asthma Risperidone, cyproheptadine, Zoloft, Antipsychotics, Assoc. Medications Depakote Cardiac Surgery, Coronary Artery Bypass, Percutaneous Assoc. Treatments Coronary Intervention, PCI Albuterol Ordered Meds
Chest Pain Electrocardiogram Heart attack Ordered Procedures 10 Evaluation 3 types suitable for evaluation Semantic Match More Specific Contradicted 10 records selected at random Average of 250 clinical notes per record MD developed list of (13-32) search terms for each Total of 169 terms, 134 unique 4th-yr medical students used as assessors Assessors generated a list of paraphrases for each search term 0-13 per term. Total of 652. Based on medical knowledge and/or lookup, not seeing medical records. Assessment task, per search term
SemanticFind used interactively to locate matches Precision: GUI enhanced with evaluation widgets for assessors to enter judgments of GOOD or BAD #GOOD = True Positives (TP) #BAD = False Positives (FP) Precision = TP/(TP+FP) Recall: System automatically searched for user-generated paraphrases (via Literal Match), and counted how many of these did not correspond to GOOD in Precision task. This count = False Negatives (FN) Recall = TP/(TP+FN) F-Measure F = 2PR/(P+R) Evaluation Interface Results (1) Precision Precision Batch Overall True Positives
11851 False Positives 1728 #matches judged 13579 Precision 0.87 Error analysis shows most FPs due to ambiguity of abbreviations negation detection error Recall: 2 modes evaluated Unconstrained = all supplied paraphrases Constrained = only those paraphrases that matched UMLS concepts Recall Mode Unconstrained Constrained
True Positives 11851 11851 False Negatives 1704 297 Recall 0.87 0.98 F-Measure Unconstrained/Constrained = 0.87/0.92 Results (2) Progressive analysis of GOOD matches: Relative to Literal Match as a baseline Semantic Match Corresponds (very roughly) to Ctrl/F + Synonym Expansion
Semantic Match + More Specific Semantic Match, More Specific + Contradicted Search types considered % GOOD matches as compared to Literal Match alone Literal Match alone 100% Semantic Match, relative to LM 121% SM + More Specific, relative to LM 190% SM + MS + Contradicted Match, relative to LM 203% 103% Dark Matter Interesting Negation Detection Error Due to somewhat informal formatting/writing of clinical notes, e.g.:
alcohol use : no smoking : yes Implicit sentence-end clear to humans, but not to computer, giving rise to recognition of no smoking On fixing problem, reduced false positives by 30% Conclusions SemanticFind is application to search within a patient record 13 searches performed simultaneously using a variety of NLP technologies Organised in a tabbed interface High accuracy F = 0.87 or 0.92 Est. 2 points higher when sentence-end problem fixed Dark Matter calculation shows that Ctrl/F misses as many desirable matches as it finds THANK YOU
The Mini Board. Ongoing, "in the moment" assessments may be the most powerful tool teachers have for improving student performance. For students to get better at anything, they need lots of quick rigorous practice, spaced over time, with immediate feedback.
Note: Students coded Y or N must have a code E, F or U on the Discipline/Resultant Action Code element. Changes. Appendix P, Definitions for Incident Reporting. SESIR Poster/Definitions and Guidelines. SESIR Edits. Reports for Request.
Workshop Goals. Explore the Beyond the Standards Math K-5 Video Series and Perspectives Videos . Learn to navigate CPALMS and explore modules and resources which can be used for Professional Development
Bugs and Drugs: an approach to the management of infections J. Pernica Division of Pediatric Infectious Diseases Useful tidbits (II) clindamycin also is NOT excreted in urine aminoglycosides almost never used alone aminoglycosides have poor lung penetration and generally should...
A sentencemust express a complete thought.Think about that for a moment. In order to be a correct sentence, it must express a complete thought. What do we mean by complete?. To be a complete thought, a sentence needs a
Branched keys and statement keys! A key is used in biology t help us identify a certain type of species. There are two types of branching keys and a statement key. Branched keys! This is a branched key diagram which...
Introduction. Smartphones and tablets have become the tools of choice for today's workers. What will tomorrow bring? Google glasses? iWatch? How do we provide and protect corporate data in a mobile world?
Ready to download the document? Go ahead and hit continue!