Courses But there are often aggregate judgments that I need to make, where many of the words don't make any difference. It's not that we made some arithmetic mistake. So let me answer the first one. If somebody is characterized as being seropositive, that's again good evidence. Course Description 6.863 is a laboratory-oriented course on the theory and practice of building computer systems for human language processing, with an emphasis on the linguistic, cognitive, and engineering foundations for understanding their design. And then erosions and so on. KATHERINE LIAO: Yes, thank you, everybody. Home So my expectation, when I heard about this study, is that this would be a disaster. Are there clinical uses that people have adopted that use this kind of approach to trying to read the note? Did you run into each other at the bus stop? And the reason that we've had the luxury of playing around with the data is because Partners was ahead of the curve and had developed an EHR. Then you get this data mart, and then you start training. And then for the narrative text, we used a system that was built by Qing Zeng and her colleagues at the time-- it was called HITex. MIT OpenCourseWare (OCW) is a free, publicly accessible, openly-licensed digital collection of high-quality teaching and learning materials, presented in an easily accessible format. Office hours: Thursdays 10.30-12, Room NE43-723. And so names and things are replaced with square brackets, star, star, star kinds of things. And if you want it to be really sophisticated, you would use an algorithm like NegEx, which is a negation expression detector that helps get rid of things that are not true. So did you seek them out, did they seek you out? You just need to know if they fit a certain pattern. But in the clinical setting, if you mess up, it's a really big deal. » Send to friends and colleagues. So now I walk into the VA, it's a completely different story. The advance could reduce the computing power, energy, and … And as you can imagine, because we're trying to go across many phenotypes, when we think about mapping, it always has to be automated. So we go, really? And in fact, to give you a slightly more quantitative version of this, Kat and I worked on a project back around 2010 where we were looking at trying to understand what are the genetic correlates of rheumatoid arthritis. He hints at it [INAUDIBLE]. So at the VA, we've kind of laid down this pipeline for a phonemic score. KATHERINE LIAO: Yeah, I don't know. So those are all things that are being developed now. There is an online tool where you can type in something and it says weakness of the upper extremities. And then summarization is a very real challenge as well, especially because of the cut and paste that has come about as a result of these electronic medical record systems where, when a nurse is writing a new note, it's tempting and supported by the system for him or her to just take the old note, copy it over to a new note, and then maybe make a few changes. This group, I was not involved in this particular project, said, well, could we replicate the study at Vanderbilt and at Northwestern University? So what they basically did was locked all us in a room for three hours every Friday. IPN stands for intern progress note. And the threshold is going to be-- so there's going to be an entire-- I think there's going to be entire methods development that's going to have to happen between figuring out where that threshold is and the fatigue from the alarms. OK, well, welcome, Kat. That's not a bad proxy for how sick they are, right? But now, instead of saying, I need to identify these patients and get the genotype, all the genotypes are already there. It turned out to be something like 19% in this cohort. And it's wonderful place, although when they built it, it was just a place to die because they really couldn't do much for you. And so what this means is that, for example, if I clutch my chest and go, uh, and an ambulance rushes me over to Mass General and they do a whole bunch of tests and they decide that I'm not having a heart attack, the correct billing code for that visit is myocardial infarction. How could you get billed three times? And it was like, what's the problem, what's the question, and how do we get there. Now, that goes into-- so now instead of, if you think about in the old days, we came up with the list, we had ICD lists and term lists, which got mapped to a concept. So there is always ambiguity. For example, if you're looking to see whether a patient has a certain disease, then you can do a little bit of diagnostic reasoning and say, if I see a lot of symptoms of that disease mentioned, then maybe the disease is present as well. I profoundly believe that. Knowledge is your reward. The same stuff is recorded over and over again. And they would be required by the chief, which is very powerful in France, to use this artificial language to write notes instead of using French to write notes. There are 233,000 findings, 172,000 drugs, organic chemicals, pharmacological substances, amino acid peptide or protein, invertebrate. Attention and the Transformer 4. And we did this from health care provider notes, radiology and pathology reports, discharge summaries, operative reports. Really? But it's definitely impacted our work. PETER SZOLOVITS: So what are the kinds of-- I mean, this study that we were talking about today was for rheumatoid arthritis. This course on natural language processing (NLP) focuses exactly on such problems, covering syntactic, semantic and discourse processing models, and their applications to information extraction, machine translation, and text summarization. Assuming it's present. PETER SZOLOVITS: So that isn't new, by the way. So as an academic group, we try to publish everything we do. And so I'm going to talk about a little bit of that approach. So this is a formalization of that idea as a machine learning problem. David Bates, who's the chief of general internal medicine or something at the Brigham, came in and gave a guest lecture. Freely browse and use OCW materials at your own pace. There are also some tools that deal with the typical linguistic problems, that if I want to say bleeds or bleed or bleeding, those are really all the same concept. That was fascinating. Explore materials for this course in the pages linked along the left. Now I'm over at the Boston VA along with Tianxi. ICD-10 maybe two decades ago. So that's the term list. Now we go straight to the article. So that's the basis of what we do. I did post to our core the consulting session here. Well, we took about four million patients in the EMR. PETER SZOLOVITS: So what map-- presumably, you don't use HITex today. And so a whole industry developed of people saying that not only should we use the terms that we got originally from the doctors who were interested in doing these queries, but we can define a machine learning problem, which is how do we learn the set of terms that we should actually use that will give us better results than just the terms we started with? And I think some of the speakers here have not yet learned that same lesson. And where does it say standardized? This lecture and the next cover the role of natural language processing in machine learning in healthcare. KATHERINE LIAO: Yeah. But I had a question about when a clinician is trying to label data, for example, training data, are there any ambiguities ever, where sometimes this is definitely-- this person has RA. So one of the things that we didn't know when we first started out was how many gold standard labels did we need and how many features did we need and which of those features would be important. So I have to say that Zak was probably very clever in bringing the right people to the table and locking those into that room for three hours at a time because, for example, our biostatistician, Tianxi Cai just, you know, she speaks AI or she has learned to speak AI. KATHERINE LIAO: Couple of things. R2D2? So they could express what they wanted to do in English and the English would be translated into some semantic representation. Just a few. And they said, OK, those are the only ones we're going to alarm on. So this is very useful for looking through this stuff. And the problem is that the incentives for the company that curated this database were to make sure they didn't miss anything, because they didn't want to be responsible for failing to alarm. PETER SZOLOVITS: It also takes the right people. Then we select those patients and we genotype them. So this is a real issue and one that we don't have a very good solution for yet. So we were able to get a positive predictive value of about 0.94. He was a doctor Le Pitie Salpetriere, which is one of these medieval hospitals in Paris. Here, the PPV is estimated from a five-fold cross validation of the data, whereas in our study, we had a held out data set from which we were calculating the positive predictive value. Advanced Natural Language Processing, Language processing. So if you think about when Pete was pointing to the number of ICD counts for ICD-9, for those of you who don't know, ICD-9 was developed decades ago. But on the other hand, I've learned over many decades not to be quite as optimistic as my natural proclivities are. MIT OpenCourseWare is a web-based publication of virtually all MIT course content. The VA happened to have an EHR. Natural Language Processing Group Contact Us Our research encompasses all aspects of NLP, from modeling basic linguistic phenomena to designing practical text processing systems, and developing new machine learning methods. The physicians sits in the office and there's another person who actually listens in and types and then clicks all the buttons that you need to get the information there. It covers syntactic, semantic and discourse processing models, emphasizing machine learning or corpus-based methods and algorithms. YSDA Natural Language Processing course. So ICD-9, you know, doesn't map directly to ICD-10 or back because there were diseases that we didn't know when they developed ICD-9 that exist in ICD-10. Here are the UMLS semantic concepts of various-- or the semantic types. Right? And what they found is that this actually, considering how incredibly simple it is, does reasonably well. Could we do as well or better? So I think the bar is much higher. We don't offer credit or certification for using OCW. So in fact, the pipeline is now part of the Partners Biobank, which is a Partner's Healthcare. So for example, if I'm doing de-identification, essentially I have to look at every word in a narrative in order to see whether it's protected health information. There may maybe a little bit more there, but there's nothing fancy behind it. I have to say, it takes a lot of time. And that's one reason why I think things have been held up, actually. Electrical Engineering and Computer Science, 6.881 Natural Language Processing (Fall 2004), Computer Science > Artificial Intelligence. And then we trained an algorithm that predicted whether this patient really had RA or not. And then also, for certain kinds of applications, what you'd really like to do is to identify what part of a textual record addresses a certain question. We might want to use natural language processing for de-identification of data. And so there is a technical reason for it, but it's still disturbing that we're getting a different result. AUDIENCE: I know we're towards the end. But now, along with EMRs came a lot of regulations on physicians. So he wasn't breathing well. KATHERINE LIAO: Well, I was really lucky. So codified data are things like lab values and prescriptions and demographics and stuff that is in tabular form. One is, when you run an algorithm right on your data set, you can't port it using the same coefficients because it's going to be different for each one. All right? And then you can use those in order to expand the kinds of phrases that you're looking for. Follow their code on GitHub. And then they threatened to kill him. And then there is the normalization function that takes some statement like Mr. Huntington was admitted, blah, blah, blah, and normalizes it into lowercase alphabetized versions of the text, where things are translated into other potential meanings, linguistic meanings of that text. So even if you can't tell what the answer is, you should able to point to a piece of the record and say, oh, this tells me about, in this case, the patient's exercise regimen. So this is difficulty breathing when you're exerting yourself, but that has decreased, presumably from some previous assessment. You find all the UMLS terms in each sentence of a discharge summary. They are being systematically recruited, blood samples are taken, they're genotyped with no study in mind. And they're running through acute stroke, myocardial infarction, all kinds of these-- diabetes-- just really a lot of all the common diseases that we want to study. So alarm fatigue is definitely one of the barriers. There was that filter with the ICD codes. NLP Foundation Skills – NLP Certification -(Practitioner) Provider: Udemy. This project provides free (even for commercial use) state-of-the-art information extraction tools. So we didn't want people to feel like they can just add it on. The subject qualifies as an Artificial Intelligence and Applications concentration subject. So for rheumatoid arthritis, I mentioned it's a systemic chronic inflammatory joint disease. And so you wouldn't want to get rid of that one. If you moused over it, it would show you. One way is to take advantage of related terms like hypo- or hypernyms, things that are subcategories or super categories of a word. And on Tuesday, I'm going to speak mostly about stuff that does depend on neural network representations, but I'm not sure where the boundary is going to fall. Well, I had mentioned that one of the things we want to do is to codify things that appear in a note. So a lot of systems are actually now just getting their EHR. I think when we first started, everyone was-- my colleagues joked with me. NLP at UIUC. So here's an example. PETER SZOLOVITS: As far as I know, that code has never been applied to anybody. It was all run together. I remember a class I taught in biomedical computing about 15 years ago. KATHERINE LIAO: As one of my mentors said, you have to kiss a lot of frogs. It also covers applications of these methods and models in syntactic parsing, information extraction, statistical machine translation, dialogue systems, and summarization. Freely browse and use OCW materials at your own pace. Intended goals of the course: By the end of the course, you should have the background to read research papers in the general area of statistical NLP; and be able to start research in the area. So that was definitely an improvement. But the idea behind SNOMED is that it's more a combinatorial system. The other thing is, you'll see an explosion in the number of ICD codes. Set Your Goals Use OCW to guide your own life-long learning, or to teach others. http://www.landsiedel-seminare.deNLP hat das Leben von Stephan Landsiedel stark geprägt und positiv verändert. You don't, MIMIC, the only way that Roger Mark's group got permission to release that data and make it available for people like you to use is by persuading the IRB that we had done a good enough job of getting rid of all the identifying information in all of those records so that it's probably not technically impossible, but it's very difficult to figure out who the patients actually were in that cohort, in that database. No enrollment or registration. PETER SZOLOVITS: Oh, regression standardized. So that's all out there. So EMRs became more prominent in 2010. We're not going to say someone has or hasn't a disease, but we are, you know, Tianxi and I have been planning this grant where, what's outputted from this algorithm is a probability of disease. » Used with permission.). So what this means is that very simplistic techniques can actually work reasonably well at times. And it had to do with the fact that cTAKES, which is a really robust system, was just too computationally intensive. mit-nlp has 8 repositories available. We looked for lab tests, mainly RF, rheumatoid factor, and anti-cyclic citrullinated peptide, if I pronounced that correctly. MIT undergrads who are interested in doing a UROP with us should [fill this form] About us Our research encompasses all aspects of NLP research, ranging from modeling basic linguistic phenomena to designing practical text processing systems, and developing new machine learning methods. You have something like the VA MVP or UK Biobank. A special guest lectureby Nikhil Garg on how word embeddings encode stereotypes (and how this has changed over the last 100 years) Tips on working with languages other than English 3. Electrical Engineering and Computer Science Speaker: Peter Szolovits. So in 1985 or '84, the newly appointed director of the National Library of Medicine, which is one of the NIH institutes, decided to make a big investment in creating this unified medical language system, which was an attempt to take all of the terminologies that various medical professional societies had developed and unify them into a single, what they called a meta-thesaurus. Learn more », © 2001–2018
We said, well, how well could we do by, instead of looking at that codified data, looking at the narrative text in nursing notes, doctor's notes, discharge summaries, various other sources. Now, what do you use NLP for? Lecture 7: Natural Language Processing (NLP), Part 1. This is a discharge summary from MIMIC. summarize the data. OK? The technical reason is described here. I think the biggest challenge right now is the mapping. So what we did is to say, well, if you train a data set that tries to tell you whether somebody really has rheumatoid arthritis or not based on just codified data. So this seems like a fairly obvious idea, but apparently nobody had tried this on a computer before. Do you know why it says standardized? But of course, there's no pushback saying that if you warn on every second order, then no one's going to pay any attention to any of them. Not necessarily the science. But I wanted something that read like real text. And he was, in fact, able to build systems that were used by researchers in areas like anthropology, where you don't have nice coded data and where a lot of stuff is in narrative text. There are also hierarchies and relationships that are imported from all these different sources of terminology, but those are a pretty jumbled mess. So yeah, the terms are really similar in UMLS. Kat? But you see that these are a useful listing of appropriate semantic types that you can then look for in such a database. PETER SZOLOVITS: I think-- so the regression coefficients in a logistic regression are typically just odds ratios, right? They may have RA of the right wrist on one day, then on the left knee the other day. The interesting thing is that the change has happened. And then we're going to talk about what is very often done, which is a kind of term spotting approach that says, well, we may not be able to understand exactly everything that goes on in the narratives, but we can identify certain words and certain phrases that are very highly indicative that the patient has a certain disease, a certain symptom, that some particular thing was done to them. You look up in this very large database of medical terms and translate them into some kind of expression that represents what that term means. PETER SZOLOVITS: So Tuesday, I'll talk a little bit about that system and some of its successors. And then I'm going to talk about some conceptually very appealing, but practically not very feasible methods that involve analyzing these narrative texts as linguistic entities, as linguistic objects in the way that a linguist might approach them. PETER SZOLOVITS: Yeah. Well, the answer is that you get billed for, you know, every aspirin you take at the hospital. So our goal was to reduce that amount of chart review. So that's a date. We had fantasized decades ago that, you know, when you get a report from a pathologist, that somehow or other, a machine learning algorithm using natural language processing would grovel over it, identify the important things that came out, and then either incorporate that in decision support or in some kind of warning systems that drew people's attention to the important results as opposed to the unimportant ones. It's a million vets and they have EHR data going back decades. AUDIENCE: I have two questions. 10 years later, I'm at the VA and I'm interested in identifying rheumatoid arthritis. PETER SZOLOVITS: OK. And also, aggregate judgment is things like cohort selection, where it's not every single thing that you need to know about this patient. And then they mapped all those together. Lecture 7: Natural Language Processing (NLP), Part 1, Electrical Engineering and Computer Science. So what would you guess is the positive predictive value of having a billing code for rheumatoid arthritis in this data set? So we looked at 500 cases, which we got gold standard readings on. So the experiment was not repeated. And this is going to be a heterogeneous kind of presentation. You want to look for relationships between different entities that are identified in the text. How many people think it's more than 25%? PETER SZOLOVITS: Well, you know, it will surely happen at some point. So we actually have three categories-- definite, possible, and no. Is that also common across all these diseases or are there different approaches? The second Huntington is actually an important medical fact. Massachusetts Institute of Technology. And here's how it works. And we played with negation detection because, of course, if a note says the patient does not have x, then you don't want to say the patient had x because x was mentioned. So it's a completely different approach to research now. It's definitely not state of the art today. Its time, its location, its degree of certainty. Yeah. NLP related courses at MIT; Michael Collins NLP Course at Columbia. And also drug systems are different even between the US and the UK. And from that, the right thing was triggered in the computer. And what you see is that if you plot the areas under-- or if you plot the ROC curves, what you see is that training on Northwestern data and testing on either Partners or Vanderbilt data was not so good. PETER SZOLOVITS: Now, you were mentioning that when you identify such a patient, you then try to get a blood sample so that you can do genotyping on them. So these guys, Walker and Hobbs, said, well, why don't we apply this idea to natural language access to medical text? So you have some kind of system where you have the first sit down, you have to define the phenotype. Actually, you were on that revision list. And the patient is afebrile, AF. It covers syntactic, semantic, and discourse processing models, emphasizing machine learning or corpus-based methods and algorithms. License: Creative Commons BY-NC-SA. So it's kind of a measure. So roughly, the outline of these two lectures is that I want to talk a little bit about why we care about clinical text. But basically, it's a dictionary look up. AUDIENCE: So wait, for the ICD-11 you don't think take that long to exist because it's a more logical system? OK, so back to term spotting. So for example, myocardial infarction and heart attack really mean exactly the same thing. Yeah? So let me give you a little historical note. It's shortness of breath. So you're as optimistic as I was in the 1990s. Online NLP Training Plus(INLP Center) Conducted by the World’s leading NLP training institute, this … So it's not really a thesaurus because it's not completely well integrated, but it does include all of this terminology. Course description And this is the pipeline we're laying down for also the Million Veterans program, which is even bigger. And then they spent a lot of both human and machine resources in order to identify cases in which two different expressions from different terminologies really meant the same thing. There are many factors, of course, but high on the list is the ability to form and convey complex ideas with a discernible language. The focus here is on NLP. So what does that mean? There was a big study called The Informatics for Integrating Biology and the Bedside Project called i2B2 led by Zak Kohane And so that was already in place. And so the billing codes-- we've talked about this a little bit before-- but they are a very imperfect representation of reality. But we've done a lot of improvements to actually automate things a little more. That it would simply not work because there are local effects, local factors, local ways that people have of describing patients that I thought would be very different between Nashville, Chicago, and Boston. And sometimes that's not even appropriate because they may not have changed everything that needed to be changed. And this is generally known as phenotyping. But that means that it's very repetitive. If you're not very sick, you tend to have a little bit of data. So it's like, you know, you're into these two different worlds. And then you always want to have more than one reviewer. Interesting enough, this algorithm ports well over there, too. So this was from an old paper by Barrows from 19 years ago. And so there are these lexical variant generator that helps us normalize that. Yeah. And so you can look through the text and say, ah, OK, so no indicates negation and urine output is a kind of one of these concepts. Then we were getting a positive predictive value of about 88%. Because it costs so much, you didn't want to genotype someone who didn't have RA. Like how did you approach that because we talk about this in other contexts in class? NLP and Applied NLP at Berekley. Deep Learning for Natural Language Processing: Tutorials with Jupyter Notebooks . And there are some details here that you can look at in the slide, and I had a pointer to the original paper if you're interested in the details. And he published this paper in 1966 called English for the Computer in the Proceedings of the Fall Joint Computer Conference.