UTOPIAN MEDICATION CLASSIFICATION

SUMMARY

Medications are classified using a linear support vector classifier, trained by a family physician. The medications are grouped in 6 categories:
1. Oral antibiotics
2. Opioids (excluding methadone and suboxone)
3. Opioid addiction treatments (methadone, suboxone, naloxone, etc.)
4. Benzodiazpeines and z-drugs (Zopiclone)
5. All other medications
6. Non-medications (massage, physiotherapy, glucose test strips, etc.)

The combined average F1-score of the classifier is ~90%. This classifier will be periodically re-assessed to ensure continued accuracy.

Raw Medication Data

All medication data used for the purpose of medication classification was extracted from the UTOPIAN database on Aug 25, 2020. This includes all data which would be included in the medication section of the various EMRs.

The total number of medication entries, referred to here as "raw entries", across all UTOPIAN contributors was 12,914,380. There are some patient charts with no entries, and others with many. These raw entries are not all unique. After removing duplicates, there are 542,094 "unique entries".
Classification Objective

The goal is to "classify" these medication entries into groups, or classes, that we wish to focus on. These classes are:
1. Oral antibiotics
2. Opioids (excluding methadone and suboxone)
3. Opioid addiction treatments (methadone, suboxone, naloxone, etc.)
4. Benzodiazpeines and z-drugs (Zopiclone)
5. All other medications
6. Non-medications (massage, physiotherapy, glucose test strips, etc.)

Classification Methodology

1. Cleaning the Raw Text

The "raw text" is pulled from the EMR and de-identified by the UTOPIAN data manager. This text is then converted into "cleaned text" (removal of extra characters, spaces, and non-medication related words). The python function for this is shown below:


    def clean_medication_text(raw_text):

        temp_text = raw_text.lower()

        #exceptions
        for item in ['b12', 'b6', 'b2', 'b50', 'b100', 'b 100', 'b 12', 'b 50']:
            if item in temp_text:
                if 'tab 50' not in temp_text and 'tab 100' not in temp_text and 'tb24' not in temp_text and 'tab250' not in temp_text and 'tab 250' not in temp_text: #this eliminates the possibility that 'b 50'='tab 500 mg' I think tb24 is short for 24 tabs, as it shows up often
                    return 'vitamin b'

        if 'injection' in temp_text:
            return 'injectable treatment'

        if 'solution' in temp_text:
            for item in ['1%', '2%', '4%', '0.1%', '0.2%']:
                if item in temp_text:
                    return 'topical treatment'

        for item in [' ottic', 'ophth', 'erythro base', 'polymyxin', 'elidel', 'clobatesol', 'glaxyl', 'glaxol base', 'glaxal', 'eye oint', 'clotrimaderm crm', ' gel ', ' gel', 'lyderm', 'neostrata', 'ointment', 'topical', 'cream', 'ophthalamic', ' opth', 'ophthalmic ont', 'oph oint', 'dilusol', 'solugel', 'erythro/hc/niz', 'biobase', 'ophthalmic', 'benzoyl', ' otic', 'lotion']: #this eliminates any of the erythromycin ointments, preventing oral forms of erythromycin from getting classified as non-oral abx
            if item in temp_text:
                if 'oral solution' not in temp_text and 'powder for solution' not in temp_text:
                    return 'topical treatment'

        for item in ['vaginal', 'rectal']:
            if item in temp_text:
                return 'vaginal or rectal treatment'

        for item in ['vaccine']:
            if item in temp_text:
                return 'vaccination'

        #find the core name of the medication
        try:
            match = re.finditer('([0-9]+(\.[0-9]+)?)', temp_text)

            for item in match:
                if item.start() == 0:
                    pass
                else:
                    temp_text = temp_text[0:item.start()]
                    break

        except:
            pass

        temp_text = temp_text.replace('prescribed:', '')
        temp_text = temp_text.replace('name:', '')
        temp_text = temp_text.replace('qty', '')
        temp_text = temp_text.replace('apply', '')
        temp_text = temp_text.replace('take', '')
        temp_text = temp_text.replace('daily', '')
        temp_text = temp_text.replace('sodium', '')
        temp_text = temp_text.replace('dom-', '')
        temp_text = temp_text.replace('teva-', '')
        temp_text = temp_text.replace('jamp-', '')
        temp_text = temp_text.replace('novo-', '')
        temp_text = temp_text.replace('(', ' ')
        temp_text = temp_text.replace(')', ' ')
        temp_text = temp_text.replace('ava-', '')
        temp_text = temp_text.replace('ratio-', '')
        temp_text = temp_text.replace('pms-', '')
        temp_text = temp_text.replace('auro-', '')
        temp_text = temp_text.replace('ntp-', '')
        temp_text = temp_text.replace('gen-', '')
        temp_text = temp_text.replace('act-', '')
        temp_text = temp_text.replace('ach-', '')
        temp_text = temp_text.replace('taro-', '')
        temp_text = temp_text.replace('phl-', '')
        temp_text = temp_text.replace('apo-', '')
        temp_text = temp_text.replace('accel-', '')
        temp_text = temp_text.replace(' hcl', '')
        temp_text = temp_text.replace(' dose', '')
        temp_text = temp_text.replace(' frequency', '')
        temp_text = temp_text.replace(' amount', '')
        temp_text = temp_text.replace('current meds', '')
        temp_text = temp_text.replace('recorded ', '')
        temp_text = temp_text.replace(' -', ' ')
        temp_text = temp_text.replace('-', ' ')
        temp_text = temp_text.replace('.', ' ')
        temp_text = temp_text.replace('|', ' ')
        temp_text = temp_text.replace(',', ' ')
        temp_text = temp_text.replace('!', ' ')
        temp_text = temp_text.replace('?', ' ')
        temp_text = temp_text.replace('/', ' ')
        temp_text = temp_text.replace(':', ' ')
        temp_text = temp_text.replace(';', ' ')
        temp_text = temp_text.replace(' take', ' ')
        temp_text = temp_text.replace('[xhcnx]', ' ')
        temp_text = temp_text.replace('[xdoctorx]', ' ')
        temp_text = temp_text.replace(' as directed', ' ')
        temp_text = temp_text.lstrip(' ')
        temp_text = temp_text.rstrip(' ')
        temp_text = re.sub(' +', ' ', temp_text)


        return temp_text

This reduces the dataset from 542,094 original unique raw text entries down to 56,514 unique "cleaned text entries".
These 56,514 entries represent 541,885 of the original "raw text entries". The difference of 209 from the original number of raw text entries, is that we have excluded "raw text entries" which were longer than 150 characters. The reason for doing this is that some "medication" entries aren't actually related to medications. For instance, we see some entries as a history of the patients past health. This happens infrequently, which is why this simple filter is used to remove those types of entries.
Below is a list of the 100 most commonly appearing "cleaned text entries", with their frequencies, after the initial cleaning step. "Topical treatment" is a label applied within the cleaning step, as outlined above.


                ('topical treatment', 42416)
                ('metformin', 6695)
                ('crestor', 5325)
                ('ventolin hfa', 4704)
                ('amoxicillin', 4536)
                ('synthroid', 4066)
                ('naproxen', 3811)
                ('vitamin b', 3483)
                ('clonazepam', 3196)
                ('cipralex', 3195)
                ('injectable treatment', 3152)
                ('lipitor', 2892)
                ('hydrochlorothiazide', 2878)
                ('percocet', 2824)
                ('coversyl', 2723)
                ('nasonex', 2706)
                ('ramipril', 2651)
                ('flovent hfa', 2646)
                ('norvasc tab', 2541)
                ('suboxone', 2539)
                ('synthroid tab', 2533)
                ('gabapentin', 2525)
                ('amlodipine', 2450)
                ('diamicron mr', 2407)
                ('ativan', 2376)
                ('azithromycin', 2275)
                ('tecta', 2235)
                ('baclofen', 2177)
                ('lyrica', 2118)
                ('tylenol with codeine no', 2063)
                ('acetaminophen', 1994)
                ('apo naproxen tab', 1985)
                ('lorazepam', 1946)
                ('reactine', 1938)
                ('bisoprolol', 1921)
                ('symbicort', 1910)
                ('cymbalta', 1879)
                ('vitamin d', 1806)
                ('effexor xr', 1770)
                ('alesse', 1766)
                ('omnaris', 1733)
                ('advair', 1732)
                ('sertraline', 1692)
                ('pantoloc', 1692)
                ('pantoprazole', 1682)
                ('trazodone', 1643)
                ('methadone', 1590)
                ('flexeril', 1507)
                ('lantus', 1490)
                ('tylenol w codeine no', 1471)
                ('metoprolol', 1426)
                ('asa', 1413)
                ('betaderm crm', 1394)
                ('keflex', 1391)
                ('arthrotec', 1366)
                ('omeprazole', 1363)
                ('amitriptyline', 1360)
                ('macrobid', 1357)
                ('amoxil', 1355)
                ('atorvastatin', 1335)
                ('ranitidine', 1324)
                ('atenolol', 1272)
                ('hydromorphone', 1258)
                ('lasix', 1241)
                ('janumet', 1239)
                ('celebrex', 1223)
                ('vimovo modified release tablet', 1197)
                ('zopiclone', 1192)
                ('altace', 1186)
                ('lamisil crm', 1176)
                ('ciprodex', 1168)
                ('elavil', 1154)
                ('valtrex', 1134)
                ('vaginal or rectal treatment', 1130)
                ('ibuprofen', 1115)
                ('levothyroxine', 1108)
                ('hycodan syrup', 1094)
                ('pariet enteric coated tablet', 1070)
                ('ferrous gluconate', 1058)
                ('peg', 1049)
                ('prednisone', 1039)
                ('concerta', 1033)
                ('chlorthalidone', 1032)
                ('citalopram', 1028)
                ('lactulose', 1013)
                ('diclectin', 1011)
                ('wellbutrin xl', 981)
                ('norvasc', 980)
                ('cipro', 972)
                ('eltroxin', 963)
                ('nexium', 954)
                ('celexa', 951)
                ('losec', 932)
                ('nortriptyline', 925)
                ('cialis', 916)
                ('warfarin', 900)
                ('candesartan', 886)
                ('viagra', 879)
                ('actonel', 877)

2. Finding Similar Names

Next, the number of entries is further reduced by looking for similarities in the medication names. For instance, naproxen and naprosyn, are two common ways of saying the same medication and have similar spellings. To condense the number of unique entries further, we use a measure of word similarity called Jaro Distance, which measures the number of characters that are similar and how far apart those characters are in the word or sentence.
A two step approach is taken here:
1) for words included in the following list, an 80% threshold is used to determine equality: ['physiotherapy', 'massage', 'chiropractor', 'orthotic', 'compression stockings', 'blood glucose', 'discontinue']
2) for all other "cleaned text entries" from the 56,514 entries, a threshold of 90% is used to determine equality.
After using this method, the number of "unique text entries" is reduced from 56,514 down to 7,108.
The result of these two steps is that we have reduced the amount of work it takes to classify 542,094 "unique raw text entries" down to only 7,108.

3. "Labeling" the Data

This is the most important step. Here, a qualified professional with strong knowledge of medications and non-medications must accurately state which "classification" each of the 7,108 entries receives.
It is important to note at this point that the goal of this process is to label a subset of the data and then "train a classifier" to take over the task once enough data has been manually classified.
For the initial round of classification, Dr. Adam Cadotte, a family physician, who is also the creator of this analysis pipeline, has labeled the data.
The process for doing so took place in several steps. Initially, the 7,108 entries were prioritized in order of their frequency. For instance, using the list above, we started at the top and classified each medication / non-medication until a large portion of the initial entries were classified.
To illustrate how this simplifies the task, classifying the 100 entries shown above, represents 222,705, or 41% of all original medication entries.
Very quickly though, the frequency of the cleaned text words falls and we are left with classifying entries which are less common and the task becomes more challenging.
It becomes important at this time to do some "training" to figure out which entries to focus on. For instance, by the time we classify the initial 100 entries, we have many examples of antibiotics and "medications" (as opposed to non-medications), but not a lot of samples of non-medications.
The following is a summary of how many items have been labeled as of the time of this writing:


            Total number labeled: 3427
            Number labeled as oral antibiotic: 662
            Number labeled as opioid: 252
            Number labeled as opioid addiction treatment: 12
            Number labeled as z-drug: 105
            Number labeled as non-medication: 837

4. Training the Classifier

This is where machine learning comes into play. We want to train an automated classifier to learn how to "label" the data for us, just as a trained professional would.
This can be done using a variety of methods and classifiers. For this project, we used a package called "scikit-learn" which is developed for Python, https://scikit-learn.org/stable/
For data training purposes, we used a Linear Support Vector classifier with the following pipeline in scikit-learn:


            model = Pipeline([('vect', CountVectorizer()),
                         ('tfidf', TfidfTransformer()),
                         ('clf', LinearSVC()),
                         ])

This generates a classification model based on data that has been labeled by a professional.
We then use this classified data to predict the label for entries which haven't been manually labeled. Using this method, we can look for unlabeled entries that we want to focus on.
For instance, if we feel there aren't enough non-medication entries labeled, we simply use the classifier to tell us which "cleaned text entries" it thinks are non-medications and then we focus on classifiying those entries. Sometimes the automated classification is correct, and other times it isn't, but as we label more data it becomes more accurate.
Using this iterative approach helps to speed the labeling process and focus where how data is needed.

5. How Much Data Do We Need?

This is were we need to figure out how well the trained classifier does when compared to a professional. Several metrics can be used to make this comparison, including accuracy, precision, recall, and F1. These are all commonly used in the machine learning field to judge the ability of a classifier to perform the intended task. The F1 score, is explained on the scikit-learn website as follows: "The F1 score can be interpreted as a weighted average of the precision and recall, where an F1 score reaches its best value at 1 and worst score at 0. The relative contribution of precision and recall to the F1 score are equal."
To determine these metrics we take all of the data that has been manually labeled, and split it into two groups, "training" and "test". For this analysis we made the test group 20% of the total data and the training group 80%.
This means 80% of the labeled data was used to "train" the classifier and the remaining 20% is used to "test" the classifier. The "test" group acts as the ground truth data.
The precision, recall and F1 score have been calculated for each of the labels and are shown below. The is based on taking the average score from 20 iterations, shuffling the train / test groups each time.


                 BENZO
                precision Min/Avg/Max: 91% / 97% / 100%
                recall Min/Avg/Max: 75% / 91% / 100%
                f1-score Min/Avg/Max: 82% / 94% / 100%

                 NOTMED
                precision Min/Avg/Max: 92% / 95% / 99%
                recall Min/Avg/Max: 88% / 91% / 94%
                f1-score Min/Avg/Max: 91% / 93% / 96%

                 OTHERMED
                precision Min/Avg/Max: 82% / 85% / 88%
                recall Min/Avg/Max: 92% / 94% / 97%
                f1-score Min/Avg/Max: 88% / 89% / 91%

                 OPIOID_TX
                precision Min/Avg/Max: 0% / 88% / 100%
                recall Min/Avg/Max: 0% / 63% / 100%
                f1-score Min/Avg/Max: 0% / 69% / 100%

                 OPIOID
                precision Min/Avg/Max: 81% / 91% / 98%
                recall Min/Avg/Max: 84% / 91% / 100%
                f1-score Min/Avg/Max: 86% / 91% / 96%

                 ABX
                precision Min/Avg/Max: 91% / 93% / 97%
                recall Min/Avg/Max: 71% / 76% / 83%
                f1-score Min/Avg/Max: 81% / 84% / 88%

The score of the classifier as a whole is below:


            Accuracy Min/Avg/Max: 88% / 89% / 92%
            Precision Min/Avg/Max: 88% / 90% / 92%
            Recall Min/Avg/Max: 88% / 89% / 92%
            F1 Min/Avg/Max: 87% / 89% / 92%

In terms of scoring for machine learning based classifiers, it is generally accepted that any score above 80-85% is considered a good score, but needs to be taken in context, depending on the importance of what the data is being used for. For instance, you want very high probabilities in mission critical tasks, but may be tolerant of lower scores for less important tasks. In any case, average values of close to 90% are very good.
We calculated these scores for each of the classes to determine where more data is needed. If there is a low score for a specific class, we label more data that we predict to be in that class. This improves the data in two ways: 1) if the classifier is wrong about an entry, we end up labeling the data properly and it gets removed from that classification; and 2) if the classifier is correct, it strengthens the relationship.
At the time of writing this methodology, 3,427 of the 7,108 entries were manually classified.
As we decide to look at additional medication classes or want to improve on the accuracy further, we simply need more professional input to classify additional raw data from the EMR.

Use of the Trained Classifier

Even though we've labeled only a portion of the original data, we use this information to infer the "label" of the remaining data. The classifier does this automatically, and with an average accuracy and F1 score of ~90%.
This can interpreted as the medication data you see on your report as being 90% accurate. It will never be perfect, unless we put a lot of professional resource time into ensuring every single data point is labeled properly. This isn't feasible. We believe that using this classification method should suffice for these purposes, and can be refined as needed.
Comparison to the UTOPIAN ATC Classification

The current UTOPIAN process uses ATC codes as a classification system.
To compare the ATC classifications with the classifications proposed here, referred to as the "ML approach" (machine learning approach), we will compare the antibiotic and opioid classes.
Looking at the original 542,094 entries, 33,791 of them are classified as antibiotics with an ATC code of J01 (Antibacterials for systemic use), J04 (Antimycobacterials), or G01 (Gynecological antiinfectives and antiseptics). Of the 33,791, the ML approach classifies 30,373 of them as oral antibiotics, meaning there is 89.9% agreement. There are 722 classified as antibiotics by ATC, but not the ML approach. These are listed below.
There are also 2,491 which were classified as antibiotics by the ML approach, but not the ATC approach.
The main reason there are differences between these classification methodologies is that the ML approach focuses on classifying "oral antibiotics" rather than all antibiotics (topical, otic, injectable, etc). The ATC codes also include some anti-fungals (e.g., clotrimazole) which are excluded in the ML approach.

MEDICATIONS CLASSIFIED AS ANTIBIOTICS BY ATC, BUT NOT THE ML APPROACH (3,418)
MEDICATIONS CLASSIFIED AS ANTIBIOTICS BY THE ML APPROACH, BUT NOT BY ATC (2,269)

SUMMARY

Raw Medication Data

Classification Objective