Classifying Smoking Status from Unstructured Clinical Notes

At a glance

Overview

This project tackles a well-known clinical NLP benchmark task: given a discharge summary written in free text, automatically determine the patient's smoking status. This is the kind of information that lives buried in narrative clinical notes rather than a structured field, and being able to extract it automatically has real value for research cohort building, quality reporting, and risk stratification. The project is worth including in a portfolio not because the final model was strong — it was not — but because it demonstrates a full, methodologically sound clinical NLP pipeline, from raw XML to a fine-tuned transformer model, and because it surfaces and correctly diagnoses a real and common failure mode: severe class imbalance in a small clinical dataset.

The data consisted of annotated discharge notes in XML format, labeled into five smoking status categories: current smoker, past smoker, smoker, non-smoker, and unknown. After parsing, the training set contained 398 labeled notes and the labeled test set contained 104 notes, with a highly skewed label distribution. The "Unknown" category alone made up roughly 63% of the training data, and the rarest category ("Smoker") had only 9 training examples. This imbalance, not the modeling technique, turned out to be the central challenge of the whole project.

Raw XML files were parsed with xmltodict into a structured patient ID, note text, label table. No missing values were found in any of the three files (train, labeled test, unlabeled test), and an early class distribution check immediately flagged the imbalance described above. Notes were tokenized with NLTK, then converted into two complementary feature sets that were concatenated into a single feature matrix: bag of words (CountVectorizer with English stopword removal, unigrams and bigrams, capped at 5,000 features) and custom keyword features (binary indicators for domain-specific smoking-related terms pulled directly from the note text). All features were standardized with StandardScaler before modeling.

Because the raw class distribution would have caused any model to simply predict "Unknown" for everything, the team applied ADASYN (Adaptive Synthetic Sampling, a variant of SMOTE that focuses oversampling effort on the minority classes that are hardest to classify) to the four non-Unknown classes, bringing each of the five classes to a roughly comparable size (120 to 135 synthetic and real examples per class) before model training.

Three classifiers — Random Forest, Logistic Regression, and SVM — were each tuned via grid search over their key hyperparameters and evaluated on the untouched, still imbalanced test set. As an extension, the team fine-tuned Bio_ClinicalBERT on the same classification task using Hugging Face's Trainer API, to test whether a pretrained clinical language model could pick up signal that bag-of-words features missed.

Random Forest was the strongest performer at 0.66 test accuracy, correctly separating "Unknown" (0.97 recall) and "Non-smoker" (0.75 recall) reasonably well, but it still achieved 0% recall on "Smoker" and only 9% recall on "Past smoker" — the two smallest classes even after oversampling. SVM and the fine-tuned ClinicalBERT model both collapsed entirely to predicting the majority "Unknown" class for every single test note, a classic signature of a model that has not found usable signal in a dataset this small and this imbalanced. The pipeline that produced these numbers was checked and ran correctly end to end.

With only 9 training examples for the "Smoker" class and 398 notes total, no realistic amount of hyperparameter tuning was going to produce strong performance on the rarest classes, and a 12-layer transformer model is more data-hungry than the classical baselines, not less. ADASYN can synthesize additional feature vectors for a minority class, but it cannot synthesize additional clinical narrative diversity, so it partially masks the underlying data scarcity problem rather than solving it.

This project demonstrates a complete, technically correct clinical NLP pipeline: XML parsing, tokenization, hybrid feature engineering, minority class oversampling, grid-searched classical ML, and transformer fine-tuning. It shows correct diagnosis of a common real-world failure mode (severe class imbalance in a small clinical dataset) rather than reporting an inflated accuracy number without context. It also shows practical judgment about when a more sophisticated model is not automatically the better choice.

I would present this project around the pipeline and the diagnosis: here is a methodologically sound approach to a real clinical NLP task, here is what happens when you run it against a small, heavily imbalanced dataset (398 training notes, a 9-example minority class), and here is the correct read on why the more sophisticated model did not help. That is a more useful thing to show a reviewer than a cherry-picked good accuracy score, and it is the honest state of the project as it stands. The clear next step would be sourcing a substantially larger labeled corpus before the ClinicalBERT fine-tuning step has a realistic chance of outperforming the classical baselines.

Highlights

• Complete clinical NLP pipeline: XML parsing, NLTK tokenization, hybrid BoW + domain keyword features, ADASYN rebalancing, grid-searched classical ML, and ClinicalBERT fine-tuning
• Correctly diagnosed severe class imbalance (63% 'Unknown', 9-example 'Smoker' class) as the central failure mode rather than inflating accuracy
• Random Forest reached 0.66 test accuracy; SVM and ClinicalBERT collapsed to majority-class prediction, confirming data scarcity rather than code error
• Practical judgment that a 12-layer transformer requires more data, not less, to outperform simpler baselines