ESC
Sure, the police can read your emails,
but what do you have to hide?
That was rhetorical. We already know.

DIRTopic (ditm) Topic Model

I undertook this project during my internship in 2026 at the very company that I had left before beginning the HLT program, at which time the company had no inkling of undertaking any projects relating to NLP or language modeling. I had maintained good relations with the company, who was eager to bring me in to serve as the primary NLP engineer for the project.

Goals

The goal of the project was to identify the salient topics among a timestamped corpus and to identify changes in the topic prominence over time. The output is presented to a portfolio manager to assess any rises or falls of such topic prominence such that they might inform approaches to investment decisions.

Thus the base requirements of the project were:

Additionally, the project needed to be:

Approach

At the time that I joined the project, the state-of-the-art BERTopic (Grootendorst, 2022) had already been selected as the basis of the model. While BERTopic itself had been unfamiliar to me at the time that I began, my prior experience with vector clustering using DBSCAN—in a project that grew as part of LING 582 (Advanced Statistical NLP)—lended itself directly to the clustering component of BERTopic.

I inherited a basic outline of such a model, although I still needed to:

While I was the expert in the room—so to speak—on language modeling, I met regularly with team members and stakeholders (daily and weekly, respectively) to consult on the progress of the project.

Corpus

At the time that I began, it was known that the production model would be fed from a database holding the corpus. It was also expected that the corpus would be expanded to different sources; the original corpus source considered was composed of company-internal emails, but there were discussions of also incorporating meeting transcripts. At the onset, however, the data available was a limited sample of emails in CSV format.

The course LING 508 (Computational Techniques for Linguists) in particular was instrumental, as it equipped me to understand how to handle the needed separation of concerns in the design.

I chose a composite-leaf design pattern for the corpus in order to allow for drop-in replacements of corpus sources as well as allowing for additional corpora to be included in the composite without affecting the API treatment of the corpus as a whole.

Each Corpus, then, has components that can be similarly-readily changed:

The Corpus contains a check to ensure the Loader and Preprocessor match for content type.

classDiagram
direction LR


namespace Loaders {

    class Loader {
        <<abstract class>>
        load()
    }

    class CSVLoader {
        path :: str
    }

    class DbLoader {
        db_url
        engine
        table
        tablecols
    }

    class EmailDbLoader {
    }

    class TranscriptDbLoader {
    }

}

namespace Corpora {

    class CorpusComponent {
        <<abstract class>>
        documents
        documents_untokenized
        timestamps
        load()
        build_stopwords()
        preprocess_docs()
        frame()
    }

    class Corpus {
        loader :: Loader
        preprocessor :: Preprocessor
        min_tokens :: int
    }

    class CorpusComposite {
        corpora :: list[CorpusComponent]
    }

}

namespace Preprocessing {

    class Preprocessor {
        <<abstract class>>
        clean_text()
        lemmatize()
    }

    class EmailPreprocessor {
    }

    class TranscriptPreprocessor {
    }

}

CorpusComponent <|-- Corpus
CorpusComponent <|-- CorpusComposite
CorpusComposite o-- "1..*" CorpusComponent

Corpus *-- Loader
Corpus *-- Preprocessor

Loader <|-- CSVLoader
Loader <|-- DbLoader
DbLoader <|-- EmailDbLoader
DbLoader <|-- TranscriptDbLoader

Preprocessor <|-- EmailPreprocessor
Preprocessor <|-- TranscriptPreprocessor

A more exhaustive UML diagram is provided at the bottom of this page.

Preprocessing

One function of Preprocessor is to clean corpus documents of various formatting and uninformative text. This is achieved primarily through regex, and I inherited much of the email-cleaning regex at the time that I began, though I later amended it. Once transcripts were introduced, we were fortunate in that they are consistently formatted, which made it straightforward to strip it of text signalling interlocutors and of the moderator’s sections, which are uninformative.

The Preprocessor also holds a SpaCy pipeline—used for tokenization and lemmatization—as a class attribute of the parent class (thus only one SpaCy model is loaded).

Preprocessor.lemmatize() is defined in the parent class to ensure that the same process is applied to all corpora. In addition to splitting documents on sentences, tokenizing, lemmatizing, and excluding stop words and select NER tags, a filter of custom stop words is applied, which is defined in .nlp_filters of the module.

There were attempts, prior to my start, to build stop words from email senders and recipients, although this added a lot of complexity and was likely redundant with excluding PERSON NER tags; thus it was abandoned. The custom stop word list remained to mitigate topics forming of common and uninformative words.

At my start it had also been decided, somewhat arbitrarily, to exclude ORG and GPE NER tags. While my intuition was that these would be informative for the model’s purpose, the filter remained in place until late in development. After consulting with the stakeholder and testing the output, I removed this exclusion.

Given that the use of SpaCy’s model was not dependent on ites pretrained vectors, en_core_web_sm is used. Motivated by computational performance, parser is excluded and replaced with sentencizer:

    NLP = spacy.load(config.SPACY_MODEL, exclude = [ "parser" ])
    NLP.add_pipe("sentencizer")

After lemmatization, the Corpus also has a min_tokens parameter, which filters documents below a certain threshold, thereby excluding very short documents whose vectors may be saturated by a single token (when set to 2). Under consideration that emails tend to be comprised of short messages, this was kept low (2). The transcripts’ min_tokens was set a bit higher, both due to consideration that spoken sentences in a formal setting would be less constrained and due to the transcripts corpus being far larger than anticipated, which presented memory contraints (setting this from 2 to 4 removed about 13% of documents without inviting concern for a loss in meaningful documents).

BERTopic

My very first task—even before officially starting the internship role—was to become familiar with BERTopic. In was in this capacity that I was uniquely qualified to manage this project; that is, through the course of the HLT program, I had become sufficiently familar with underlying concepts related to those used in BERTopic (DBSCAN, vector embeddings, transformer embeddings, machine learning, Tf-Idf) to handle the model in an informed manner.

The bertopic Python package is built with a number of defaults that can be used “out of the box,” with the caveat that even some defaults (namely UMAP dimensionality-reduction) are prepared in advance and fed to the model; this separation was held, whereas the modular components used were mostly the BERTopic defaults, while the specific hyperparameters were calibrated through testing and assessing the results via mlflow.

Below, I discuss the more notable components and the considerations made while calibrating them.

Sentence embeddings

The BERTopic default for generating sentence embeddings is SentenceTransformers with the model all-MiniLM-L6-v2. While I saw no impetus to deviate from this, it was a deliberate decision to feed the embedding model the lemmatized form of the documents rather than the unaltered text.

The reason for this was empirical. While I had seen it suggested that a transformer embedding model, being trained on natural language, would more accurately represent the meaning of the original documents, testing revealed that the output of the BERTopic model showed clusters that were more focused on sentiment than content. By contrast, using lemmatized documents resulted in clear content labels. Thus, we used the latter because it was more suitable to our needs.

SentenceTransformers also served to help motivate the decision to split the corpus sources into sentence-level documents before embedding and clustering. This was an early consideration that I had, based on concerns about inconsistent corpus-document size and about the semantics of long documents being semantically diluted.

The model all-MiniLM-L6-v2 has a maximum input token length of 256 tokens, where any tokens beyond that would have been silently discarded and lost from the model. This would have been especially problematic in the Transcripts corpus, where file sizes vastly exceed this. Thus, all documents in the model were split on sentences. A natural consequence of this is a much greater number of document vectors.

UMAP

UMAP (McInnes et al., 2020) is the default algorithm of dimensionality-reduction in BERTopic, which is necessary because HDBSCAN is adversely affected by the curse of dimensionality, familiar to me from my exposure thereto in INFO 521 (Machine Learning), and all-MiniLM-L6-v2 outputs vectors of dimensionality 384.

We settled on UMAP’s hyperparameters via experimentation and subjective evaluation with mlflfow.

One noteworthy change was to set UMAP’s default spectral initialization to PCA. This was motivated by memory limitations after introducing the sizable transcripts corpus, leading to UMAP’s spectral eigen-decomposition met memory limits. Using PCA initialization avoided this without presenting concerns about model performance.

HDBSCAN

The clustering algorithm is the means by which documents are grouped into semantically-similar clusters, and is thus how topic groups are identified. BERTopic uses HDBSCAN (Malzer and Baum, 2020) by default, due to its ability to handle clusters of varying densities.

At the onset, clustering presented a major computational bottleneck; a single run—on the original small corpus sample without splitting documents on sentences— took well over 24 hours on our hardware. This bottleneck was enirely eliminated by setting HDBSCAN’s prediction_data and BERTopic’s calculate_probabilities (which is dependent on the former) to False. These were originally included as motivated by the stakeholder’s curiosity, but unneeded for the final model.

The primary parameters for calibrating HDBSCAN were:

Topic Labels

An innovation of BERTopic is c-Tf-Idf, a modification of Tf-Idf that treats clusters (topics) as individual documents, thus identifying the words that most characterize those topics in relatoin to others.

The only modification to defaults here was to set reduce_frequent_words, which, as the name implies, lessens the prominence of words that appear commonly throughout the corpus but are not stop words. This had a favorable affect on the labels generated.

Similarly, many labels ended up with some redundancy. Adding Maximal Marginal Relevance as a Representation Model helped to reduce such redundancy, though the effect was not as apparent as reduce_frequent_words. A low-stakes change, different diversity values were not deeply explored.

Visualization

Clusters

In early stages of the project, the 2-dimensional plot provided by BERTopic.visualize_documents() was useful in demonstrating to the stakeholder the effects of HDBSCAN clustering. As the project progressed and these effect were already clear, the computational and time cost of an additional UMAP reduction to 2 dimensions outweighed any benefit. This became especially true after the time visualization, discussed below, was introduced. As such, the 2d plot was removed and is not included here.

A simple bar plot, meanwhile, conveys much of the same information in an arguably more-straightforward representation, with minimal cost.

Below is an example of the topic-size distribution from a late test model. For confidentiality, topic labels have been removed from the tooltip.

Topics over time

Aside from the topic model itself, the project had the goal of presenting how these topics become more or less prominent over time.

While BERTopic includes a topics_over_time visualization, it is not well-suited to this project. It primarily concerns how the words used within a given topic may change over time, which is not a concern here. Meanwhile, the included visualization plots raw frequency, which is problematic because large topics dominate.

Much to my begrudging chagrin—but at the encouragement of my employer— the superpowered search-engine known as Claude produced a suggestion for an alternative: to borrow from (Monroe et al., 2008) to use a log-odds ratio with an informative Dirichlet prior. I thus reviewed the paper and the appropriateness of the application of this method to the goals of the project. Claude produced this calculation and the visualization.

While the Fightin’ Words paper concerned identifying words’ prominence between separate corpora, this adaptation of that method compares a topic’s share in a period to its own baseline of k periods. Comparing a topic’s share rather than raw counts helps to mitigate the effect of quiet periods, while the comparison of a topic to itself helps to represent that topic’s own change.

Because the share is relative to the entire corpus in the same period, drops in other topics could result in an apparent increase in a given topic, and vice-versa.

The Monroe et al. calculation produces a z-score for each topic’s share relative to its baseline. To determine which topics are flagged, ditm.viz.topic_changes (which stands alone as a product of Claude) sets a threshold by generating n_perm random shuffles of topic labels and keeping the highest z per permutation, setting the threshold at the level quantile of those highest random scores.

Then those flagged topic cells are plotted according to the strength of their log-odds ratio (in either direction; red indicates a rise and blue indicates a fall), resulting in a plot representing topic prominence over time.

Below is a sample timescale visualization. For compactness, only the top 20 topic changes are shown. For confidentiality, topic labels have been removed; however, I am at liberty to highlight a topic whose prominence was expected and served as an indication of the visualization working as expected: The top rows, seeing a sharp increase in prominence in early 2020, represent topics relating to the pandemic, vaccination, and remote work.

Other Challenges

Computational Hurdles

Computational limitations characterized the main difficulties during the course of the project, though this did not prove insurmountable.

Difficulties with memory arose when expanding the corpus with more source documents

The discovery of setting a low min_samples was a happy accident while trying to force errors to validate error-handling.

Future of the Project

The state of the project at the time of the conclusion of my internship

Streamlit

classDiagram
direction LR

namespace Microsoft {

    class MSExchange {
        User[*]
    }

    class User {
       Inbox
    }

    class MSGraph {
        export_email()
    }

}

namespace Database {

    class SQLServerDatabase {
    }
    class AzureDatabase {
    }

}

namespace Loaders {

    class Loader {
        <<abstract class>>
        load(datewindow)
    }

    class CSVLoader {
        path :: str
    }

    class DbLoader {
        db_url
        engine
        table
        tablecols
    }

    class EmailDbLoader {
    }

    class TranscriptDbLoader {
    }

}

namespace Corpora {

    class CorpusComponent {
        <<abstract class>>
        documents
        documents_untokenized
        timestamps
        load()
        build_stopwords()
        preprocess_docs()
        frame()
    }

    class Corpus {
        loader :: Loader
        preprocessor :: Preprocessor
        datewindow :: DateWindow
        min_tokens :: int
    }

    class CorpusComposite {
        corpora :: list[CorpusComponent]
    }

    class DateWindow {
        <<frozen dataclass>>
        start
        end
        data_start
    }

}

namespace Preprocessing {

    class Preprocessor {
        <<abstract class>>
        clean_text()
        lemmatize()
    }

    class EmailPreprocessor {
    }

    class TranscriptPreprocessor {
    }

}

CorpusComponent <|-- Corpus
CorpusComponent <|-- CorpusComposite
CorpusComposite o-- "1..*" CorpusComponent

Corpus *-- Loader
Corpus *-- Preprocessor
Corpus --> DateWindow
Loader ..> DateWindow : load(datewindow)

Loader <|-- CSVLoader
Loader <|-- DbLoader
DbLoader <|-- EmailDbLoader
DbLoader <|-- TranscriptDbLoader

Preprocessor <|-- EmailPreprocessor
Preprocessor <|-- TranscriptPreprocessor

AzureDatabase -- DbLoader
SQLServerDatabase -- DbLoader

MSExchange o-- "*" User
MSGraph -- MSExchange
MSGraph -- AzureDatabase
MSGraph -- SQLServerDatabase


namespace Model {

    class BERTopic {
        <<bertopic.BERTopic>>
        fit_transform(documents, embeddings)
    }

    class VectorReduction {
        <<umap.UMAP>>
        fit_transform(embeddings)
    }

    class NoOpReduction {
        <<BaseDimensionalityReduction>>
    }

    class Clustering {
        <<hdbscan.HDBSCAN>>
        min_cluster_size
        min_samples
        cluster_selection_epsilon
        cluster_selection_method
    }

    class EmbeddingModel {
        <<sentence_transformers.SentenceTransformer>>
        "all-MiniLM-L6-v2"
        encode(docs)
    }

    class Weighting {
        <<ClassTfidfTransformer>>
        c-TF-IDF
    }

    class Tokenizer {
        <<sklearn.CountVectorizer>>
        ngram_range
        min_df
        max_df
    }

    class Representation {
        <<MaximalMarginalRelevance>>
        diversity = 0.2
    }

}

BERTopic o-- NoOpReduction
BERTopic o-- EmbeddingModel
BERTopic o-- Clustering
BERTopic o-- Weighting
BERTopic o-- Tokenizer
BERTopic o-- Representation

namespace params {
    class CorpusSpec {
        <<frozen dataclass>>
        years
        baseline_quarters
        min_tokens_email
        min_tokens_transcript
    }
    class UmapParams {
        <<frozen dataclass>>
        n_neighbors
        n_components
        min_dist
        metric
        random_state
        init
        verbose
    }
    class HdbscanParams {
        <<frozen dataclass>>
        min_cluster_size
        min_samples
        cluster_selection_epsilon
        cluster_selection_method
        prediction_data
        __post_init__()
    }
    class VectorizerParams {
        <<frozen dataclass>>
        ngram_range
        min_df
        max_df
    }
}

CorpusSpec ..> DateWindow : datewindow(display_years)
CorpusSpec ..> Corpus : min_tokens
UmapParams -- VectorReduction
HdbscanParams -- Clustering
VectorizerParams -- Tokenizer

namespace Pipeline {

    class pipeline {
        <<ditm.pipeline>>
        pipeline(corpus_spec, umap_params, hdbscan_params, vectorizer_params, cache_dir)
        returns metrics, artifactpaths
    }

    class Cache {
        <<ditm.util>>
        get_or_make(path, make_fn, save_fn, load_fn)
        make_runhash(parts)
        timeit()
    }

    class NlpFilters {
        <<ditm.nlp_filters>>
        STOPWORDS
        NER
    }

    class viz {
        <<ditm.viz>>
        topic_changes()
        topic_sizes()
    }

    class TopicChanges {
        z
        ratio
        threshold
        heatmap()
    }

    class flow {
        <<ditm.flow>>
        sweeps
        one MLflow run per combination
        failed runs tagged and skipped
    }

    class main {
        <<ditm.__main__>>
        default specs
        one MLflow run
        failures tagged then re-raised
    }

    class MLflow {
        log_params()
        log_metrics()
        log_artifact()
    }

}

pipeline ..> CorpusSpec
pipeline ..> UmapParams
pipeline ..> HdbscanParams
pipeline ..> VectorizerParams
pipeline ..> CorpusComposite : documents, timestamps
pipeline ..> EmbeddingModel : encode()
pipeline ..> VectorReduction : reduce embeddings
pipeline ..> BERTopic : fit_transform()
pipeline ..> Cache : cached corpus, embeddings, UMAP
pipeline ..> NlpFilters : hashed into cache keys
pipeline ..> viz
viz --> TopicChanges
flow ..> pipeline
flow ..> MLflow
flow ..> Cache : make_runhash
flow ..> NlpFilters : hashed into run name
main ..> pipeline
main ..> MLflow

References

  1. 2022
    Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure.
    arXiv Link 
    @misc{grootendorst2022,
  title = {BERTopic: Neural topic modeling with a class-based TF-IDF procedure},
  author = {Grootendorst, Maarten},
  year = {2022},
  eprint = {2203.05794},
  archiveprefix = {arXiv},
  primaryclass = {cs.CL},
  url = {https://arxiv.org/abs/2203.05794},
  arxiv = {arxiv:2203.05794}
}
  2. 2020
    Claudia Malzer and Marcus Baum. 2020. A Hybrid Approach To Hierarchical Density-based Cluster Selection. In 2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI), pages 223–228. IEEE.
    arXiv DOI Link 
    @inproceedings{malzer2020,
  title = {A Hybrid Approach To Hierarchical Density-based Cluster Selection},
  url = {http://dx.doi.org/10.1109/MFI49285.2020.9235263},
  doi = {10.1109/mfi49285.2020.9235263},
  booktitle = {2020 IEEE International Conference on Multisensor Fusion and Integration for Intelligent Systems (MFI)},
  publisher = {IEEE},
  author = {Malzer, Claudia and Baum, Marcus},
  year = {2020},
  month = sep,
  pages = {223–228},
  arxiv = {arxiv:1911.02282}
}
  3. 2020
    Leland McInnes, John Healy, and James Melville. 2020. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.
    arXiv Link 
    @misc{mhm2020,
  title = {UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction},
  author = {McInnes, Leland and Healy, John and Melville, James},
  year = {2020},
  eprint = {1802.03426},
  archiveprefix = {arXiv},
  primaryclass = {stat.ML},
  url = {https://arxiv.org/abs/1802.03426},
  arxiv = {arxiv:1802.03426}
}
  4. 2008
    Burt L. Monroe, Michael Colaresi, and Kevin M. Quinn. 2008. Fightin’ Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict. Political Analysis, 16:372–403.
    DOI Link 
    @article{mcq2008,
  title = {Fightin' Words: Lexical Feature Selection and Evaluation for Identifying the Content of Political Conflict},
  author = {Monroe, Burt L. and Colaresi, Michael and Quinn, Kevin M.},
  journal = {Political Analysis},
  year = {2008},
  volume = {16},
  pages = {372 - 403},
  url = {https://api.semanticscholar.org/CorpusID:9792267},
  doi = {10.1093/pan/mpn018}
}