Techniques for Chat Data Analytics with Python | by Robin von Malottki | Nov, 2024

Half II: Subject Extraction with BERTopic

Within the first a part of this collection, I launched you to my artificially created pal John, who was good sufficient to supply us together with his chats with 5 of the closest folks in his life. We used simply the metadata, equivalent to who despatched messages at what time, to visualise when John met his girlfriend, when he had fights with one in every of his finest pals and which members of the family he ought to write to extra typically. In case you didn’t learn the primary a part of the collection, yow will discover it here.

What we didn’t cowl but however we are going to dive deeper into now’s an evaluation of precise messages. Subsequently, we are going to use the chat between John and Maria to determine the subjects they talk about. And naturally, we won’t undergo the messages one after the other and classify them — no, we are going to use the Python library BERTopic to extract the subjects that the chats revolve round.

What’s BERTopic?

BERTopic is a subject modeling method launched by Maarten Grootendorst that makes use of transformer-based embeddings, particularly BERT embeddings, to generate coherent and interpretable subjects from massive collections of paperwork. It was designed to beat the restrictions of conventional subject modeling approaches like LDA (Latent Dirichlet Allocation), which frequently battle to deal with quick…