From Webscraper to Wordcloud

A project undertaken in Belgium involved scraping comments from articles on the Belgian newspaper Het Laatste Nieuws to understand public engagement with the news. The process navigated challenges such as bypassing cookie consent barriers and adapting to Ajax loading of comments, requiring regex patterns to access all comments. The project utilized Python libraries like BeautifulSoup, Requests, and SpaCy for data extraction and NLP tasks, and WordCloud for visualizing word frequency. The SpaCy library helped categorize words, although the accuracy was initially limited, prompting plans to improve the model for Belgian dialects. The project aimed to create a comprehensive understanding of public discourse through data visualization and sought future enhancements like improved tagging, object-oriented restructuring, and sentiment analysis.