Social Media Sentiment Crawler | Web Crawling | Twitter API | Reddit

Client Background

A brand-monitoring agency wanted to track brand mentions and sentiment across Twitter, Reddit, and niche forums.

The Challenge

APIs had strict rate limits; forums required session cookies and anti-bot measures. Data needed to be enriched with sentiment scores.

Objectives

โœฆ Collect posts mentioning target keywords
โœฆ Enrich with basic sentiment analysis
โœฆ Deliver daily CSV reports

Our Approach

๐€๐๐ˆ ๐”๐ฌ๐š๐ ๐ž: Leveraged official Twitter API for tweets; fell back to scraping with Selenium for rateโ€‘limit breaches
๐…๐จ๐ซ๐ฎ๐ฆ ๐’๐œ๐ซ๐š๐ฉ๐ข๐ง๐ : Authenticated via Selenium; scraped threads with dynamic load
๐€๐ง๐š๐ฅ๐ฒ๐ฌ๐ข๐ฌ: Ran VADER sentiment on post text in Python; flagged highโ€‘impact posts
๐‘๐ž๐ฉ๐จ๐ซ๐ญ๐ข๐ง๐ : Compiled daily summaries in Pandas; output CSV

 

Results & Impact

โœฆ Gathered 30K+ mentions daily, 95% sentiment accuracy
โœฆ Automated reports cut manual monitoring by 90%
โœฆ Helped clients respond faster to negative sentiment

Tools & Technologies

Python, Tweepy, Selenium, BeautifulSoup, NLTK (VADER), Pandas

Client Testimonial

“very quick and good quality delivered”

Add your Comment