I was always fascinated with cloud computing and working with cloud services. That is the reason I took the Cloud Computing class while pursuing my Masters in Computer Science at New York University. Here is a demo for one of the projects I worked on using a bunch of Amazon Web Services, Node.js, and socket.io.
Architecture
Twitter Stream Listener
We have a Twitter Stream Listener backend service that listens to random samples of all public tweets and sends them to AWS SQS.
Sentiment Analyzer
Sentiment analyzer is a backend service that retrieves the tweets from SQS and runs sentiment analysis through IBM Watson’s Alchemy API. It then publishes the tweet with sentiment information to an AWS SNS topic.
Node Server
A node server subscribes to the AWS SNS topic and also serves our front-end application page. It uses socket.io to transmit the tweet and sentiment information to all the browser clients connected via web sockets.
Sequence of events
- The program listens to random sample of all public statuses using twitter streaming API
- It adds the tweets containing location information to Amazon SQS for processing
- Another thread periodically retrieves these tweets from SQS for sentiment analysis on the text of these tweets
- It uses Alchemy API for sentiment analysis
- It publishes the tweet’s text and location along with the sentiment to all the endpoints subscribed to AWS SNS topic
- Node server receives the tweet and sentiment via webhooks from AWS SNS topic and transmits it to web clients using socket.io
- The SPA (Single Page Application) on user’s browser (served from node server) connects to the node server through web sockets and updates the heat map by using tweet’s latitude/longitude and sentiment score.
Demo
Real-time Twitter sentiment heatmap