Monthly Archives: June 2013

Use StreamInsight to Assign Sentiment Scores to a List of Items in a CSV (Part Two: Assign Sentiment)

In part one of this series, we discussed how to import a CSV file and then read from it using StreamInsight.  This post will take that a step further and show how to use the Sentiment140 API to assign sentiment to a string.  Within the ForEach(activeInterval) loop there are two calls that are crucial to assigning sentiment(below).

//Score the current SocialBlurb’s content attribute.
var result = sentiment.Analyze( activeInterval.l.CONTENT);
//Set the active SocialBlob’s sentiment140_mood to the sentiment received.
activeInterval.l.Sentiment140_Mood = (int) result.Mood;

Below is the Sentiment140 class that these calls relate to.  The analyze function takes the content, appends it to a URL, makes/receives a HTTP call, parses the result, and returns the sentiment score.

public class Sentiment140
private string _jsonURL = @””;
public SentimentAnalysisResult Analyze(string textToAnalyze)
//Format url to include the text we want to analyze
          string url = string.Format(“{0}?text={1}”, this._jsonURL,
HttpUtility.UrlEncode(textToAnalyze, System.Text.Encoding.UTF8));
//Maximum url length check and set to neutral if larger than allowed
if (url.Length > 600)
SentimentAnalysisResult results = new SentimentAnalysisResult() { Mood = SentimentScore.Neutral, Probability = 100 };
return results;

//Create the HTTP Request
var request = HttpWebRequest.Create(url);
SentimentAnalysisResult result = new SentimentAnalysisResult() { Mood = SentimentScore.Neutral, Probability = 100 };
//Get the Response from Sentiment140
var response = request.GetResponse();
using (var streamReader = new StreamReader(response.GetResponseStream()))
// Read from source
var line = streamReader.ReadLine();
// Parse
var jObject = JObject.Parse(line);
int polarity = jObject.SelectToken(“results”, true).SelectToken(“polarity”, true).Value<int>();
switch (polarity)
case 0:
result.Mood = SentimentScore.Negative;
case 4:
result.Mood = SentimentScore.Positive;
default: // 2 or others
result.Mood = SentimentScore.Neutral;
catch (System.Exception)
result.Mood = SentimentScore.Neutral;
return result;

This specific project uses Sentiment140 to assign the sentiment scores but could just as easily use another sentiment engine that has API calls.  The framework would stay similar but the URL and Parsing would change slightly to fit what was required by the new API.
Combining these two blog posts makes it is possible to read from a CSV file using StreamInsight and add sentiment values to the content from the CSV.  This is a solution that fits nicely with customers who are using a service that crawls the web and returns mentions of their enterprise.  Most of these services provide some sentiment scoring capability but if a customer wants to implement their own engine, a sentiment engine that is not offered though the service, or multiple engines, this solution is one we can offer to them.

Use StreamInsight to Assign Sentiment Scores to a List of Items in a CSV (Part One: Reading Input)

There will come a time when you need to read from a CSV file using StreamInsight.  It may be to match reference data with a current stream or utilize the processing power of StreamInsight to do many things at once.  This example uses the observable programming method to add input from a CSV to a list then enumerate through that list creating point events.

First things first, we need to get the data from the CSV file into a list.  I was using a “small” data set and was able to load the entire file into one list.  When working with larger files, simply process the CSV file in chunks.  The CSV parser(KBCsv) that I used can be found here.  The code below is taking the CSV file, reading it in, and creating a list of the social blurb class.

if (File.Exists(filepath))

using (var reader = new CsvReader(filepath))
while (reader.HasMoreRecords)
var columnline = reader.ReadDataRecord();
myList.Add(new SocialBlurb())
{          ARTICLE_ID = columnline[0].Trim(),
HEADLINE = columnline[1].Trim(),
AUTHOR = columnline[2].Trim(),
CONTENT = columnline[3].Trim()
Now that we have a list of SocialBlurb’s we need to create a SI application, turn the list into a point stream, add a CTI event, and do processing.  The code for this is shown below.

using (var server = Server.Create(“StreamInsightDefault”))
//Create SI app on Server
var application = server.CreateApplication(“My Application”);

//Query of all Items in the List
Var listquery = from l in myList
select l;
//Add CTI to the point event stream by increasing starttime
var streamquery = listquery.AsEnumerable().ToPointStream(application,
l => PointEvent.CreateInsert(DateTime.Now, new { l }),   AdvanceTimeSettings.IncreasingStartTime);

//read the results as starttime and payload(SocialBlurb)
var results = from Pointevent in streamquery.ToPointEnumerable()
where Pointevent.EventKind != EventKind.Cti
select new
//Do something with each point event(the Sentiment part will come in a part 2 post 🙂 )
foreach (var activeInterval in results)
var sentiment = new Sentiment140();
var result = sentiment.Analyze activeInterval.l.CONTENT);
activeInterval.l.Sentiment140_Mood = (int) result.Mood;

This post explains how to read lines from a CSV file and turn that input into a point event stream.  My next post will detail the sentiment scoring class shown above and point out how another sentiment engine could be implemented.