Life Analytics

Text Analytics for Telecommunications - Part 1

2012-01-25T13:37:00.000+02:00

As discussed in the previous post, performing Text Analytics for a language for which no tools exist is not an easy task. The Case Study which i will present in the 9th European Text Analytics Summit is about analyzing and understanding thousands of Non-English FaceBook posts and Tweets for Telco Brands and their Topics, leading to what is known as Competitive Intelligence.

The Telcos used for the Case Study are Telenor, MT:S and VIP Mobile which are located in Serbia. The analysis aims to identify the perception of Customers for each of the three Companies mentioned and understand the Positive and Negative elements of each Telco as this is captured from the Voice of the Customers - Subscribers.

By analyzing several thousands of Tweets and FaceBook posts and comments we can have a first glimpse of Competitive Intelligence. For example when we wish to identify which words frequently occur with mentions about postpaid packages this is what we find :

Red boxes show Telco Brands - notice "mts" and "mtsa" which point to the same Telco, namely mt:s. Blue boxes indicate similar words that should be merged. From a first look at the results above we see that :

a) mt:s is found more frequently when users mention PostPaid packages.

b) Telenor and VIP Mobile are not found as frequently as MT:S in PostPaid package conversations.

c) We see several problems from insufficient pre-processing : Kredit and Kredita (=credit) should merge into one word, the same applies for telefona - telefon, internet - interneta and mts - mtsa.

Notice that we can perform the same High-level analysis for several Telco Topics such as Network, Billing, Customer Care, Promotions, Questions of subscribers and so on. The next task is to identify the reason(s) why MT:S was found to have more mentions about PostPaid packages. Note that at this point we do not know why this is so : It could be the fact that MT:S prices of prepaid packages are high, very cheap or something else is happening that needs to be identified.

The Serbian Language poses extra work because it is a highly inflected language : Even the ending of Brand names change according to the usage. Consider the following :

U mts-u (at mts)
Sa mts-om (With mts)
Bez mts-a (Without mts)

It is evident that a highly inflected language explodes our feature space and for this reason R can come to the rescue with some success. We can use R for changing several synonyms to one word, removing (Serbian) stop words, removing URLs and performing several other pre-processing steps that are necessary prior to an extensive analysis. More on the next post.

Case Study : Competitive Intelligence for Telecommunications

2012-01-09T11:55:00.000+02:00

Telcos are a good example of a fast moving business environment and a good candidate for using Competitive Intelligence analysis from Social Media sources. The Case Study involves three major Telcos located in an Eastern European Country and shows the results from the analysis of thousands of Tweets and FaceBook wall posts to understand the following :

- How subscribers perceive each Telco Brand?

- Which information do subscribers tend to Re-Tweet and "Like" on FaceBook Wall Posts?

- Which words and Topics are commonly found with Intense feelings / thoughts?

- Which topics are mostly discussed when subscribers compare two or more Telco operators?

- What do subscribers discuss about Network Quality and Speed, Billing, Promotions, Marketing Events, Customer Care, TV Commercials etc.

- How do they prioritize these topics and which of them are interesting and why?

- What do subscribers talk about in general (i.e without any Telco Brand being mentioned) regarding Internet speed, Charges and what would they expect to see more?

I will present the Case Study mentioned above in the forthcoming 9th Annual European Text Analytics Summit in April in London - UK. The Case Study is an example of application of Text Analytics to a language for which currently no tools exist and thus all difficulties and possible solutions will also be discussed. Examples will be also given on analyzing information to different conceptual levels and how this technique provides even more insights in consumer behavior.

The following tools were used for the analysis :

- GATE to annotate all Topics that occur within Telco conversations (such as "sms", "internet", "dropped call", "network","promotion") and for setting up Conceptual Levels.

- R for pre-processing Text and performing Text Classification, Topic Detection and Cluster Analysis.

- WEKA for Feature Selection and Text Classification.

- Finally, Java is used to manage the information that is generated from GATE such as understanding how subscribers prioritize various Telco Concepts and Topics and also identify important phrases and/or words that frequently occur when these Topics are being discussed.

New Insights from Text Analytics

2011-11-28T13:22:00.002+02:00

Text Analytics has gained the attention it deserves in the past few years. Sentiment Analysis is perhaps the most frequently discussed type of analysis but there will be always new ways to analyze and gain insights from text data.

Examples of new types of analysis -and they have a vast potential- are in my opinion two : Sequence Detection and Concept Mining. I am not aware whether these types of analysis are currently being implemented by any Text Mining practitioner at the moment and if there is one, feel free to add your comments below.

So what is Sequence Detection and Concept Mining ? Some examples :

Suppose that you receive several similar e-mails sent from customers as the one seen below :

"I have been trying repeatedly to solve my billing problem through customer care. I first talked with someone called Mrs Jane Doe. She said she should transfer my call to another representative from the sales department. Yet another rep from the sales department informed me that i should be talking with the Billing department instead. Unfortunately my bad experience of being transferred through various representatives was not over because the Billing department informed me that i should speak to the......"

Currently Text Analytics software will identify key elements of the above text but a very important piece of information goes unnoticed. It is the sequence of events which takes place :

(Jane Doe => Sales Dept =>Billing Dept =>...)

Being able to detect the sequence of events is an important element in understanding customer interaction. In our example above, imagine the possibility of detecting similar sequences through thousands of e-mails or call center transcripts and running a sentiment analysis, a process which then could correlate sentiment with specific event sequences.

Next, is the usage of Concept Mining (this is just a phrase i coined for this post) : Being able to analyze information to different conceptual levels. A very powerful technique indeed and let's see why this is so.

People that have attended the 7th annual Text Analytics Summit in Boston had the opportunity to listen to several presentations regarding Semantics. The discussions between experts from the Semantics Panel and the attendees revealed that people could not find Semantics practical for several reasons. Yet, in Semantics lies the power of being able to find patterns on different conceptual levels.

As a -very basic- example, if we use Information Extraction to annotate -say- the Tweets containing mentions of American Telcos we can tag each one as a more general category called TELCOS. We can also tag individual prepaid packages as a more general category called PREPAID_PACKAGES. By doing that we can then search for patterns in a more general conceptual level than searching for patterns only at a Telco Brand level or a specific Telco's prepaid package. As an example we can run a sentiment analysis on all prepaid packages mentions, identify patterns of negative or positive sentiment and see which Telco is the winner of positive sentiment at a conceptual level.

The possibilities are endless.

Big Data : Case Studies, Best Practices and Why America should care

2011-09-20T08:50:00.001+03:00

We know that Knowledge is Power. Due to Data Explosion more Data Scientists will be needed and being a Data Scientist becomes increasingly a "cool" profession. Needless to say that America should be preparing for the increased need for Predictive Analytics professionals in Research and Businesses.

Being able to collect, analyze and extract knowledge from a huge amount of Data is not only about Businesses being able to make the right decisions but also critical for a Country as a whole. The more efficient and fast this cycle is, the better for the Country that puts Analytics to work.

This Blog post is actually about the words and phrases being used for this post : All words and phrases on the title of the post (and the introductory text) were carefully selected to produce specific thoughts which can be broken down in three parts :

Being a Data Scientist has high value.

"Case Studies" and "Best Practices" communicate to readers successful applications and knowledge worthwhile reading.

"America should". This phrase obviously creates specific emotions and feelings to Americans.

"Case Study" and "Best Practices" were phrases found to be commonly associated with posts of high visibility. You might also get many views if you create a post which proves that whatever concept you are writing about is the right thing to do (for example write a post that clearly demonstrates yet another reason to use Social Media and have this post shown to Social Media Professionals). Regarding our example : It is very probable (and logical) for Data Miners to look at and then re-tweet (or otherwise share) information which is a "proof" about Data Mining being useful and also a "cool" profession. The higher concept / motive which works behind the scenes is that "I am doing the right job and this post proves it".

You might also get many views by submitting a post which disproves well-accepted concepts or posts that demonstrate the difficulties that well-accepted concepts face : For example, if you were a Data Scientist or a BI Professional, you would be inclined to read a post titled "Big Data is a Big Hype". Whether you will re-tweet or share the post is of course under your discretion. At this point it should be noted that there is a big difference between number of clicks of a post and the number of shares it got (by Retweeting it, Liking it, etc) because sharing a post means that this post is considered worthwhile to read.

All of the above (and much more) have been found by analyzing thousands of Blog posts along with their number of clicks and shares they got (either by RT's , FaceBook "Likes", etc) and this is what i will be presenting in Text Analytics World in New York this October. It was also very interesting to see that some findings are in tandem with findings discussed by Joseph Carrabis during the Text Analytics Summit 2011 in Boston back in May.

Of course it is not suggested that by using specific words and phrases you are guaranteed a successful post being re-tweeted from thousands of people and there are many reasons for this which i will not get into here. Additionally, Text Analytics cannot infer the higher meaning and concepts suggested within Text and this problem deserves a post on its own. This analysis however identifies concepts and/or phrases that point Bloggers and Marketers to look at a specific direction and with this knowledge to have increased probabilities for a successful Web presence. Again, this is an example of true Social Media Intelligence. Not (just) Reports.

So, in case that this post title immediately got your attention from other posts, you've just had a little taste of Predictive Analytics in action.

Do Social Media Monitoring tools provide True Intelligence?

2011-09-09T11:48:00.000+03:00

Having recently read a report from WebLiquid one of the interesting facts to consider is that around 70% of Marketers replied finding the insights gleaned from Social Media Monitoring tools "Somewhat Valuable". Slightly more than 20% of them found these insights "Extremely valuable". The report also shows that most Marketers plan to invest more in SMM tools with few of them retreating from any further investment.

This is Big News. 70% of Marketers finding insights gleaned from SMM tools "Somewhat" valuable is not a good thing and perhaps there are reasons for this. It would be very interesting to know what do Marketers consider Insights, how they prioritize those Insights and how easily they can act once they have those insights . The problem can be summarized in one sentence:

- Marketers do not want (just) Reports.

There is a lot of useful information provided by many Social Media Monitoring tools : The number of mentions of a Brand (or Product or Service) per channel, which users talk frequently about your Brand (and which of them are considered influential). Sentiment Analysis provides Marketers with the perception of a Brand but also the perception about competitive Brands leading to what is known as Competitive Intelligence. Perhaps Social Media Monitoring platforms have many types of metrics still to offer : For example, a potentially useful metric could be the ability to identify Consumer Intentions ("I will definitely buy...") and how these intentions differentiate - such as "I would buy 'ABC' if it was cheaper" or "I would buy 'ABC' if i hadn't purchased 'XYZ' already".

Notice that SMM tools provide metrics : Number of mentions per channel, Top influential users, percentage of positive / negative / neutral sentiment and sentiment intensity, how mentions of a new product disperse through different social media channels, etc.

But what is considered Intelligence in Social Media? Would someone identify as intelligence the fact that during the past 2 months there was an increase in specific Brand Mentions on Twitter but not on YouTube? Or is it Intelligence when we notice that there has been a decline in positive sentiment about a product? All of this information is Reporting and Feedback. It is not meant that this is not useful information : It is important to know what is happening and why.

So what True Intelligence is all about?

True Intelligence is about knowing how to successfully Promote and Market a Brand, Product or Service. To do that a Marketer wants to know the Best Practices : With Social Media Reports, Marketers know what is happening (a decline in positive mentions on our new smartphone) and why this is happening (a potential hardware problem). Social Media Analytics can identify the right strategies to make things happen. True Social Media Intelligence is about knowing which parameters (channels, number of mentions) are important in achieving a result. Is it important to have a product associated with intense (positive) sentiment? Or could it be more important to have a Product being highly associated with Rumors?

There is still a long way to go in terms of Insights from Social Media Monitoring tools. There are many processes and parameters that will eventually used for deriving more Insights and better Strategies. The answer to true Social Media Intelligence is the use of Predictive Analytics (Data and Text Mining) applied to Social Data : One area that is currently untouched by most Social Media Monitoring tools.

More Trends of the Greek Debt Crisis

2011-07-15T13:37:00.001+03:00

Here are some more results on mentions of various Concepts being discussed in Greek Blogs about the Greek Debt Crisis. Using Text Analytics, thousands of Greek Blogs are being annotated on a daily basis with the purpose of identifying the frequency with which several aspects of the Greek Debt crisis are discussed.

First let's have a look at the trend line of the Indignant Citizens Movement :

We can see that there is a clear down-trend in the number of Blog Mentions. This is also supported by a very significant reduction of the total Tweets found for this subject.

Next let's see how the trend of the mentions of "Greek default" looks like in the past month :

We notice a severe spike beginning from July 12th because several Blogs and News sites were having mentions on a possible "Selective Default" which could happen to Greece.

Interestingly, the trend on mentions of a US Default is also rising in Greek Blogs but is found with a much smaller frequency :

The Greek Debt crisis - Some Trends

2011-06-17T19:49:00.000+03:00

Several friends and blog readers ask me very frequently on what i think about Greece and the problems that Greece has on a Social and Economic level. Since this is not a blog about Politics or the Economy i will try to give my point of view with some analytics added.

It is always interesting to know how people feel and what do they think about the economy,their future, the politicians and how the general sentiment is. Also of great importance is the trend of all opinions and/or sentiment as this is recorded in Blog posts and other Social Media sources.

Here are some examples from data that i collect on a daily basis, several times a day from Greek blogs. Hundreds of Concepts are annotated within thousands of Blogs entries and collected for further analysis.

The results that i will show here are for :

- the latest Government Reform

- words that communicate Negative Sentiment.

-The "Indignants Movement" : Citizens that do not agree with the practices of both 2 largest Greek political parties during the past 30 years and spending cuts directed by the IMF.

- Debt Crisis

Let us begin with the trend of "Government Reform" which at the time of writing (17/06/11 - Note that date format is DD/MM/YY) has just happened. Here is the trend of mentions :

Notice how during the previous days not many mentions were captured and how much the trend increases until June 17th were the reform took place.

Next, let's look at entries that communicate "economic default" and their trend :

Again notice how on previous days mentions of Greek default start to rise (starting from June 3rd) and gradually the trend appears to fade out (French and German leaders said they will back up Greek debt on June 17th). It was no surprise that on June 8th and 9th (yet more) Greeks rushed in Banks to withdraw their money.

Here is the Trend of "The Indignants" movement :

Notice dates May 29th-30th, June 5th, Jun 12th-13th. All of these dates are Sundays (or close to Sundays) which is the day that most people gather in Syntagma square to express their anger for the IMF and Government practices. The trend however appears to be falling but this may well be changing in the next days. Time will tell.

How about the words that communicate Negative Sentiment? Here is the trend :

Negative sentiment words appear to be somewhat rising after 31/05 but are coming down to previous levels.

FYI, words that frequently occur with the concept "Politicians" are : "leaders", "cheats", "traitors".

Apple Products on Twitter - A Text Analytics example

2011-06-16T10:20:00.001+03:00

My presentation on the 7th annual text analytics summit was a tutorial in one of the methodologies one could use to analyze unstructured text. The sample consisted of 365000 tweets that contained keywords of Apple products and concepts such as iPad, iPhone, iPod, Apple Store, Mac, Steve Jobs and the goal was to get an understanding of what people where tweeting about each product or concept.

The first step is to use a text analysis toolkit (i used GATE) to annotate the tweets and identify which concepts and keywords occur within the tweets. But this is not always easy. Take the word Mac for example. According to the context, Mac could be a computer type, a burger type, the MAC beauty products or Mac Arthur airport. So when a query sent to Twitter API that contains the word Mac we end up with lots of erroneous information.

So one of the things that have to be done to ensure good results is word sense disambiguation. We know for example that if a tweet contains a word such as fries, lettuce and/or salad then quite likely the word Mac that was also found within this tweet was about the Big Mac (even though the word Big may not be present). If we find the word Arthur next to the word Mac then the tweet is about the Mac Arthur airport, etc. Here is GATE in action, identifying different keywords and concepts in Tweets :

Now we can see which concepts and keywords appear frequently in Re-Tweets ('USER' denotes that a '@' was present in the Tweet, 'URL' that a URL link was found in the Tweet,etc)

We can also see which words frequently occur with iPhone5 :

Event Detection: Analytics becoming more personal

2011-04-26T12:53:00.000+03:00

Sentiment Analysis is a hot technology at the moment. Marketers are interested in the perception that consumers have about a specific brand, product or service as this is found in unstructured text. Some people claim that Sentiment Analysis does not meet their expectations but also that it is not straightforward for a company to find the "right" solution. Comparing different Sentiment Analysis solutions could prove a difficult task.

Marketers and Decision makers need insights with which they can make better decisions - They need both Reports and Intelligence. Therefore the question that always follows the finding that "Your product has a 35% negative sentiment in the past 10 days" is "Why". Social Media Monitoring tools must also provide actionable Intelligence.

All this is important information as it shows why your Brand / Product / Service could be losing customers. You monitor what is being said, identify whether a negative or positive Sentiment Trend is declining or rising and take necessary actions accordingly.

One of the questions i often get is what other applications can emerge from using Text Analytics and Data Mining. With Text Analytics and Data Mining we can find behavior patterns on many levels and -assuming that information such as Tweets will keep coming- the understanding of consumers can go to the next -and sometimes more personal- level.

One of these applications is Event Detection. I am not aware if Event Detection is provided by any tool at the moment but i believe that this type of analysis could become a next major source of consumer insights. But what exactly is "Event Detection"?

Since we are able to have a computer automatically identify whether a phrase contains positive, negative or neutral sentiment, perhaps we could use Text Analytics and Machine Learning to detect that a specific event has occurred to an individual from the Tweets that someone posted such as "i've just returned from holidays". But that's not all. We can mine for patterns of consumer behavior given the fact that an event has occurred. And that potential knowledge from such an analysis could be very powerful. Because apart from the emotions that a product / service / person generates, the same applies for events happening in our lives. These events and the emotions they create can sometimes change our lives and also drive our decisions. A logical next step is to collect several behavioral Data and use Data Mining to analyze this information.

I will discuss an example of using Event Detection towards the end of my presentation on the 7th Annual Text Analytics Summit in Boston this May along with the reasons for such an analysis being important and i am looking forward to the reactions.

The fact is that with more insights, privacy issues arise even more and I get an increasing number of people asking me about privacy. I was also interviewed by a major British newspaper last month on what companies can learn by applying "Super Crunching" on Tweets. I tried to show both worlds of "Super Crunching" but the truth is that consumer insights become more personal as companies understand the value of structured and (more recently) unstructured information.

Social Media Data and what analysts can do with it

2011-03-15T16:06:00.000+02:00

It is worth looking at what having our lives "digitalized" means since all of the information currently generated from usage of Social Media is available for analysis : "Collective Intelligence" and "Behavior Mining" are terms that are becoming increasingly known.

But what exactly is Social Media Data? Here are some examples :

The number of followers you have on Twitter and number of friends on FaceBook.

The number of links you provide, groups you join, retweets you make and how often you talk with other friends / followers.

The number of re-tweets, FaceBook "likes", comments and views that a blog post generates.

The personal information you provide (such as Twitter Bio)

The concepts being discussed in Tweets and FaceBook walls.

By applying Predictive Analytics to all of this information an impressive number of applications arises such as :

- Analysis of your Twitter Bio and words that are contained in your Tweets. For example we can identify what do people stating in their Bio being "Computer Geeks" discuss more frequently (in terms of Electronic Brands, technology trends etc). (See more here)

- Analyze thousands of Twitter accounts and find words that could make a difference in your follower count. (It appears that you should keep things positive -at least most of the time-. See why here).

- Identify best practices on how to use Social Media : When to post your new blog post, which words and concepts to avoid writing about and ultimately what concepts (such as Personal Branding) you should focus on. ( See more here).

- Understand consumer behavior : What people liked, how they feel and what they would like to see in upcoming products and/or experiences. See this example on how different aspects of consumer behavior in shopping malls is "mined".

Note that these are just some examples. The list goes on.

There is no doubt that new exciting Social Media apps will become available. This in turn will produce even more Social Media data (such as ones that contain location information). Being able to combine Data Mining and Text Mining techniques to extract insights from Social Media Data will become a very important skill to have.

7th Annual Text Analytics Summit

2011-02-16T16:21:00.001+02:00

I would like to say a few words about an upcoming major event for all of those interested in Text Analytics and its various uses in Social Media, Marketing and Business Intelligence. Starting on May 18th, the annual Text Analytics summit (the only conference dedicated completely to Text Mining) will take place in Boston, MA with a total of 28 speakers presenting material on applications of Text Analytics including :

Social Media Analytics

Sentiment Analysis

Voice of the Customer

Marketing

Semantics

Well-known names in the industry will be there ( Seth Grimes, Tom Anderson, Gregory Piatetsky-Shapiro, Ronen Feldman) as well as experts from companies such as SAS, IBM, Forrester Research, Attensity, Adobe, J.D. Power&Associates, Clarabridge and others.

My presentation will be about Behavior Mining in Social Media using Text Analytics and i will be giving a step-by-step tutorial on the analysis of data originating from Twitter regarding a major Electronics Brand in USA. More specifically :

I will show how Tweets can be transformed and then analyzed using various statistical NLP techniques and software.

Discuss the various problems that are found when one wants to analyze Text data

Discuss and introduce new ways of seeking for valuable information and extracting insights when it comes to Mining Behavior.

Although the Case Study will be using data from Twitter, the techniques shown can be applied to any other Text Data such as those found in FaceBook, Blog posts, User comments, etc.

I am looking forward to seeing the work done by others, learning about successful applications of Text Analytics and the knowledge gained and also seeing the issues that professionals come across and how these are faced by them.

Forex Trading with R : Part 2

2011-02-11T09:37:00.002+02:00

In the previous post the first steps were given for building the basis for trading forex. Now it is time to build the actual classifiers that can give us future buy / hold / sell signals.

Assuming that everything is in working order and the instructions given in the previous post were followed we can start building these classifiers.

First let's train a Neural Network. The following command trains a Neural Network and then applies the trained model on our test data and outputs the predictions for buy/sell/hold signals :

set.seed(134)
nn <- nnet(class~.,traindata, size = 3, rang = 0.1,decay = 0.001, maxit = 3000,trace="F")
table(actual=testdata$class,predicted=predict(nn,newdata=testdata,type="class"))

Note that a seed number was used. You should either try different seed numbers (so that network weights are re-initialized) or omit the set.seed() directive. You should also experiment with other Neural Net parameters such as the number of iterations (maxit), the learning decay (decay), etc.

The confusion matrix shows us the necessary information for calculating TP, FP, TN,FN rates for each class (ie for each signal type).

Similarly we can train and test a Random Forest :

rf.model<-randomForest(class~.,data=traindata,nodesize=40,importance=FALSE,mtry=3,ntree=100)
table(actual=testdata$class,predicted=predict(rf.model,newdata=testdata,type="class"))

Now let's train an SVM for our data. We can issue the following command :

###train SVM

sv<-svm(class~.,traindata,gamma=0.01,cost=5,kernel="radial")

To see how the classifier did on the test set, we enter :

table(actual=testdata$class,predicted=predict(sv,newdata=testdata,type="class"))

Next we can try to optimize parameters of the SVM classifier as follows :

#find optimal values of Gamma and Cost for an RBF- SVM classifier
tuned <- tune(svm, class~., data = traindata,ranges = list(gamma = c(0.0001,0.001,0.05,0.1,0.2,0.3), cost = c(1,5,10,20,50,100,120,130)),tunecontrol = tune.control(sampling = "cross"),cross=10)

tuned

The first command uses 10-fold cross validation to identify the best gamma and cost parameters among some predetermined values. We then issue the command tuned to see which combination of parameters gives us the lowest classification error. Knowing these parameters we can then use these parameters to train an SVM classifier and see how this model performs (as was shown previously).

Be aware of the following key points :

Three sets of data should be used : Training, Test and Validation. The Validation set should not be a part of the optimization (=finding the best algorithm parameters) process.
Make sure that you create classifiers for several time periods. Test the performance of any classifier according to the percentage of available data you use for training / testing / validation and the number of periods you use for the sliding window.
Make also sure that once you have chosen your model, you use a correct way to test your system by simulating buy / hold / sell signals and taking under consideration all associated trading costs.

Forex Trading with R : Part 1

2011-01-10T19:09:00.001+02:00

I recently started learning R - probably something i should have done a long time ago - and since learning by doing is the best way to learn something i decided to use R to generate buy/see/hold signals for the EUR/USD Pair. For those that wish to use R for making Trading decisions, this series of posts is a short introduction with which one can pursue the subject further. By no means it is implied that this post's methodology is the one that you should use to trade : different response variables, signal thresholds, technical indicators and classifiers than the ones presented here should be tried. Then, elaborate testing methods should be put to use to assess the performance of each classifier and the worth of your trading strategy.

First the resources needed to be downloaded and installed :

1) Packages : quantmod, nnet, e1071, tseries, randomforest

2) A file that contains OHLC Data. For this example EUR/USD Data are used. You can download an example file here. I downloaded the EUR/USD historical data here.

I would also highly recommend getting Data Mining with R : A concise book that goes through an introduction of using R and then presents various case studies one of which is about Using R to predict variations of the S&P Index. The author also provides package "DMwR" that includes all necessary functionality for generating signals, extracting precision / recall metrics of generated models, performing Monte Carlo Estimates and evaluating trading strategies.

After downloading the file from step (2), place the file to a directory of your choice. Now copy and paste the following commands in R :

library(e1071)

library(nnet)
library(randomforest)

library(quantmod)

library(tseries)

Next we import the csv file from the directory that was originally saved (change "YOUR_PATH" accordingly with the directory path you saved the csv file) :

#get data OHLC from csv file

raw<- read.delim2("/YOUR_PATH/EURUSD.csv",header=TRUE,sep=",")

Now, paste the following to R (change again YOUR_PATH accordingly) :

#convert date

stripday<-strptime(raw$DATE,format="%Y%m%d")

fxdata<-data.frame(stripday,raw)

fxdata$TIME<-NULL

fxdata$TICKER<-NULL

fxdata$DATE<-NULL

colnames(fxdata)<-c("Date","Open","Low","High","Close")

#write data to .csv

write.table(fxdata,"/YOUR_PATH/eurusd.csv",quote=FALSE,sep=",",row.names=FALSE)

##transform to an xts object

EURUSD<-as.xts(read.zoo("YOUR_PATH/eur-usd.csv",sep=",",format="%Y-%m-%d",header=T))

Now we define some technical indicators, the model to work on and a function that generates our trading signals :

#setup Technical Indicators

myATR <- function(x) ATR(HLC(x))[,'atr']

mySMI <- function(x) SMI(HLC(x))[,'SMI']

myADX <- function(x) ADX(HLC(x))[,'ADX']

myAroon <- function(x) aroon(x[,c('High','Low')])$oscillator

myBB <- function(x) BBands(HLC(x))[,'pctB']

myChaikinVol<-function(x)Delt(chaikinVolatility(x[,c("High","Low")]))[,1]

myCLV <- function(x) EMA(CLV(HLC(x)))[,1]

myMACD <- function(x) MACD(Cl(x))[,2]

mySAR <- function(x) SAR(x[,c('High','Close')]) [,1]

myVolat <- function(x) volatility(OHLC(x),calc="garman")[,1]

myEMA10 <- function(x) EMA(Cl(x),n=10)[,1]

myEMA20 <- function(x) EMA(Cl(x),n=20)[,1]

myEMA30 <- function(x) EMA(Cl(x),n=30)[,1]

myEMA50 <- function(x) EMA(Cl(x),n=50)[,1]

myEMA60 <- function(x) EMA(Cl(x),n=60)[,1]

data.model <- specifyModel(Delt(Cl(EURUSD)) ~

myATR(EURUSD) + mySMI(EURUSD) + myADX(EURUSD) + myAroon(EURUSD) +

myBB(EURUSD) + myChaikinVol(EURUSD) + myCLV(EURUSD) +myEMA10(EURUSD) +myEMA20(EURUSD) +myEMA30(EURUSD) +myEMA50(EURUSD) + myEMA60(EURUSD) +

CMO(Cl(EURUSD)) + EMA(Delt(Cl(EURUSD))) +

myVolat(EURUSD) + myMACD(EURUSD) + RSI(Cl(EURUSD)) +

mySAR(EURUSD) + runMean(Cl(EURUSD)) + runSD(Cl(EURUSD)))

Tdata.train <- as.data.frame(modelData(data.model,

data.window=c('2008-01-01','2010-01-01')))

Tdata.eval <- na.omit(as.data.frame(modelData(data.model,

data.window=c('2010-01-02','2010-11-01'))))

# a very simple signal function

signals<-function(x) {

if(x>=-0.005&&x<=0.005) {result<-"hold"} else

if(x>0.005) {result<-"buy"} else

if(x<-0.005) {result<-"sell"}

result

}

#create class vector that holds TRAINING buy,sell,hold signals

class<-sapply(Tdata.train$Delt.Cl.EURUSD,signals)

#paste both to a new list that holds everything

traindata<-cbind(Tdata.train,class)

#remove Delt.Cl.EURUSD - not needed anymore.
traindata$Delt.Cl.EURUSD<-NULL

#create class vector that holds TESTING buy,sell,hold signals

class<-sapply(Tdata.eval$Delt.Cl.EURUSD,signals)
testdata$Delt.Cl.EURUSD<-NULL

#paste to a new list that holds everything
testdata<-cbind(Tdata.eval,class)

#get a summary of our traindata
summary(traindata)

The last command prints out some summary statistics of the traindata sample. Notice the 'class' attribute and the distribution of buy, hold and sell signals .

Now we are ready to apply some modeling techniques using traindata and testdata as the datasets to work with. Although i would suggest using also a third sample for validation (remember the "elaborate testing" discussed at the beginning), for this example we will keep things simple. More on the next post

What Women Want - As seen in Tweets

2010-12-09T09:32:00.032+02:00

Probably a post which will be interesting to many of us. 15000 Tweets were collected that contained the phrase "women want". What do women value most when it comes to how they want to feel? What do women really love? How important is for women to feel special? And finally, can Tweets really tell us this information by applying Text Mining techniques to them?

Normally at this point i would describe technical details such as how i pre-processed Tweets and the problems i ran to while trying to analyze this information. Thanks to @nathalief i was advised to focus on giving information not only about what women want but also on their feelings.

First let's see the results from Tweets that apart the phrase "women want" they also contain words such as "feel, feeling, feels, felt" in them. The following chart shows what words where frequently found in these Tweets (and thus what feelings a woman wants to experience) :

So it appears that one of the first priorities in terms of how women want to feel is security (shown as safe and secure in the chart). Notice how important for women is also to feel special and to feel that someone loves them (words love, loved, like).

How about words that frequently occur with the word "Love" :

It appears that women want "to love and to be loved" with "respect", "affection" and "sex" coming next.

Surely there must be men that also give their opinions on "what women want" within these Tweets. Quite possibly many guys would say that "women just love money". In order to capture those who believe that women want money, let's see which words occur frequently with the word "money" within these Tweets :

Notice how the landscape of keywords changes here : Apart from "love", "secure" and "hurt" we see words labeled as "censored" (for obvious reasons), "shoes" and "future" : these words communicate a more materialistic and logical point of view on what women want. Unfortunately for this analysis there was no way to identify which Tweets were originated from women and which Tweets originated from men. Also at the time these tweets were collected a specific Re-Tweet was about a more 'materialistic' profile of women (words multiple, and shoes). I decided to keep this re-tweet in the data that was analyzed because i felt that since this tweet was heavily re-tweeted then it was also liked by a large audience.

Perhaps these results show once again that "Men are from Mars and Women are from Venus"

Spam Detection in Social Data : A new business?

2010-11-24T09:21:00.019+02:00

All of us who use Twitter know the problem of spam Tweets. Spamming on Twitter can happen in several ways. For example spammers can use a trending topic to make their tweets visible (that also happen to have nothing to do with the current topic). Other tweets, although they do not contain erroneous hash tags they contain uninteresting information.

In a previous example, Tweets were used to analyze the sentiment of Twitter users on U.S Economy. The study used several thousands of Tweets to extract insights. However between all tweets that originally discussed about the economy there were several spam Tweets such as "make money online even if the economy is bad".

It is well known that the most time-consuming process in a Data / Text Mining project is pre-processing. Therefore when one wants to analyze tweets and extract knowledge from them, obviously one step is to remove spam and uninteresting Tweets to minimize the chances of GIGO.

Spam detection in Tweets -and Social Media unstructured data in general- is a difficult task. It requires "concept-aware" analysis of Text. One of the interesting facets of analytics is the ability to solve the same problem in several ways, or -perhaps even better- to combine all available tools to reach a better solution.

There is an ever growing number of companies that analyze Social Media Data and erroneous data may be seriously altering their insights - even if millions of records are available. Perhaps in the very near future, providing cleaned social media data to analytic companies or other information consumers could be a business in its own.

It is possible to perform spam detection in many ways : Using machine learning methods is one : In other words, training a classifier to sift through -say- hundreds of thousands of tweets that are marked accordingly as "spam" or "no-spam". We could use a more elaborate methodology to actually build and define rules by non-automatic methods that characterize spam Tweets. We could even consider other information such as who Tweeted, how many followers this user has or how often '@' is used to address other users. Once again, problem representation and how / which algorithms are used should be carefully selected.

Spam detection in Social Media Data is one of the problems that will become more important as more analytic companies are created. Detecting interesting information is another area to watch. People want real insights.

In the previous post, tweets were used to identify what people want / feel / don't like when they visit a shopping mall. While analyzing this information it was found that word 'Omaha' was associated with the word "Mall". Under close inspection i realized that "Omaha Mall" is a song by Justin Bieber. Of course i am not suggesting that these Tweets about Justin's song were spam but they had nothing to do with the purpose of the analysis. Could an automated technique identify this inconsistency and suggest to filter out this information? Being able to automatically select the right information will probably become more important as text information increases and a fast, correct and actionable intelligence becomes a necessity.

Mining consumer behavior in Tweets

2010-11-03T02:43:00.024+02:00

In the previous post we discussed the first steps necessary to understand what consumers write in their Tweets regarding their recent visit to a shopping Mall. In this post we will see how from this information Marketeers are able to understand spending patterns, know what consumers liked about their visit to a shopping Mall and know what is important for consumers. According to Dr Dimitriadis with whom i teamed up for this analysis, important things to look at include (list not exhaustive) :

- spending patterns and situations (when, what and with whom people spend money)

- tenant mix preference (which products people like to buy and what else / new they want)

- experience evaluation (safety, availability of stores / products, cleanliness etc)

- perception of shopping mall communications (what people think about mall ads / messages)

Although there are many behaviors and opinions we could look for, let's identify first what makes a consumer happy. To find this out we can analyze all Tweets containing a :-) (smilie) and find which keywords co-exist in these Tweets. Here are some of the results:

Apart from some typical words that suggest positive feelings, we also identify that 'friend' and 'birthday' are commonly found with smilies. It was found that consumers that shop for a birthday present or outfit use smilies often. Let's see what happens with tweets that contain negative feelings :-( (frownie):

A frown appears more often when consumers do not find what they were looking for and also when they are at the mall alone. But what about what people hate when they visit a mall? A similar statistical test is performed to identify words which co-occur with the phrase "I hate it when." These words are:

-Park

-People

-Walk

By looking at the actual tweets we can identify that many people hate it when:

1. a mall is very busy

2. it is difficult to park at the mall

3. people in front walk at a much slower pace (particularly older people)

Next we can perform a cluster analysis for these tweets to identify common "thought clusters" of the consumers and their behavior. As an example i have used Rapid-I to generate these clusters using the following setup:

Without getting into technical details (such as usage of tokenization, stop word removal and optimization of the process) by executing the stream shown above, a cluster analysis is run that identifies common consumer thoughts on their visit to the shopping Mall. Some of the clusters found are :

- People that state their intent to buy something

- Consumers which eat a meal and then go to the movies

- "saw a cute guy / girl looking at me"

- "I had a good time at the mall"

As discussed in previous posts, cluster analysis not only allows us to find common groups of behavior and thoughts but also to identify the frequency with which these behaviors and thoughts appear in consumer Tweets.

This behavior mining seems endless : In the same manner we can look for mentions of food, (for example see how often 'Chinese', 'Indian' or 'Pizza' appear in Tweets) or buying patterns (which items are discussed more frequently in "i want to buy" Tweets) or whether users feel more happy when they buy gifts for themselves or for others.

Inside a consumer's mind with Text Analytics

2010-09-28T08:17:00.051+03:00

So far we have seen several examples on how Predictive Analytics applied in Social Media and Blog posts can help us suggest better strategies in Marketing, Branding, Sales and PR . This post is a walk-through example on how we can choose a concept, extract what users write about this concept on Twitter, get insights on how consumers think / behave about it and finally group similar consumer thoughts and experiences using Cluster Analysis. A "concept" could be :

- Any activity

- A Brand (e.g Apple Inc.)

- A Product / Service

- A Politician

and -almost- anything discussed in user Tweets .

What we will look at is work that was made specifically for understanding what consumers think, liked or disliked while visiting a shopping Mall. What do people feel when visiting a Mall? Which words are associated with a positive experience or when a smiley is present in Tweets about Malls? Using the Twitter API approximately 36000 distinct Tweets where collected on consumer experiences from visiting a shopping Mall (sample below shows an example of a consumer's negative sentiment ) :

So how can an analyst get into a consumer's mind by analyzing Tweets and how would this information be useful? To find some answers I teamed up with Marketing Strategist Dr Nikos Dimitriadis to assist me in the actionability and interestingness of each extracted insight. Note that we capture thoughts from a biased sample which means that we cannot make inferences about the general population. However this work can be a great additional tool for finding new ideas and insights for Marketing initiatives -on top of more traditional methods such as focus groups- and also enables us to form several hypotheses as to what could likely work.

After a number of pre-processing steps to clean captured Tweets from irrelevant information (such as links), replace words with their synonyms and remove frequently occurring words such as 'and', 'to', 'at', 'in' and 'mall' and also filter all Tweets with small length i started performing frequency counts of the words contained in Tweets about Malls :

We immediately notice how often LOL and :-) (smiley) appear in Tweets about being, going or returning from the Mall which also gives us examples of consumers being in a specific mood . Here is what happens when we look at the most frequently occurring 2-word phrases :

and 3-word phrases (Note : ive = i've) :

Looking at the two charts we also notice that we frequently find the phrases :

- My best friend : since consumers Tweet the fact that are visiting a Mall with their best friend.

- My nails done : appears to be one of women's frequently discussed activity.

We then could look at Words and Phrases that seem interesting in understanding consumer experiences and values when visiting a Mall, such as :

- Shop

- Shoes
- Parking Lot
- Food Court
- Need / Want
- Walk around
- Made my Day
- Post Picture FaceBook

and mine through all these words / phrases to understand what consumers think : What exactly made the day of consumers who used the phrase "Made my Day" in their tweets? How do consumers feel when they visit the Mall with their best friend? when they are alone? Which activities trigger positive feelings? But more importantly : How could one use this information to better understand consumers and Market a concept? More on the next post.

Social Media Insights from Predictive Analytics

2010-09-22T09:17:00.019+03:00

Here is one more example on how Predictive Analytics may help professionals to make better decisions. For this post a total of 3000 Social Media title posts where analyzed to gain -hopefully- important insights for Social Media professionals. To achieve this, Text Mining was used to analyze the text of titles, identify the most important subjects (do posts about Personal Branding tend to be re-tweeted more than Social Media Monitoring?) and also try to prioritize the various areas of Social Media.

We start with the basics. Many of Social Media pros read (and write) about various subjects : How-to's, things to avoid, Adoption of Social Media etc). The first goal was to identify the most frequently occurring subject areas in Social Media posts using simple keyword frequencies. The following chart shows this information :

Although the fact that Social and Media is on top of the list is not much of an insight or that Twitter appeared in posts more frequently than FaceBook, we see that Brand is found more frequently than Marketing or Strategy.

However, there is a slight problem : The chart shown above is about single words and perhaps measuring how often 2 adjacent words occur in Social Media posts could be more useful with Social Media being omitted (click to enlarge):

This leads us to the fact that most of Social Media posts where found to be about How-to's (note that phrases How to and ways to have similar meaning). One could dig more to identify the concepts for which How-to's apply (How to monetize, How to be successful, How to avoid mistakes etc)

The next goal was to find words and phrases that are commonly found in posts with a high number of retweets (>40). To get this insight various Text Mining techniques where used. The following features have been taken into consideration :

- Author of Post

- Title of Post

- Number of Retweets

and here are some of the results :

Words that have a negative weight tend to be found in SM posts with a low number of re-tweets (write, talk, trust, sentiment) while launch and America where commonly found in popular posts. Please notice (the reason will be explained later) that personal is one of the hot words but also link and increase.

With this information, an analyst may then identify why such words tend to commonly exist in popular Social Media posts. Here are some insights :

Personal Branding appears to be a hot area. People are primarily interested on the various ways they can increase their "personal worth" in the Social Media arena.

IWOM : Internet Word Of Mouth is also a concept that frequently occurs in SM posts with many re-tweets.

Positive & Possible : It appears that posts that discuss various possibilities in a positive way (use of the word could) where found to be re-tweeted more (recall link and increase keywords discussed previously).

"Ways to stop Social Media and Sentiment Mining"

2010-09-05T15:40:00.015+03:00

While looking at my Google Analytics account i came across a keyword search originated from Australia which was different from keywords that usually drive traffic to my blog. The keywords were the following :

"Ways to stop Social Media and Sentiment Mining"

I decided to write this post assuming that the person who submitted this search does not like the fact that machines are mining his points of view about people or products or "understand" to some point whether he/she feels happy or not.

Among the many interesting aspects of being a Data Miner is to explain to other people what a Data Miner does (this was also discussed by G Piatetsky - Shapiro if my memory serves me well). When asked, i sometimes say that i also "analyze emotions as these are expressed on the Web". At first people are very interested but after a short amount of time almost always the next responses go along these lines :

- Are you allowed to do this?

- Is this legal?

- Have you ever heard about Big Brother?

It's no big secret that emotions play a major role in our lives and drive our decisions. Many people start to realize that companies are already using Information Extraction and Data - Text Mining techniques to extract the things that we discuss about various products or people and better understand our behavior. I believe that the most important thing in this area is not just Sentiment Mining or in other words whether we feel positive or negative about a Person, Product or Brand but the ability of Analytics to extract our core values and analyze our emotions.

When applying Text Mining or a mixture of Data and Text Mining methods on -for example- Twitter, we are not only able to see the sentiment for a product. We can identify a user that is alone, feeling bored and watching television. We can form several hypotheses on whether users that survived from Cancer express more positive thoughts than other user groups (see Surviving Cancer, Happiness and Twitter), find what sort of lifestyle makes a CEO happy or whether a specific profession increases your chances of being single (see Twitter Analytics : Cluster Analysis reveals similar users). Cluster Analysis can also identify core values of people and what they want or what trying to avoid.

Some of the examples discussed above have a clear business value while others don't. The important fact however is that analysts now have data to analyze emotions and our responses on facts happening in our lives on a much deeper level. This information has not been available on this scale before.

Should we stop extracting these insights and how dangerous can these insights become?

Banks, Risk Disclosure and Text Analytics

2010-08-31T11:52:00.004+03:00

A UK-based MSc student of Kingston Business School - Christos Gkemitzis had an idea for his MSc project which immediately caught my attention : Use Text Analytics methods to annual reports given by Banks and extract metrics on how these Banks handle their Credit and Interest rate risk as explained in these reports and then test several hypotheses ( do Banks of a higher risk profile disclose bigger amount of risk-related information compared to those having lower risk profile?) and also identify any correlations :

between the size of the Bank and volume of risk disclosures
between the risk of the Bank's profile and volume of risk disclosures
between the profitability of the Bank and volume of risk disclosures

Essentially the problem is to -automatically- identify mentions of credit risk but in a specific way :

1) Identify sentences mentioning risk refer to the present, past or future

2) Identify positive, negative or neutral sentiment mentions about Credit Risk

3) Identify qualitative versus quantitative information regarding the Bank's Credit Risk

For example consider the following text which is part of an actual Bank report :

"A substantial increase of credit risk and provisions is also expected, as from 2009 on, theeconomy will be entering a period of low growth."

The sentence above contains qualitative information ("substantial increase of credit risk and provisions") and negative Sentiment referring to the future ("also expected" and "will be entering a period of...").

while the following sentence :

"The Group’s ongoing efforts to manage efficiently credit risk led the level of loan losses to 3.3% in December 2008"

contains quantitative information ("level of loan losses to 3.3%") with a positive sentiment about Credit Risk handling in the past.

After receiving some PDF samples of Bank reports from Christos, i began feeding these reports to the GATE Text Analysis toolkit in order to assess the feasibility of such analysis. After some tutorials through Skype, Christos -who had no prior knowledge of programming- started using the toolkit on his own in a very short amount of time. Here is a snapshot of GATE in action for the analysis :

The snapshot shows how GATE correctly identified a part of text that communicates a negative sentiment for Credit Risk in a qualitative manner for the future (notice that "QualitativeBadNewsFuture" is checked).

After running GATE in many documents, Christos had the necessary metrics (=how many mentions of different Risk types exist in a document) to test his hypotheses using a 2-tailed Wilcoxon test. To identify correlations, Spearman coefficient was also used.

Since this is work which has not been submitted yet, it is not permitted to post the findings of this research. The post shows however another application of Text Analytics and the many sources of unstructured information that could be mined for knowledge.

Interview with Kaggle CEO Anthony GoldBloom

2010-08-09T08:17:00.015+03:00

For those that haven't heard of Kaggle before, Kaggle is a team of people that provide the functionality and support to host Data Mining contests. Here is how it works : Suppose that you are working for a Telco and wish to implement a new Churn prediction model. Rather than running this project in-house, you submit your data to Kaggle. What happens next is that -hopefully- many statisticians globally will each analyze your dataset, produce a model and then submit their prediction model(s) to Kaggle. The best model (and hence its creator) gets the prize which is given by the Telco company. Here is the interview with Kaggle CEO, Anthony GoldBloom :

- What is Kaggle and what new ideas brings to the predictive analytics arena?

Kaggle aims to help companies and researchers make predictions more precise by providing a platform for data prediction competitions. Competitions turn out to be a great way to get the most out of a dataset. This is because there are infinitely many approaches to any data modeling problem. By opening up a data prediction problem to a wide audience, a competition makes it possible to get to the frontier of what is possible given a dataset's inherent noise and richness.

- Can you tell us more about "real-time science" and how it could help Research globally?

Data modeling competitions can facilitate real-time science. Consider the recent announcement about the discovery of genetic markers that correlate with extreme longevity. Work on the study began in 1995, with results published in 2010. Had the study been run as a data modeling competition, the results would have been generated in real time and insights available much sooner (and with a higher level of precision).

Data modeling competitions also benchmark, in real time, new techniques against old. A technique that performs well in competitions can prove its mettle long before any paper can be published, helping the science to progress more quickly.

Competitions also help to avoid situations where valuable techniques are overlooked by the scientific establishment. This aspect of the case for competitions is neatly illustrated by Ruslan Salakhutdinov, now a postdoctoral fellow at the Massachusetts Institute of Technology, who had a new algorithm rejected by the NIPS conference. According to Ruslan, the reviewer ‘basically said “it’s junk and I am very confident it’s junk”’. It later turned out that his algorithm was good enough to make him an early leader in the Netflix Prize (he called his Netflix Prize team NIPS_reject).

- How can companies benefit through Kaggle?

Companies can use Kaggle to gain an advantage over their competitors. Consider a bank that wants to improve the algorithms that vet loan applicants. If a bank can develop a more effective algorithm they will have fewer defaults and can charge lower interest rates than their competitors. Kaggle has proven to be an effective way to improve existing models very quickly.

Competitions are also really useful to companies that want to develop new products and capabilities. Consider a hedge fund that wants to be able to generate long-range weather forecasts in key agriculture regions. They can attempt to hire a weather forecasting expert or they can use Kaggle to throw the problem open to a wide audience. Using Kaggle they can be sure they'll get great results very quickly.

- How is the best model selected?

The competition host will typically split their dataset into two parts - a training dataset and a test dataset. The training dataset includes all explanatory variables as well as the dependent variable (or the answer). The test dataset also includes all the explanatory variables but the dependent variable (or answer) is withheld.

Participants train their models on the training dataset. They then apply their models to generate predictions on the test dataset. Those predictions are then scored on-the-fly against the actual answers (using one of several evaluation methods). Once the competition deadline passes, the team that generates the most accurate predictions gives the winning methodology to the competition host in exchange for the prize money.

Summarization of Blog posts with "Web Pulse" Reports

2010-07-27T11:00:00.015+03:00

In the past couple of months i was looking for a way to best capture and understand what happens on the Web -and more specifically what people write in blogs- in terms of sentiment and emerging trends. The first thing that i came up with was the the idea of creating a "Web Pulse" Report : A way to summarize what people are discussing on the web. Although the implementation was not as complex as i expected, i was pleased to find that the knowledge that can be extracted is -to say the least- very useful and interesting. Before looking at an actual Report examples here are the elements that comprise it :

1) Concept Frequencies : Identifies the concepts that bloggers most frequently write about

2) Global co-occurence Matrix : Identifies most frequent word bigrams

3) Keyword Associations for Concepts : Which keywords tend to co-exist with a specific concept?

4) Most frequent n-grams associated with a given Concept (where n=2,3,4,5)

As an example we will identify what bloggers were discussing in Greek blogs on July 27th, 2010 and specifically the Blog titles in more than 300 Greek blogs.

Here are the concept frequencies found (in descending order) on that date :

[Turkey]=178
[Politics]=128
[Economy]=101
[International Monetary Fund - IMF]=62
[Banking]=61
[Public Sector]=50
[Negative Characterizations]=30
[Political Parties]=29
[George Papandreou]=29 (=Prime Mininster of Greece)
[Loans]=22
[Society]=20

The first interesting fact was that "Turkey" appears to be in the top of the list of Greek blog articles, even though Greek mass media did not place so much weight in the latest Turkish behavior in the Aegean sea on that day. The second concept is Politics with the Economy following next.

Here is the top part of the Global Co-occurence Matrix found (in Greek) :

ΣΚAΦΟΣ,ΤΟΥΡΚΙΚΟ : 25
ΡΕΙΣ,ΤΟΥΡΚΙΚΟ : 25
ΠΙΡΙ,ΤΟΥΡΚΙΚΟ : 25
ΡΕΙΣ,ΣΚAΦΟΣ : 24
ΕΛΛAΔΑ,ΧΩΡΑ : 23
ΥΠΟΥΡΓΕΙΟΥ,ΟΙΚΟΝΟΜΙΚΩΝ : 22
ΡΕΙΣ,ΕΡΕΥΝΗΤΙΚΟ : 22
ΠΙΡΙ,ΕΡΕΥΝΗΤΙΚΟ : 22
ΡΕΙΣ,ΠΙΡΙ : 21
ΑΝΑΜΕΝΕΤΑΙ,ΣΥΜΦΩΝΑ : 21

ΠΟΛΙΤΙΚΗ,ΧΩΡΑΣ : 19
ΟΙΚΟΝΟΜΙΑ,ΕΛΛΗΝΙΚΗ : 19
ΜΟΝAΔΩΝ,ΔΕΗ : 19
ΚΥΒΕΡΝΗΣΗ,ΠΑΠΑΝΔΡΕΟΥ : 19
ΗΓΕΣΙΑ,ΠΟΛΙΤΙΚΗ : 19
ΕΡΕΥΝΗΤΙΚΟ,ΤΟΥΡΚΙΚΟ : 19
ΥΠΟΧΡΕΩΣΕΙΣ,ΜΝΗΜΟΝΙΟ : 16
ΧΩΡΑ,ΜΝΗΜΟΝΙΟ : 15

The top 4 frequent keyword associations is -again- about the latest problems of Greece with Turkey and more specifically with the fact that a Turkish boat named "Piri Reis" (in Greek : ΠΙΡΙ ΡΕΙΣ) has been repeatedly entering without a permission a Greek part of Aegean Sea.

Let's look at the Associations frequencies found between specific Concepts : The following is an example of concepts associated with "Giorgos Papandreou" (Greek Prime Minster)

International Monetary Fund - IMF=32
Politics=28
Political Reform=6
Nea Dimokratia=3 (=Oppositional Political Party)
Politics, International Monetary Fund, Loans,Political Parties=2
Negative Sentiment=2
Public Sector=2

Uncertainty=2

It appears that George Papandreou is frequently mentioned where the IMF is involved and also a political reform might be on its way.

The fourth element of the report shows phrases that are commonly found in Blog posts. Since many blogs tend to use the same titles, with this functionality one is able to look at the distribution of the information from one blog to another.

The report can be enhanced in various ways : For example by tokenizing Blog posts in sentences i have added the option of performing chi-square tests to identify co-occurences in a more concise way, rather than using strictly absolute term frequencies. Through different types of analysis and knowledge representation we are able to look to our subject(s) of interest in different ways, which -hopefully- leads us to better insights.

From my experience so far, this type of report is a simple but efficient way to summarize the content of Blogs and also show what is 'hot' at the moment and why.

Concept Trending : A Glimpse into the future?

2010-05-26T21:45:00.025+03:00

In the previous post some ideas were presented on the trends of Text Analytics. Analyzing and extracting knowledge from text is a hard thing, whether this involves Sentiment Analysis, Text Classification, Cluster Analysis or Information Extraction.

A particularly interesting application of Text Analytics is the identification of trends for specific concepts. In contrast with simple keyword trending, this type of trending attempts to disambiguate keywords according to their context and use co-reference resolution to identify the subjects for which the sentiment relates to.

To better understand concept trending let's look at an example : Suppose that one wishes to identify the trend of negative characterizations -and even swear words- that exist on the Greek web. The first step would be to collect the information from various blogs and forums whenever a negative keyword is found. A Text analysis toolkit could then provide the means of identifying the subject(s) of negative characterizations on the Greek web such as Politicians, the Economy or the International Monetary Fund which recently came in to the rescue.

From a post dated December 28th, 2009 :

"Over the past month there has been a considerable amount of increase in negative economy sentiment, crime-related incidents and/or terms that communicate future social instability and uneasiness."

Although not stated on purpose, the country which the article addressed was Greece and the trend increase on negative sentiment was found to be starting in the beginning of December 2009. This is a photo of a Greek newspaper taken on February 4, 2010

The title shown writes about the "Fear of Social Explosion". On May 6th 2010 after clashes in the center of Athens, mentions about "Social Explosion" in Greece started appearing on the Web. The following Google search uses a timeline for "Social Unrest". The increase of mentions appears to be starting on February 2010.

Although concept trending has significant challenges it is a process which in my experience has proven itself many times. A recent article at NewScientist suggests that by capturing the sentiment of the crowds we are able to predict the moves of S&P 500 or by looking at keyword searches such as "job search engine" we can predict coming changes of the US unemployment rate.

The future and trends of Text Analytics

2010-05-17T17:58:00.019+03:00

I recently attended a GATE seminar on the University of Sheffield. Having used GATE for quite some time now, i was happy to see that the GATE team is well committed to developing the GATE Text Analysis Workbench by constantly adding more functionality.

Although many of the participants were PhD students i was also happy to see people from companies that now wish to leverage the hidden knowledge that exists in unstructured text. Whether it was analysis on text of Patents information, intelligent search on Text of Photo Captions for a large News Agency or understanding what a customer wants, Text Analytics are becoming an important tool for making better decisions.

I also had the opportunity to speak with several people about the future of Text Analytics. What are we likely to see happening in the next years on Information Extraction and Text Analytics?

First we have to understand how Text Analytics deliver results. In order for a computer to 'understand' unstructured text, it should be 'taught' that the word 'Dollar' is a currency of a country that is called 'US' and also that US, United States, USA and U.S.A is the same concept. This means that hundreds of thousands of concepts and synonyms have to be specified so that a computer identifies them in unstructured text. This process is called Text Annotation.

The Golden Standard of Text Annotation is annotations done by humans : A computer sifts through the text of a web page, annotates it with concepts and then these annotations are checked against annotations made by humans on the same text to assess the accuracy with which a computer 'understands' this text and the concepts and entities that exist in it.

So what does the future hold? First of all, since unstructured text becomes more available there will be a greater need for 'annotation farms' : Groups of people who will be manually annotating free text, identifying an ever-growing number of Companies, Managers, Politician names, or anything else that has to be 'taught' to a computer. Note that Annotation Farms already exist but the need for this service will become greater.

The second trend on Text Analytics could be something equivalent to what we have seen happening with NetFlix. Suppose that you own a company that produces Brand 'X' and you wish to track the reputation of your product online. You would then submit a sample of your product's mentions to various companies that analyze text and have them compete against each other in terms of -for example- Precision and Recall. The one that produces consistently the best metrics (whether Precision - Recall, Kappa statistic or F-Measure) will also get the job.

A third trend could be the development of text analytics for specific concepts : Sentiment Analysis and Named Entity recognition is hard work if one wants to produce sound and accurate results. So it could be probable that Text Analytics experts will choose a specific concept -For example reputation of Banks- and then work in the analysis of this -very specific- concept so that they achieve better metrics.

Predictive Analytics and Politics - Part 2

2010-03-23T23:33:00.019+02:00

In the previous post we have seen an example of analyzing messages sent from citizens regarding a new taxation plan. We identified some correlations between keywords and concepts but there are more ways to gain knowledge from such unstructured information.

By using Cluster Analysis we can extract groups of similar concepts among thousands of comments written by citizens but also presenting an order within them. Let's assume that Cluster Analysis reveals the following clusters (or similar concepts) within submitted messages :

- battling tax fraud

- requests for a fair tax plan

- requests for less taxation for large families

- various incentives for citizens

Our problem is finding the order of importance that people place on the various concept categories shown above : Is battling tax fraud considered more important (=discussed more frequently by citizens) than requesting a fair tax plan? How about taxation for larger families?

A cluster analysis can reveal to us the size of each cluster and -as a consequence- how important each cluster is :

We make the assumption that in the text representation shown above Cluster 5 (which contains 329 citizen messages) is about requests for a fair tax plan and Cluster 10 contains messages with requests that tax fraud should be minimized. It appears that significantly less people are concerned with a battle against fraudulent activity but they request -more immediate- benefits through a fair tax plan.

Collecting and analyzing information found in blogs and forum entries is another area of analysis that could prove very interesting. Let's see an example with the Political / Social / Economic situation in Greece : The goal is to identify and extract trends and co-occurences of key concepts from blog titles and forum posts such as :

- Names of major Political parties

- Names of Politicians

- Economy (words/phrases such as "austerity plan")

- Negative characterizations

- Company Names

...etc

For this kind of data several applications can emerge. We could track specific concepts through time and see their trends. We can also identify which concepts are discussed together. As an example we could identify the reasons on why Giorgos Papandreou (PM of Greece) is characterized in a bad way in blog posts. (= what other concepts are found in Blog posts containing keywords 'Giorgos Papandreou' AND Bad Characterizations?) :

(Note : PASOK = Governmental Political Party )

Politics = 120

Economy=72

Economy, Politics=40

PASOK=24

Politics, PASOK, Referendum=8

Economy, Politics,PASOK,Referendum, Immigrants=8

Economy, Politics, Society=8

Society, PASOK=4

In other words : Giorgos Papandreou is criticized mainly for his Political decisions and the Economy followed by criticism on PASOK. Negative sentiment also exists because of the fact that a percentage of Greek citizens require that a referendum should take place concerning the latest decision of the Greek government to give to a large proportion of Immigrants the Greek citizenship.

Predictive Analytics and Politics - Part 1

2010-03-12T16:16:00.032+02:00

One of the most interesting applications of Data/Text Mining and Information Extraction is Politics. I started collecting information from various blogs, websites and forums and applying Information Extraction and Data/Text Mining techniques to extract potentially useful knowledge in this area. By combining different pieces of information one could come up with trends that may tell us what lies ahead of us.

The latest developments in Greece are more or less known to most of people that read International News. The situation is difficult and the voice of citizens in various blogs and forums could give us the sentiment of Greek Web Users. For example :

- Which are the most frequently occurring words?

- Which are the most frequently occurring thoughts?

- What are the things that have to be changed by Greek politicians?

To answer these questions i have started collecting information found on the top 120 Greek blogs, the OpenGov website (a state-run website where Greek citizens express their opinions) and a couple more Greek sites of economic content. For blogs and forums a Java program scans every 20 minutes for new information :

This information is then sent to an annotation engine which analyzes the textual content. Once the text is analyzed we can -for example- produce a keyword vector that we can later use to understand what citizens are saying on the Web. We can then find out answers to many interesting questions such as :

- With which words is Mr George Papandreou (PM of Greece) associated with?

-When there are some very negative words (such as swearing) what other words are found in the same text?

- What does keyword trending tell us? (For example, we identify an increasingly number of swear words in citizen posts)

First let's see some examples regarding the OpenGov website where thousands of citizens have expressed their opinions on the tax policy of the Greek state. The following chart shows us a number of pairwise correlations between written words in these comments :

Under the red rectangle appear two words (dikigoros,iatros) which in Greek mean "Lawyer" and "Medical Doctor" respectively. This essentially tells us that these two professions are used together frequently in citizen discussions. By looking closely at these messages one can reveal that professionals in these two sectors are said to avoid taxes by not issuing receipts.

Next we could use association rule learning to look for some more -potentially interesting - rules :

The highlighted rule although one of low support it could prove interesting : A subset of citizens are requesting that freelancers and the self-employed should be more closely monitored for tax fraud.

Apart from rule learning, it is interesting to identify the proportion of the total dataset for which each rule holds. That also gives us a sense of order with which different ideas and thoughts exist on the mind of citizens.

In the next post : What is the Voice of the Citizen tells us in Blogs and forums?

Detecting Novelty in Twitter posts

2010-01-04T19:55:00.026+02:00

A question one could come up with is the following : How can we easily identify and extract novel information from the web? Although we could apply this "novelty detection" into many areas i would like to discuss for now the idea of semi-automatically identifying novelty among posts on Twitter.

Let's take for example the IPhone. Thousands of Tweets are generated every day regarding the Apple IPhone. These tweets mainly discuss about :

Which new apps are available / used / liked.
New accessories (cases, chargers, etc)
User Experiences and sentiment (such as blaming IPhone's short battery life)
Pros and cons of the IPhone vs other similar devices
Upgrading / hacking etc.

So the problem is : How can we identify novel information among thousands of tweets? Some would argue that we should first define what is "novelty" such as finding a new application or a new accessory for the famous mobile device. Others might argue that novelty is a customer idea that not many people about the IPhone thought about and for which Apple would be interested in identifying among thousands of Tweets. As an example consider the following Tweets :

A subset of users experiences problems with the automatic orientation of the IPhone : This subset of IPhone users is perhaps very small but identifying these tweets could give Apple some ideas to work on.

Here is another subset of Tweets that talk about the charger's cable length :

In the example shown above notice that using just "iphone cable" as search terms would return a large number of Tweets, making it hard to identify novelty among all these Tweets.

Searching for novelty and identifying new ideas among Tweets is not an easy task. The problem is that we do not know what we are looking for in the first place : We can define the general context -such as wanting to identify novelty in user experience- but then we come to a halt in terms of what techniques to use (with an exception perhaps being cluster analysis).

The potential of using semi-automatic novelty detection on Twitter and other websites -such as delicious links- is very big. Although this is work still in progress, the general methodology of novelty detection in Twitter could be to :

1) Collect a large subset of Tweets mentioning IPhone and a keyword that identifies context (such as the word charger).

2) Identify keyword frequencies

3) Generate search queries using a subset of keywords chosen in an "intelligent" way, otherwise the number of search queries would be practically impossible to be evaluated.

4) Test these combination of keywords by submitting them to Twitter search and evaluating the results.

Steps (3) and (4) shown above are the key to success of course. In our example about the IPhone cable being too short we had results returned because the combination of keywords submitted could make sense. Trying out IPhone, cable, snow tells us that such keyword combination is not a valid one and -hence- not an "intellligent" keywords subset :

Building a Knowledge Hub

2009-12-28T18:13:00.027+02:00

The web is a huge source of information. It stores facts, thoughts, feelings and intentions of people. It also records what people like and what they don't in an indirect way - something that we are going to be looking at shortly . Some of the examples on harnessing this information were shown previously in this blog, such as :

Extraction of user opinions, beliefs and values from Twitter

Prediction of popular stories on Digg

Prediction of popular Tweets

Consider the following snapshot from a BBC webpage :

The table above shows a representation of the most popular business stories on BBC on the 22nd December 2009. Even though we do not have specific metrics, we intuitively understand that the order with which the stories are listed also tell us the popularity of each post. Notice that the first post on the most read stories talks about the British economy while the last one is a title regarding football.

This is knowledge that we can harness. No doubt it is a very specific kind of knowledge because it tells us only what -mostly- British readers of BBC have found interesting. In other words this is knowledge for a specific population : Most likely in another country -say France- the title about UK being still in recession would not be so interesting but a title about France being in the same situation would. Subject, Time and Location are all important parameters that need to be captured and taken into account.

Let's consider the idea of creating a Knowledge Hub : This could be done by collecting massive amounts of information from Social Media, blogs, comments from forums and news titles (and their popularity). Techniques such as Information Extraction with concept annotation, Data and Text Mining could be used to extract knowledge by combining incidents, opinions, intentions and emotions found from different sources.

I have been monitoring and collecting for the past 3 months news and forum posts generated from/for a specific country. The information collected is then annotated in such a way to extract concepts. This text annotation is matched with keywords of concepts, incidents and intentions. Over the past month there has been a considerable amount of increase in negative economy sentiment, crime-related incidents and/or terms that communicate future social instability and uneasiness.

It is a very interesting fact that our behavior is recorded -up to a point- by the web. Again, the key is the way that we are able to organize this information into logical chunks and then use this representation to find possible insights.

2009 has been a year of big changes. Best wishes for a Happy and Prosperous New Year for everyone.

The sentiment on US Economy from Twitter

2009-10-31T16:22:00.059+02:00

Is the economic crisis over? What is the sentiment of people regarding US Economy and the future? These are some of the questions that many people ask these days and the signs are somewhat mixed. Dow Jones is close to the 10000 mark and some US Economy Indices show that the worse is behind. But do people feel the same?

To answer these questions 10000 Tweets containing the word economy were collected with the purpose of finding out what people think and how they feel about the US Economy and the economic crisis. The following web chart shows some of the results :

PositiveSentiment is an annotation type that includes all words that suggest positivity such as good, better,advances while the opposite annotation (NegativeSentiment) exists for all keywords that suggest negativity.

The bolder the lines between words the heavier the association. To get an idea of how people feel, look at the line that connects NegativeSentiment and the word still which implies that the strongest sentiment is that US Economy is still under big problems.

Some other findings :

- US President tells that the economy gets better but people don't feel the same.

- Economy cannot be getting better while at the same time there are layoffs.

- People expressing very negative feelings after losing their jobs.

Notice also the association between NegativeSentiment and people, job, money, sales. Interesting insights can also be found if brand names and product categories are also taken into account : In this analysis a specific brand was found that was associated with word sales and a good overall sentiment. Buying behavior can also be found regarding consumer intentions.

You will also find that an association exists between finance_institution keywords (implying keyword Fed) and PositiveSentiment. This association exists because a number of Re-Tweets is about the Fed signaling the start of exit from recession and its impact on housing. Interesting also is the association between the words fool and annotation PositiveSentiment (...)

Specific Tweets were removed such as spam Tweets (that try to sell investing products). Re-Tweets were kept intact since we are making the assumption that if someone Re-Tweets -say- a positive sentiment Tweet then he/she also feels the same -positive- sentiment. Tweets that were jokes were identified, marked accordingly and removed.

As with many examples in the past, the software that was used consisted of GATE (for annotating unstructured text from Tweets) but also SPSS Clementine (now PASW Modeller). Here is the setup from GATE :

Specific rules (JAPE) were used that identify and annotate accordingly negative and positive sentiment. Consider the following sentences :

- The economy is most likely bad at the moment
- If the economy is great then why so many people can't find a job?

The first sentence has clearly a negative sentiment since the word bad exists. However the second phrase contains the word great so a specific matching rule should take into consideration the word If and annotate this phrase as one having negative sentiment despite the presence of word great.

After running GATE here is how the -now structured- data look like from a smaller sample of the original dataset (notice the highlighted record and the IfGood flag) :

With data in a structured form as the one depicted above we are then ready to identify which Tweets were found having a positive or negative sentiment, see erroneous annotations , take corrective actions and finally analyze the information and extract knowledge from it.

Mining the Tweets

2009-10-12T10:21:00.010+03:00

I received through my Google Alerts a very interesting article : Twitter is in talks with Microsoft and Google regarding the use of Data Mining technology on user Tweets.

Despite the fact that Twitter execs do not appear so eager in making the deal as soon as possible, these news clearly show where things are going. If and when the deal is finalized it will be very interesting to see :

1) What kind of Data and Text Mining techniques will be mostly used? Which of them will prove useful?

Many examples of what can be done in terms of Data and Text Mining application on Twitter were given in this blog (starting from January 2009). In my opinion, types of analysis that will prove to be interesting -apart from Sentiment Mining for Products and Services which is already taking place- are Cluster Analysis (see post "Clustering the Thoughts of Twitter Users" here) and Prediction of Virality.

Although Twitter will be able to monetize through insights extracted from Cluster Analysis and Opinion - Sentiment Mining perhaps the most important analysis is finding patterns in user emotional states. Recall that everything needed for such an analysis exists in user Tweets : Life Events, thoughts and their associated emotional states. What emotions drive people in making several decisions such as which Product to buy or which Politician to support? What kind of feelings are generated during a bad economy? Perhaps by analyzing Tweets we could understand people (and thus consumers) in entirely new ways since this is the first time that this information is available to us.

2) How will Twitter users react when knowing their Tweets are being analyzed?

My first impression is that Twitter users do not care too much if companies extract the insights discussed above however this does not mean that people's opinion will stay like this. Again, user reaction on this matter is something that could be changed anytime and should be looked at closely.

3) Which other technologies will be mostly sought?

Although no one can give a definitive answer, i would likely expect Natural Language Processing (NLP) and Ontologies to be also heavily used and sought as expertise.

Surviving Cancer, Happiness and Twitter

2009-08-09T21:14:00.015+03:00

Twitter is a great source of information on how people feel and how they behave. In previous posts we have discussed several examples of extracting from Twitter posts the feelings of Twitter users, their beliefs and values.

My latest analysis goal was to extract specific life events (such as the birth of a child) and the associated feelings and emotions of such an event.

First i wanted to identify life events associated with happiness. To do this i used text classification and a great piece of software called GATE. The data used originated from tweets of 60K Twitter Users and their biographies.

After completing the analysis, several "patterns of happiness" emerged but i believe that there is one that deserves a post on its own and should be disclosed : One of the most happiest groups of people on Twitter are cancer survivors. I was really amazed to find out that these people who faced -and possibly still facing- this life threatening disease were amongst the happiest people on Twitter and used very frequently words expressing happiness, satisfaction and blessedness.

I do believe that Twitter is a huge source of information and insights for Marketing, Branding and PR. It also appears that by analyzing Tweets we could also learn some important life lessons as well.

More to come soon.

A computer program predicts Viral Tweets

2009-08-04T12:55:00.023+03:00

In the previous post we have seen that the author of a Tweet is the most important factor for making a viral Tweet. This time we will use Text Mining to score Tweets and see how much viral they could become. Each Tweet is fed to a computer program (an algorithm) and the algorithm responds with the probability each Tweet has to become viral (we assume that when a Tweet receives more than 30 Re-Tweets it is considered viral).

The information that is given to the algorithm is the Text of the Tweet and its author. Many other parameters can be taken into consideration such as the time that the Tweet has been posted, the type of the Tweet (ie. politics, technology, health, etc) or even whether this Tweet is part of a novel subject. Here is the output of the software that performs the predictions :

The number of Re-Tweets is shown in squares. Pay also close attention to the circled text shown above. For each Tweet the most probable outcome is given ('t'= Tweet will become viral, 'f'=otherwise) and a confidence for each prediction is given as a number from 0 to 1. As an example, the first Tweet shown above was posted from Paula Abdul saying that she will not return to American Idol. The algorithm predicts with a confidence of 63.38% that what Paula Abdul posted will be interesting (and it actually was).

The predictive model has an overall accuracy of 72.88% in predicting which Tweets will be viral in a total of 59 Tweets. An example of an incorrect prediction can be seen at the 4th circle from the top. The algorithm gave a 53.66% confidence that this Tweet will not become viral but actually this was a viral Tweet.

You can find the text file of the actual run from the algorithm here.

By looking the text file, results metrics such as TP (True positives) versus FP (False positives) can be calculated. It is also interesting to see how the algorithm switches to negative predictions when the number of Re-Tweets of each Tweet become less than 30.

Even though the example given here is very simplistic -and optimistic-, the application of a tool of this kind for PR, Marketing and Branding could prove very useful. Marketeers can try different messages and see what impact each message is likely to have. Consider the following run that shows that @mashable is more influential than @lifeanalytics :

The following run shows that specific keywords raise our chances in making a Viral Tweet :

In theory this information could provide the basis for performing A/B tests : One could simply use the 2 messages shown above and record what impact each one has using Google Analytics (a process which could prove whether this technology works or not).

Finding information that is interesting to masses is actually a much harder problem. Twitter is a data source that is biased for many reasons : Specific people can pass their messages with great ease and Twitter is used by specific population segments. Almost a week ago i came across reddit and i believe that this site (and also Digg) is able to capture the preference of masses in a more efficient way than Twitter. The truth is that the available information from forums, blogs and many other websites can capture different aspects of human behavior. All that is needed to extract useful knowledge is an efficient blending of these facts, emotions and beliefs of people from different web sources.

Predicting the next Viral Tweet

2009-06-30T15:52:00.031+03:00

It is time to use Twitter data for another reason : Can Predictive Analytics be used to identify which tweets have an increased probability to become viral?

First we have to identify the problem and see what information we should consider. Every Tweet has an author, a content and is posted on a specific day and time. More specifically, for every tweet we can collect usage data such as

Day of Post
Time of post
Elapsed minutes since tweet has been posted
Author of tweet (Twitter username)
Number of followers of the author

and also information such as :

Subject of post
Whether the tweet involves a question being asked
Whether the tweet contains hashtags
Whether the tweet contains a "Please Re-Tweet" directive (or variants)
Whether a user is mentioned
The text of the tweet itself.

Our goal then is to combine the information mentioned above and come up with a predictive model that when given an author, day, time of post and text of the tweet it will be able to tell us whether this tweet has an increased probability to become viral.

For this Data & Text mining exercise (and keeping in mind that tweets have been sampled from one website and not Twitter itself) let's define what is a viral tweet : After collecting approx. 8000 tweets from dailyrt.com it was found that the median value of Re-tweets is 17. Here we make the assumption that if a tweet exceeds 30 Re-tweets it is considered viral (and actually this specific assumption makes the classification task much easier).

As discussed above, usage data do not tell us anything about the content of a tweet. Usage data tell us about the name of the author, his/her followers, when the tweet has been posted and how many minutes elapsed since its post. Can this information alone predict whether a tweet will become viral? A data mining model predicted (without using the elapsed time as input field) with an overall accuracy of 75.03% whether a tweet can be viral and -perhaps as expected- shown that the most important factor for making a viral tweet is its author. Running a process called Feature Selection tells us just that :

But what we have seen so far only tells us one -the Data Mining- side of the story. With Text Mining we can see the importance of words and authors. To do that, each author is appended at the end of each tweet (so essentially the author becomes a part of each tweet text). Here is what Feature Selection tells us :

A Tweet mentioning Michael Jackson has a great probability of becoming viral but perhaps it should be also posted by a popular author to make a greater impact. Pay attention also to the fact that @mashable and the @theonion are on top of our feature selection list shown above.

The difficult -but also interesting- task is to predict a viral tweet that has an impact not because of its author but because of its content and to do this the methodology of data collection and analysis differs significantly.

On the next post we will see a model predicting viral tweets in action : We will submit several tweets and their author and the model will tell us the probability that each submitted tweet has to become viral.

How Habitat UK should have used Twitter

2009-06-23T14:05:00.011+03:00

Following the great post from Tiphereth Gloria i wanted to take the opportunity to show an example of how Habitat UK should be using Twitter.

My suggestion would be that instead of the "initiative" they took they should identify the values, beliefs and needs of their customers by capturing and analyzing relevant tweets instead. And here is how they could do it :

First they should capture all relevant Tweets every -say- month :

The second step would be to identify what people want when they talk about furniture. If they used Text Mining they would have found specific furniture products that customers want to buy and the values associated with these types. For an example look at the following table :

The table shows us (pay attention to dark red cells) that customers looking to buy baby furniture have Safety as their number one associated value. With this knowledge then perhaps Habitat UK would make sure that when they advertise Baby furniture they would use this word on their advertisements to capture the interest of their customers. Of course what was shown above is not some new information but is meant to be given as an example.

Some more things that Habitat UK could have done with Text Mining would be to see :

How important it is to suggest solutions to customers

Which rooms people want to re-furnish more often and -more importantly- why.

How problems (such as furniture received is damaged or difficult to assembly) affect their brand.

How people feel excited when they wait for their new furniture...and how bad they feel when furniture is not delivered on time.

There is much more that can be done : By running Cluster analysis many kinds of customer thoughts can be grouped together : One of them was how much "Feeling good" is closely related to new furniture and how it affects people's psyche.

By using Social Media Analytics, Habitat UK -and most other companies- would understand their customers better, see what is important for them and with this knowledge they would be able to take informed decisions that would -most likely- make a real difference.

How people use Twitter - 10 distinct usage groups

2009-06-19T11:10:00.018+03:00

During this post we will be looking at another example of cluster analysis performed on Twitter. The analysis was performed on 17000 Twitter users with the goal of extracting distinct groups of usage which essentially shows us the different types of Usage behavior of Twitter users. The following parameters were taken under consideration :

Number of Followers

Number of Links posted per 20 Tweets (not during RT)

Number of Updates

Elapsed Days

The following table shows the results :

Note that each cluster has a specific number from 1 to 10. Clusters are listed according to their size which means that cluster "10" is the largest usage group, while cluster "5" being the smallest.

Let's see what the table tells us, starting with the first line : Cluster 10, is the largest (=more frequent) type of usage behavior. Users of that group have an average number of followers, have been using Twitter for relatively many days (elapsedDays=high) ,have a high number of updates while the number of links they provide per 20 tweets is average - say around 3 links-

Now consider -highlighted- cluster 8 which we will call The Information providers : Notice that even though this group of users have relatively few elapsed days and average number of updates, they achieve a High number of followers. The reason is that these users provide a large number of links per 20 Tweets ( Note that this confirms findings during a previous analysis).

See also cluster 3 : Even though this group of users has been on Twitter for many days but also has a high number of updates, it appears that it pays a price for not providing links.

Recall that the "#OfLinks" parameter counts only these links that are NOT part of a Retweet. This tells us that users that are able to find original content and provide it to the community tend to gain more followers.

This analysis was given with the aim of providing a simple example and should not be considered as a detailed analysis since few parameters have been taken into account. Cluster Analysis on Twitter data (which include things that people like doing, professions, interests, marital status, mention of products or opinions to name a few) can -potentially- give us excellent insights on different aspects of user behavior.

Social Media, Corporate Decisions and Analytics

2009-06-04T09:32:00.062+03:00

Over the past 6 months we have seen real-world applications of Data and Text Mining applied on Social Media Data from Twitter. We went through many examples that look at Social Media Data in different ways :

We identified what Twitter users don't want, grouped their beliefs and also ordered all of this information accordingly

We identified which usage behavior increases our chances of having a large number of followers (if a large number of followers is our goal)

We found which words appear to be associated with a large number of followers. (We have seen that negative thinking and words in Tweets possibly drive people away)

We extracted segments of Twitter users with similar characteristics.

The list of possible applications does not end here. Over the next posts we will also discuss about :

Predicting whether a Tweet has the potential to become "viral".

Associating specific events and user emotional states.

To recap : A computer program is able to monitor the words - phrases that you say and your emotions, flag them as positive or negative, track the rate with which you increase your follower count, track the number of updates, Re-Tweets, replies, hashtags, smileys and questions that you make, flags any mentions about products and services and assigns you to a predefined segment of users sharing similar behavior and interests. Then for each segment its "social media fitness value" is identified (by looking at the follower count).

Usage of Google Wave will possibly reveal other insights : Due to the fact that the sequence of posts will be easily extracted then we could also take under consideration the number of consecutive posts who had a positive sentiment and whether these positive posts appeared at the beginning, center or the end of each thread's sequence. We could also look at the number of posts -that are part of the same thread- having videos or pictures attached and ultimately identify how all of this information may affect one's point of view. Of course I am not certain whether such a scenario could prove useful. I sure would like to try though.

We are presented with a unique opportunity to understand people much better than before and with the examples shown so far this should be more clear by now. Predictive Analytics is about extracting knowledge and identifying what is more likely to work. As Ian Ayres put it in his book Super Crunchers, Decisions are beginning to be based even more on facts and less on intuition. It appears that Social Media Analytics will play an important role in making Corporate decisions for PR, Branding and Marketing and this will happen through better understanding of human behavior.

Twitter Analytics : Cluster Analysis reveals similar Twitter Users

2009-05-21T21:38:00.051+03:00

So far we have seen various examples of using analytics to gain insights from Twitter. Using cluster analysis is a personal favorite : It enables us to identify common groups of users and in this post we are going to look at a segmentation based on user biography keywords. This analysis was also presented in an older post but some readers asked me to elaborate a bit more on this type of analysis.

Biography information allows us to segment Twitter users in groups of similar interests, professions and qualities. What is more interesting however is that we can identify the words that each segment appears to be associated with. Let's see an example of words that tend to co-exist with the phrase "social media" in the Biographies of Twitter users :

By looking at the column named "social_media" we see some associated keywords like : addiction (synonym for addicted, junkie etc), evangelist, enthusiast, analytics etc.

Other groups found and their associated words were :

The Geeks : Developer, Linux, Mac, gaming, photography
The Parents : married, boys, girls, christian,conservative
The business owners : CEO, entrepreneur, marketing, founder, lifestyle

Note that "The Geeks" have Mac as an associated keyword which of course refers to Apple Macintosh : An example suggesting a possible strong bond between a brand and a specific customer segment.

Now imagine running a similar analysis for other segments such as Single Dads and Mothers, Teenage Girls, Nice Guys, IT Developers, VIPs or any other "segment" you prefer (see this entry -posted Jan. 2009- for more)

On a personal Note : Having used Text Mining on Twitter over the past 6 months i realized that whenever a new cycle of analysis is made i come up most of the time with things that i already know. But apart from expected results some of the fine details of people's lives also appear such as the implications of a life-changing event, the joy of owning something new or the plain fact of "watching TV and feeling bored". Many of the insights found during these months -although not discussed here on purpose- are highly thought provoking.

Perhaps Twitter Analytics could also give us some possible clues on:

Whether a specific profession could be a risk factor for being single.

How important is fashion for girls.

How mobile phone user requirements change according to the "segment" they belong to.

What are the most common things that people don't want.

Finding individuals that do not fit any "segment".

But the list of potential applications does not end here : Using a technique called Association Rule Learning (or Association Discovery) we can extract emotions or thoughts that appear to co-exist and also emotions that seem to be associated with specific events. Classification Analysis can also play an important part (more on these techniques soon).

Each technique looks at the Social Media Data world from a different perspective. The usage behavior, cluster membership, the emotions and thoughts and also the Tweets that users seem to prefer most (using data from sites such as repeets.com) may be combined. What we can potentially achieve from a combined analysis of this kind will be discussed in later posts.

As already stated in previous posts: The use of the methods described so far enables us to form hypotheses but in no way it is assumed that associations found are the definite cause of a specific event.

Twitter Analytics : Bio information and popularity

2009-05-15T09:16:00.059+03:00

In the previous post we identified words used in Tweets that appear to be associated with low number of followers : We found that when someone uses foul or negative language then his/her follower count appears to be affected negatively (see here for more).

It is time to identify the words contained in the biographies of popular Twitter users and to be more specific the biographies of users being in the top 30% (in terms of no. of followers) of a random sample of 10000 users. As i always have stated in these series of posts : Treat results as possible clues only. Please also notice how i used (in this and older posts) the words "appears" or "were found" when discussing correlation. The technique shown is the same as discussed in the previous post. Results are as follows :

Student appears to be correlated with low popularity accounts.

Engineer also appears to exist often in low popularity accounts although the correlation was not found to be as strong as for students.

Common words existing in popular users Bio appear to be the following : social, media, marketing, CEO, founder, author, entrepreneur, blog, twitter, news, writer, internet.

Some comments :

It is not suggested that by having specific words in your bio, you will get more followers. Many other things are and could be important in achieving a high follower count. Same applies for unpopular accounts.

Looking at the results i wondered why students were found to be associated with low follower numbers and i think that this requires more attention. One possible reason could be that students might be spending most of their social media time on FaceBook or other SM sites. There can be many pitfalls in performing a random sampling from Twitter and "Students" could be one of these cases. However please share your comments.

Notice that some words that appear to be associated with high follower numbers are words that communicate authority ( ex. founder, CEO).

To recap from the last 3 posts :

1) Do not use foul language - keep your conversations positive.
2) Use "Thank you" often. "Stay tuned" seems to work well also.
3) Post frequently. Posting some links is also important.
4) Make sure you have a good Bio filled in.

Finally, if you find the contents of this blog interesting you can always have a look for more updates on my new account on Twitter @lifeanalytics and also send me your suggestions and/or comments.

Twitter Analytics : These words may be affecting your popularity

2009-05-06T00:59:00.106+03:00

Text Mining techniques can be used to identify specific words that are correlated with Twitter accounts having high or low popularity. This can be done in two ways : (1) By analyzing the text of the Tweets of each user and (2) By analyzing the text of the biography of each user.

Let's start with the results of the first type of analysis with data originating from user Tweets. Pay attention only to cells that are highlighted in red, their corresponding category column (LOWFOLLOWERS , HIGHFOLLOWERS) and the word at the beginning of each corresponding row. Results show which words appear to be important especially because the affinity shown here is moderate. Use results as possible clues only.

The results so far show us that :

hate, bed : are found to be correlated with low popularity
top, online, send, list,web,media, join : with high popularity

Here is another portion of the results table :

The pattern should be evident by now : Words of negative attitude appear to be influencing a user's follower count negatively. As also shown above, foul language appears to work negatively also. Several other insights were found such as the existence of specific phrases that are correlated with low popularity ("watching TV") while other phrases ("stay tuned" ) with popular accounts. The number shown in parentheses quantifies the magnitude of the association that each word has and thus enables us to order words by their importance.

Some of the words -and their synonyms- that were found to be associated with very low follower counts are :

- Sleep, Hate, Damn, Feeling, Homework, Class, Boring, Stuck

A total of 63 words and 25 phrases were found having either a positive or negative association with the followers count. Interestingly, specific phrases that communicate any kind of opportunity are also associated with high number of followers. "Thank you" is highly related with a user's large popularity.

Here comes the interesting part : Once the Text Mining analysis is completed, a predictive model can be generated that may be used for scoring future Tweets. Let's assume that you are about to send the following 2 Tweets :

1) 'Today i feel like sleeping all day. Yawn...'
2) '@xyz Your website traffic can be increased with good marketing'

Before you post however, you decide to feed these 2 sentences to a predictive model. The predictive model returns for every Tweet the predicted result (GOOD or BAD) and the associated probability. Here are the results for these 2 examples from an actual run :

In other words :

1) The first Tweet may have a negative effect with a probability of 83.5%
2) The second Tweet may have a positive effect with probability 99.9%

Note that :

A predictive model is able to consider combination of words, not just single words. This raises considerably the accuracy of any prediction.

In any real world application of Text Mining a 100% prediction accuracy cannot be achieved: Although application-specific, a 72-78% accuracy may be achieved - with considerable effort. Of course many more things are important to achieve high popularity and the example above is given merely to discuss what techniques currently exist. A combination of analytical techniques is the best option and this will be discussed in a future post.

Several other types of analysis can extract similarly interesting insights : Let's not forget that Twitter Tweets contain the emotions, beliefs and values of users. They contain what people want and what they don't want. See Clustering the thoughts of Twitter Users and Know your customers the Twitter way for a further discussion on this.

There will be more to say about Text Mining and how it can be put to use by PR Agencies and Marketing companies with practical examples shortly.

Twitter Analytics : Which usage behavior attracts many followers?

2009-05-03T23:25:00.032+03:00

This is the first part of a series of posts where Data Mining and Text Mining will be applied to extract potentially useful facts about the usage of Twitter and to draw some conclusions such as what makes a Twitter account interesting enough to other users.

The conclusions that will be presented here are from the analysis of 3651 Twitter accounts and are meant to show how Predictive Analytics can help. Please note that results are shown for informational purposes only.

First, the data used can be summarized with the following table :

You can immediately see problems in the ranges of the data used especially on the number of "followers" and "following". This is something to be expected since among the users captured were Jack Dorsey (founder of Twitter), Sen. McCain and George Stephanopoulos - users that obviously have a huge amount of followers.

Before finding which usage behavior attracts many followers, one should be able to identify what exactly is a "popular twitter account". Is it just the absolute number of followers? Perhaps it could be equally important -or at least interesting- to also look at :

1) The followers/following ratio

2) The number of followers per day

For our example the absolute number of followers was used as the only criterion of a successful Twitter account. The results can be summarized with the following decision tree :

Some usage patterns that raise the chance of having a successful Twitter account are the following :

Having a bio is an absolute must : 82.3% of unsuccessful Twitter accounts have their biography information missing.

You should provide more than 3 links per 20 tweets and also more than 0.960 updates per day

If you don't want to provide more than 3 links per 20 tweets, then try to post more than 5.857 updates per day.

Users that post more than 3 links per 20 tweets but post less than or equal to 0.960 updates per day, will need more than 222.5 days of usage to get an adequate amount of followers.

By using Feature Selection we are able to look also at the relevant importance of each parameter on achieving many followers : Here are the results of Feature Selection from using ChiSquare, GainRatio and InfoGain attribute evaluators.

=== Attribute selection 10 fold cross-validation (stratified), seed: 1 ===

average merit average rank attribute
362.743 +-10.419 1 +- 0 4 numberOfLinks
319.397 +-10.133 2.4 +- 0.49 6 hasBlankProfile?
311.661 +- 8.612 2.6 +- 0.49 7 updatesPerDay
192.525 +- 7.481 4.1 +- 0.3 3 retweetsNumber
178.236 +- 5.963 4.9 +- 0.3 1 elapsedDays
36.148 +- 3.579 6 +- 0 2 otherUsersTalk
17.843 +- 4.475 7 +- 0 5 questionsAsked

average merit average rank attribute
0.1 +- 0.003 1 +- 0 6 hasBlankProfile?
0.042 +- 0.001 2.4 +- 0.49 4 numberOfLinks
0.039 +- 0.002 3.2 +- 0.6 3 retweetsNumber
0.04 +- 0.004 3.4 +- 0.92 7 updatesPerDay
0.025 +- 0.001 5 +- 0 1 elapsedDays
0.011 +- 0.001 6 +- 0 2 otherUsersTalk
0.005 +- 0.001 7 +- 0 5 questionsAsked

average merit average rank attribute
0.082 +- 0.002 1 +- 0 4 numberOfLinks
0.074 +- 0.003 2.1 +- 0.3 6 hasBlankProfile?
0.071 +- 0.002 2.9 +- 0.3 7 updatesPerDay
0.044 +- 0.002 4.1 +- 0.3 3 retweetsNumber
0.041 +- 0.001 4.9 +- 0.3 1 elapsedDays
0.008 +- 0.001 6 +- 0 2 otherUsersTalk
0.004 +- 0.001 7 +- 0 5 questionsAsked

We see that all three attribute evaluators agree that the number of links provided on Tweets and whether the profile of the user is filled in are the two most important parameters in achieving many followers. Notice also that sending messages to other users (otherUsersTalk) and asking questions (questionsAsked) is not as important as one would expect.

The analysis shown above gives many insights but it does not take into account what the users say and how this affects the popularity of a Twitter account. Text Mining will try to give some answers for this question and also identify which keywords on Twitter profiles seem to be associated with many followers.

Twitter Analytics : Words that make a difference

2009-04-27T20:39:00.015+03:00

Predictive Analytics are already widely used on Twitter to extract -potentially- interesting insights. In previous posts we discussed about :

Sentiment Analysis and Ontologies

Analyzing the biographies of Twitter users and identifying clusters of similar users.

Cluster Analysis on the thoughts of Twitter users

Identifying the values and beliefs of Twitter users.

One additionally interesting insight is the knowledge of what makes a Twitter user having many followers. Consider the following questions :

Are there words that could potentially decrease the popularity of a Twitter account?

How important is to have an actual photo (and not the default o_O photo)?

Which interests or professions tend to be associated with many followers?

How important is to have at least a small text of biography information?

To answer these questions, data from 100000 Twitter users were collected over the past few weeks. Information collected includes the number of followers, number of friends, total updates, number of Retweets (per 20 tweets), number of replies to other users, number of links to external URLs, number of months that the user is on Twitter, etc. Here is how the data looks like :

You will notice that the separator tilde '^' is used. The first portion of each line contains the user name, date of account creation, months elapsed since account creation, number of friends,number of re-tweets etc.

The first analysis that was performed was to identify whether specific keywords that exist on user biographies seem to be associated with a large number of followers. A second type of analysis was performed only with numeric data (such as number of re-tweets, number of user replies, number of updates,etc). Then a third type of analysis uses both a vector of keywords plus numerical data. Since a lot of work is needed, the process (but not all results) will be presented during the next posts.

FYI : Users that tend to use a lot the words "boredom", "boring" or "bored" tend to minimize their chances of being popular.

Social Media Monitoring with ScoutLabs - Interview

2009-03-12T12:56:00.010+02:00

During my previous posts i have shown some examples of Sentiment Analysis using Twitter. I came across a Sentiment Analysis product named ScoutLabs which is able to give insights on what customers are saying on the Web about a product or service and decided to get an interview from ScoutLabs CEO Jennifer Zeszut :

- Please tell us about ScoutLabs and how companies may benefit from using it.

Scout Labs is a powerful, web-based application that finds signals in the noise of social media to help teams build better products and stronger customer relationships.

Scout Labs is a product company, not an agency. We provide cutting-edge technology and a collaborative platform for companies and their agents to listen to customers and engage with them out across the Internet. With Scout Labs, our users:

* Know when to tune in and what’s most important to pay attention to
* Hear what customers love and hate about brands
* Reach out to influential customers to build relationships
* Engage in proactive customer service
* Let the voice of the customer inspire new product and marketing ideas

Scout Labs has grown significantly since it was founded in 2006. With offices in San Francisco and users all over the world, the company currently employs over 20 professionals. Our CEO and product team guide the application with insight from the world of marketing, brand management and product management, but the majority of Scout Labs employees are senior engineers with expertise in search technology, high-performance systems, natural language processing, machine learning, web crawling and data visualization.

- How is ScoutLabs different from other sentiment analysis solutions?

Many sentiment analysis solutions are really human-powered, which is great, if you’ve got a big budget and or a lot of time. Ours is automated, and we process millions of posts per day and score it for sentiment as it happens, with an accuracy rate (agrees with humans) 73% of the time and it will get better, because as users (across our system) change sentiment values in our system, we aggregate that data and use it as labeled data to improve our algorithms even further. We also can “backfill” or back-score 3 months of previous data with its sentiment scores in 20 minutes to (at most) 24 hours.

- What kind of information (product names, company names, areas, city names) can ScoutLabs identify in user conversations "Out of the Box" ?

Scout labs lets you track anything you like. It is purely search based (not a database of x fixed company names). We see searches on everything! Company names, product names, people, industries, “the price of rubber”, green energy, styrofoam, “tricked out shoe”, “canceling cable”, “favorite hotel in bali”-- you name it. For some of these, sentiment doesn’t make a ton of sense, but we’ll score it for you nonetheless (just in case).

- One interesting functionality is to be able to identify associated keywords by products. For example consider two different brands of running shoes such as Nike and Adidas. "Great design" might be associated with the first brand while the other brand is perceived as being "comfortable". Is ScoutLabs able to identify automatically such information between two or more similar brands-products?

We do offer an analysis of the top conversations associated with any search. This comes stright from the conversations themselves. They are the frequent words that are emerging from the conversations for the time period. And yes! Very often we find powerful adjectives that describe a brand (sometimes good, but not always). For example, when i did a search for dora the explorer (the popular cartoon character for the pre-school set), the words that emerged was “skank” “sexy”. Huh? Sure enough, parents are outraged by a mattel announcement about how they are going to have dora grow up, move to new york and get fashionable (with a short skirt). This is a true story: http://www.Scoutlabs.Com/2009/03/12/scandalous-doll-drama-scout-labs-style/

- How much should one expect to pay for such a service?

A single team with 25 searches (which is where most companies start) is only $249. For now. We have only committed to this low price for the first 1000 companies to try us. That price may go up one day soon. All our pricing plans are here: http://www.Scoutlabs.Com/plans/

Making more sense out of Twitter Tweets

2009-02-23T13:49:00.033+02:00

Over the last 5 posts i have described how unstructured text information from Twitter can be used for Knowledge Extraction. Specific examples were given such as Sentiment Analysis for products (Amazon's Kindle), Segmentation of Twitter users, and finally cluster analysis of the emotions and thoughts expressed from twitter users.

So far i have discussed some ways that text mining could help us in getting more insight on how people think. Now it is time to put Information Extraction and Ontologies to the equation.

Information Extraction (IE) is the automated extraction of any information such as (to name a few) Names (first names, city names, country names etc), facts or events from unstructured text. An example of IE was given in these posts where thousands of adverts of flats are extracted and then data mining analysis is performed to identify what characteristics are important for achieving a high renting price.

Ontologies are used for knowledge representation and may also be used for structuring the information that exists on the web. To give an example, consider the following product keywords :

Coke

Sprite

Dr Pepper

If one asks you what is common about them, your brain looks for generalizations and comes up with the following answers :

They are all Carbonated Drinks

(Possibly) they all contain sugar since the word "Diet" or "Zero" or "Light" is not mentioned.

Now let's assume having an Ontology Engine that is able to do this and to be able to infer automatically that all these products are sugar-carbonated drinks. Such an action enables us to extract facts in a more coherent way. The reason behind this is that we lessen the effect discussed on The Statistics of Everyday Talk and thus are able to capture growing trends such as people expressing their thoughts regarding carbonated drinks rather than matching "Coke", "Sprite" and "Dr Pepper" individually. Without Ontologies such a trend could be easily missed.

By using Ontologies or taxonomies where applicable, an associations discovery algorithm can search in different levels of information detail. For example data miners usually employ taxonomic information (ex. Sprite, Coke, Pepsi = carbonated drinks) when performing associations discovery analysis on Super Markets and the effort of applying taxonomies almost always pays back in terms of the knowledge extracted regarding consumer behavior.

I have used Ontologies over the past 3 years and have seen them in action. The fact that with Ontologies one could possibly have access to inference and deductive reasoning techniques is of great use. The application of Information Extraction, Natural Language Processing and subsequent insertion of this information in an Ontological setting has many potential applications.

Know your customers - The Twitter way

2009-02-15T12:40:00.024+02:00

The more i analyze tweets on Twitter, the more interesting i find the whole process. First it was Cluster analysis of specific thoughts expressed from Twitter users and then it was Sentiment Mining for Amazon's Kindle. It was just a matter of time from having the urge to analyze Tweets on a broader perspective.

So i decided to perform a segmentation of the Twitter users : extract common groups of users but this time not for specific thoughts or specific products but a segmentation based on a more generic basis.

I had two goals in this cluster analysis :

1) Cluster the biographies of users
2) Cluster the tweets of the users.

I then decided that the more information i could collect the better, so the first thing i did was to make a 'spider' program to extract 10,000 twitter user names. Then for each twitter user the software visits his/her page and extracts :

a) The user's bio
b) Number of followers
c) Number of people following
d) Number of updates
e) 20 latest Tweets
f) Number of re-tweets
g) Number of replies to other users (ex when @user directive exists)

Let's see now what we could -potentially- do with such information :

1) Cluster analysis on user bios

2) Cluster analysis on user tweets

3) Classification analysis for identifying the common characteristics of users with many followers

4) Associations discovery between products : Which products tend to be mentioned together in each user's tweets?

5) Identification of common keywords per cluster : If we identify a cluster of users that we characterize as the "Parents", what keywords do "Parents" tend to use more? What about the "Tech junkies" cluster?

But let's start with the first analysis : Clustering the biographies of Twitterers. The analysis generated 30 clusters of users. Some of them are :

1) The Parents
2) The computer Geeks
3) The students
4) The social media addicts
5) The entrepreneurs

I looked at the "Parents" cluster more closely and wanted to find keywords that this cluster is associated with : Single and Jesus where some of them.

So we immediately identify one of the many customer groups : The parents, of which a significant percentage of them are single. The "Parents" cluster also expresses one of its values : Christianity.

By moving on to each generated cluster and finding the associated keywords, i was able to retrieve the values and beliefs of each cluster. Knowledge Extraction at its best.

Sentiment Mining for Amazon's Kindle

2009-02-11T16:32:00.020+02:00

Following the post on Clustering the thoughts of Twitter users, it is time to look at another example where Twitter can be used. So i decided to analyze -just- 1054 tweets that are about Amazon's e-reader kindle to see what i could come up with.

My goal was not to classify between positive or negative sentiment but to extract the general "buzz" about the product by means of cluster analysis. After extracting the tweets that contain the word "kindle" i continued in removing non-relevant information (such as tinyurl links) by using regex expressions.

Next, it was time to understand the data and a good way to do this is to look at word frequencies using TextStat. Here is what i came up with :

On the top of the word frequency list are the usual suspects : "I", "and", "to", but also "kindle", "kindle2" and "amazon" which is something that was expected. Now, let's see what are some of the words that do not occur frequently :

Here appears a fact that requires attention : Text miners use stop-word lists to remove the most frequent words but they also remove words that do not occur frequently. The table above shows that a non-frequently occurring word is disappointed and if we had chosen to omit words of a specific frequency range -such as less than 3- we could loose this important information. So caution is needed.

After running the analysis, i came up with 20 different clusters of similar "thinking". Note that we are not only interested in which those clusters are but also -more importantly- to the proportion of cases that each cluster contains (see previous post). Some of the examples of clusters found are :

1) A cluster of users that are questioning the usefulness of the product
2) Excited users
3) Users that are happy about the text-to-speech recognition feature of the product
4) Text-to-speech recognition and potential copyright issues

Twitter is a great source for sentiment extraction but one problem is the fact that people are re-tweeting the same news (" The new Kindle 2 is out") or they tweet about similar information from various tech news websites.

Clustering the thoughts of Twitter Users

2009-01-21T02:51:00.054+02:00

During the last two posts i presented the reasons and some problems on analyzing the thoughts of users on the web and particularly Twitter. (For more see Part1 and Part2 ).

As an example, we are going to be looking at a specific kind of thought that Twitter users make : What they don't want. By using the Twitter API i managed to extract all tweets having the phrase "i don't want to". The following text file shows the results :

The next step is to remove all phrases that do not give us any information about what users do not want :

Finally we remove the phrase "i don't want to". However, consider the following example:

"I must go to Chicago. I don't want to do that"

The steps discussed above will discard the first sentence which is actually what the user does not want to do and leave only the phrase "i don't want to do that" which is not particularly informative. At this point we must quantify the problem -let's assume it involves the 8.5% of our records- and recall what the pareto principle is all about.

After some additional pre-processing steps which are not discussed here, i feed the data to K-Means to see the clusters the algorithm comes up with. For a better presentation of the results, here is a screen capture from IBM's UI Modeler :

We immediately see -in descending order- what Twitter users do not want :

1) They do not want to go to work
2) They do not want to go to school
3) They do not want to hear about various issues
4) They do not want to buy things

Notice also the top two categories named Miscellaneous and None. These categories contain thoughts that have a very small frequency to form a cluster. These two categories consist the 69.56% of our records and at this point we should think again about the pareto principle.

Please note that not all necessary work is discussed here and i had to omit several actions that have to take place. In trying to understand what people actually think i am using an approach which uses Ontologies, Information Extraction, Clustering and Classification analysis with the ultimate goal to minimize the percentage of thoughts (69.56% in this example) that cannot form a cluster and to increase the accuracy of the analysis.

It is also an interesting fact that we could move further down the sentence branch (see this post) for even better insight. Here i presented a cluster analysis about what users do not want. As an example we could apply clustering on user thoughts specifically for "I don't want to feel".

The Statistics of Everyday Talk

2009-01-15T18:52:00.027+02:00

As discussed in the previous post, the analysis of free text on the Web -and as an example the thoughts expressed by Twitter users- could extract very interesting insights on how users think and how they behave.

In 2001 i visited Trillium where i had a very useful seminar on Data Cleaning, Data Quality and Standardization during which the pareto principle became -once again- evident. When someone wishes to standardize entries in a Database so that the word "Parkway" is written in the same way across all records, he might find the following distribution of "parkway" entries :

15% of records contain the word "Parkway"
3% of records contain the word "Pkwy"
0.2% of records contain the word "Prkwy"
0.01% of records contain the word "Parkwy"

What that essentially means is that with a single SQL query one can find and correct 15% of "parkway" word synonyms to whatever standardized form is needed. But for the remaining variations one query solves only a very small fraction of the problem and this in turn increases the amount of work required, sometimes overwhelmingly.

In capturing and analyzing natural language we are confronted with the same problem : 60% of people might be using the same phrase for describing the fact that they don't want to go to sleep with a simple "I don't want to go to sleep". But another 20% might be using something like : "i don't feel like sleeping" and another 10% something like "i don't want to go to bed right now".

So we immediately see one of the issues that Text Miners face : The fact that we can use different phrases to communicate the same meaning. If we wish to analyze text information for classification purposes -say the sentiment of customers- we could achieve a 60-65% accuracy in our results with some effort. For a mere 4% increase in accuracy -from 65% to 69%- the amount of extra effort required could prove prohibitive.

Consider the following chart :

These are all examples of phrases people use in their everyday talk. We can visualize such phrases starting with" i don't want to" and then each branch adds a new meaning to the phrase. So branches marked with numbers are the parts of speech that give us an idea of what a person doesn't want to do : To go, to feel, to visit,to know. Things are getting much more difficult in terms of the effort required if we wish to add more detail -and probably insight- to our analysis by moving further down the branches in our sentence tree.

Perhaps for marketeers, the ability to quantify the distribution of words on the 1st level of the tree depicted above could be enough : If we end up with the following words distribution :

To feel : 15%
To know : 7%
To go : 1%
To visit : 1%

Then, we get an insight on which words to use to market products more efficiently.

On the next post we will go through a hands-on example of analyzing the thoughts of Twitter users and specifically what people seem to "don't want".

Emotions, Beliefs and Analytics

2009-01-05T18:34:00.007+02:00

When i first came across Data Mining and Machine Learning in 1998 i had no idea of the kind of applications that this field can have. As time passes by, the knowledge that can be available to a data/text miner becomes more and more a serious business....actually, a very serious one.

Not long time ago i have seen a presentation where a map of emotions from the web was created in real time by aggregating specific keywords from blogs and forum posts. Twistori is an example of such an application. Now, let's take this idea one step further.

Twitter is a "social messaging utility" in which users describe what they are doing -or what they are feeling/thinking- now. Users are able to send "tweets" even through SMS messages. The way that these messages are written is an ideal format for text mining : Short phrases that summarize what a user wants to say are a text miner's paradise.

It is logical to assume that Text mining and Information extraction techniques will become more important, since more data will be generated in the future. It is only a matter of time until the next "killer app" like FaceBook, YouTube and Twitter appears. Data/Text miners will be able to identify common "thought clusters" of people.

Now, consider the following example : By visiting this link you will get a list of people that have written in their "tweets" the phrase "I don't want to....".

Once this textual information is captured, preprocessed and then analyzed through cluster analysis we could end up with the following clusters of "I don't want-er's " :

- The cluster of users that do not want to work again/tomorrow/today (18.5%)

- The cluster of users that do not want to go to sleep (6%)

- The cluster of users that do not want to hurt someone (4.2%)

What is also interesting, is the ability to quantify the proportion of cases belonging to each cluster to the total of tweets. As shown in the example above, the most frequently occurring thought is from people that do not feel like working.

Now in the same way one could perform this type of analysis for :

"I Believe...."
"I wish i...."
"I want to buy..."

Essentially, what we are talking about is the extraction of the values, hopes and beliefs of hundreds of thousands -or even millions- of users...and in descending order. Once a first run is performed and clusters are extracted one could run this process again every month and see the trends of those clusters in time. It would be also interesting to see how these thought clusters change after specific World events.

For some people such as marketeers and social researchers -providing that results are accurate enough- this information is invaluable. Others, might feel that such an analysis is bad practice. Of course, there are companies that already capture brand sentiment across the web : Crimson Hexagon and Twitrratr are just two examples.

This post is the first in a series of posts discussing the application of Analytics to capture the thoughts that -as we speak now- exist on the Web. We will go through ways that one could explore this information and more specifically we will look at :

How clustering can group people's values, beliefs and emotions.

Why Ontologies and Natural Language Processing are needed for better results.

How classification analysis might give us knowledge on what are the common characteristics of various 'categories' of users.

Personalizing your RSS Feeds

2008-12-18T22:24:00.022+02:00

78.8%. This is the estimated accuracy with which an algorithm is able to predict what kind of information i like to read and thus what kind of news seem to be more interesting to me.

The system proves itself every day and it actually helps a lot because it can spot instantly the information that i like among hundreds of RSS news headers. Such service is so much more than simple keyword matching because it takes into account the combination of words and thus it can differentiate news (in terms of how interesting they are) even if these news are about similar concepts.

Personalization of RSS feeds is a well-known application of text classification. The amount of information -the header- is almost always 2-3 sentences long which makes it ideal for feeding it to a classifier. The software that i built is quite simple : First i have a list of about 10 RSS sources : Financial, Medical, International News, Tech news etc. The application scans the RSS feeds every 20 minutes and each new header is appended to a text file on my hard disk.

When i ran the application for the first time, i simply saved all headers on the hard disk and built my first text classifier. But that was back then. Today the classifier scans the RSS feeds and automatically appends an RSS header to either the "Interesting" text file or the "Uninteresting" text file...and it does so correctly most of the time.

When i have some spare time, i look at the classified headers and correct the errors my classifier made by putting the right headers to the right place. I then re-train the classifier and everything is ready for the next run.

Also interesting is the fact that, the more i am using the classifier the better it gets in terms of the training frequency it requires. During the first week of usage i had to produce a model almost every day...but not any more.

So RSS personalization is an application to look at. First of all it saves a lot of time. Second, really useful applications can emerge. For example, consider one investor that wishes to know if something significant enough occurs on the news that might affect the markets for better or for worse. Notice that on a previous post, i described how i am flagging every news header as an important or unimportant one. Therefore if a classifier is able to differentiate accurately enough as to what is important, then an investor can receive e-mail alarms -or even SMS messages if he is not online- of the event. Perhaps, the message might even include how much a stock or an index is likely to be affected by the breaking news.

There is also the personalization that results from collaborative filtering. However, i believe that a "personal news classifier" -if i may call it like that- after sufficient training time, does a much better job in terms of its predictive accuracy.

When Telecom customers complain-Pt. 2

2008-12-09T01:14:00.017+02:00

On the previous post i explained the first steps in deploying Information Extraction, Text Mining and Computational Linguistics to capture the essence of Telecom customers complaints.

We have already discussed about the big picture : Retrieve data (essentially user messages from forums) and then use Information Extraction to transform unstructured information to a structured form. This transformation is done by building a set of matching rules for specific phrases or keywords such as

-signal
-antenna
-customer care

and words of sentiment such as

-worse
-worst
-better
-best
-outraged

among many others.

So here comes the interesting part : Suppose that a telecom company has in its possession an application that is able to search and extract sentiment from unstructured information. Having such a tool means that :

A user can query directly on user forums for -example- specific network problems and break down those problems by area name.

A user can directly query for hot phrases such as "canceling my subscription" and cluster keywords around those messages. If the telecom company is also running (and most likely it is running) churn prediction models, then analysts have yet another source to cross-check and/or enhance the conclusions of their churning models with this new information.

Special matching rules can be applied to extract why users prefer company XYZ over company ABC.

This technology can be applied to e-mails and/or free text complaints to the customer care center, which means that analysts can further enhance their churning models with additional data.

Matching rules can be built that associate keywords to Telecom companies in terms of their co-occurrence. So telecom company XYZ has the phrase "good signal" associated with its brand whilst company ABC has the phrase "bargain" as the associated keyword.

Match billing plan keywords and then cluster them with sentiment keywords. In other words, how do customers perceived the new billing plan and what is the sentiment about it?

It is easy to realize that Information Extraction combined with Text Mining and linguistics is a powerful combination that can extract many "knowledge nuggets". The fact that such an application cannot be 100% accurate may arise acceptance problems but its sure worth the effort in the end if potential problems are clearly presented before implementation of this application.

Let us not forget that a complaint given by a customer to the customer center remains there - between the boundaries of the company. A complaint posted on a forum can be seen by hundreds of thousands of others (and it will most likely stay there for a long time ) ,influencing potential and existing customers in a non-positive way.

A Sentiment analysis application may be also used for :

Banking

Pharmaceuticals

Insurance

Consumer Products (Customer Reviews)

.
and of course for capturing the sentiment of citizens for politicians (...)

When Telecom customers complain

2008-12-07T16:48:00.008+02:00

Probably one of the best uses of Information Extraction, Text mining and Computational Linguistics combined together, is their ability to show us the sentiment of customers. Today we are going to see an example for capturing the sentiment of Telecom customers.

When a customer writes his/her opinion on a forum, a wealth of information is generated because -more importantly- a customer uses words and phrases that cannot be found during a controlled study. The words, phrases and expressions are far more emotionally powerful than a Likert scale answer of type "Totally disagree / agree".

So let us see the steps required :

First Step : The first thing of course is to actually find the data : User forums where people talk about mobile phones and mobile companies is obviously the place to look and there are lots of those places. Perhaps the volume of the messages is not enough but usually the available information is more than enough. Special code can be written to extract text from posts but without loss of the nature of the posting. As an example, the fact that a post has generated 20 replies is considered valuable information. The more posted replies, the more sentiment exists and this information has to be taken into consideration.

Second Step : Deploy information extraction techniques to identify phrases of good or bad sentiment (and actually many other things) about Telecom keywords such as :

- Signal
- Customer Care
- Billing

....etc

The following screen capture shows an example which is in Greek but i will provide all necessary explanation - Please also note that this is a simplified version of the process :

Notice that on the right hand-side there are some bars that denote the type of keywords found : The first category is called "Characterization" and if it is checked (which on the above screen capture it is) the software will highlight posts that only have some kind of characterization, whether good or bad. Notice also the yellow bar which has the name "Network". Because it is checked, words that are synonyms of "Network" are highlighted and indeed this is the case because

Signal = σήμα (in Greek) and
Flawless = άψογο

so the highlighted phrase άψογο σήμα means "flawless signal", which is a good characterization for the signal of two particular telecom companies. Notice also a line under the "Features" tab which says that between positions 3425 to 3429 there is a mention about signal ("mentionsSignal = true").

Again, i have to point out that this is a simplified version of the process. Text Mining and Information Extraction is actually very hard work but it is also very rewarding for those that ultimately deploy and use it. On the next post we will see the problems (and there are many of them) but also how this unstructured information is turned to "nuggets of gold".

Analytics and the Financial Markets

2008-12-03T17:42:00.012+02:00

On previous posts, i explained ways to analyze the financial markets by using data mining and text mining techniques. I also went through some potential pitfalls and perils during such type of analysis.

By combining different data sources (worldwide indices, moving averages, oscillators, clustering or categorization of financial news) an investor could take better decisions on where and when to invest. After such an analysis our goal is to make sufficiently better predictions than mere chance.

Some days ago i came across a website called Inner8. Inner8 is a really interesting idea : Collaborative filtering of stock picking. Combine this with analytics and an investor has on his arsenal -yet- another investing tool. Imagine thousands of Inner8 subscribers making stock predictions and giving their ideas, insights and sentiment for the stock market. After a few months some users will be "prediction super stars" from mere chance, so one has to proceed with caution. Nevertheless it is a website to keep looking at in the future, especially if the subscriber volume increases significantly.

So let us go back to our problem : We have to think of a good way to combine the information in our possession (aka problem representation) and feed this data on one or more algorithms with the goal of achieving models of high predictive value.

Some of the things to consider :

1) Should the "sliding window" technique be used? Could repetition of training data (because there is repetition of data in sliding window training) affect the predictive power of the model?

2) How many variables? Which are good predictors?

3) Do we care only about predictive power of the model? How about the interpretation of why a stock behaves as it does?

4) How can we represent the "additive effect" of 2 straight days of bad market news if a sliding window is not used?

5) Prediction Goal : Are we after price prediction (Regression) or price limits? (Classification)

Unfortunately the list does not end here : Since i am after predictions of stock prices in the Greek Stock Exchange, the data should be presented to the learning algorithm in a coherent way. European Markets are affected by the closing of US Markets and Asia. During Greek trading hours the US Markets open (approx. 45 mins before the end of trading - at 16:30 EET) , a fact that should be also taken into account.

I am sure that there are many users out there that have read a couple of data mining books, downloaded an open-source data mining tool, fed some data in and expect to see results. My only advice to them without the slightest sign of criticism: Paper-trade first...

Predicting popular stories on Digg

2008-11-26T09:45:00.007+02:00

On its latest news, KDNuggets mentions a paper from HP Labs that outlines a process of analyzing and predicting the popularity of a Digg story or a YouTube video submission.

No question that this is interesting material. On my post dated October 16th, 2007 i presented an analysis that i performed on what keywords seem to play a part on a post being popular on Digg. You can find all 3 parts of the post here.

So i made a new run to collect the stories from Digg and this is an example of what i came with (please note : For illustrative purposes only) :

The paper from HP Labs takes a different route and makes its predictions based on the popularity of a submitted story in the first few hours rather than after some days. The authors also conclude that after a digg story is out, users tend to vote for it in the beginning but when a specified threshold time has passed the rate with which the story is digged fades away. On the contrary, videos submitted on YouTube are being viewed by users on a linear trend after the video is submitted.

It is true that there is an inherent nature of seasonality on the news and the way that users 'digg' stories. It is also interesting to see at buzzwords that seem to keep repeating (in terms of how interesting they are -or not) over time.

Between the previous runs that i have made and the current one, i have seen some repeating patterns. One of these patterns shows Microsoft on a declining trend in terms of how much of an interesting subject it appears for digg users. Here is what Google Trends shows about the term 'Microsoft' :

Could such a trend may be a glimpse on Microsoft's 'future' somehow?

I have already built a text classifier which accepts phrases and shows the probability of this phrase being highly 'digged' based on the keywords that the phrase has. More on this on a future post.

Reality Mining vs Life Analytics

2008-11-21T13:43:00.002+02:00

We will be taking a short break from making predictions for the financial markets because i just came across a novel term (at least for me) regarding an analytics application : It is called Reality Mining

In short, Reality Mining is about using smart phones to record the interaction of a user with his device and the interaction of the user with other cell phone users. By analyzing this data, patterns of behavior can be extracted that can potentially be interesting to social researchers. For more information on Reality Mining see here.

What made me start this blog was the following question: If there could be a way to capture -and thus record- what a person feels, thinks and otherwise experiences everyday, what kind of patterns might emerge from analyzing this information? I think that this is one step (or even more steps) further from Reality Mining. What would happen if this kind of information was recorded for a vast amount of people and then analyzed? What if we could predict how thoughts we make and what we experience might affect our life later on and the decisions we will make? This is full-blown 'Life Mining'.

I get some e-mails from readers on "how this life analytics project is going" and for this subject there will be some future posts very soon.

Text Mining on Financial News

2008-11-16T10:20:00.010+02:00

As discussed previously, an analyst should give specific attention to problem representation particularly when we are dealing with text data. A way to do this will be discussed below, however something has to give and there is no perfect solution for such a task.

First of all we have to find the source of the news : It could be financial news sites such as Bloomberg, Financial Times, or RSS Feeds URLs such as the ones provided by MarketWatch. RSS Feeds might be a better solution because there is already some predetermined categorization of news according to the feed type and this can be great help for some analysts.

After finding the news sources and making the necessary code to get the actual information we could end up with the following text file :

You can see that i use a '^' separator to differentiate between :

1) A date stamp,
2) A date string
3) The news string
4) A characterization of the news (important or unimportant)
5) A categorization of the financial news.

This simple file could provide the basis for a training file for text categorization. Assuming that we have trained algorithms to automatically classify news, we could use a news classifier to first categorize news to important or unimportant and pass only the important news to a second classifier which will do the detailed classification of the news.

Another option is to use clustering : You can imagine that the solution detailed above has a tremendous amount of work depending on how much data you are planning to collect...so too much data means too much work, less data could mean -usually but not always- less accuracy.

But how could clustering be performed on such data? Simply, we just use field number (4) on our training text file to train a clustering algorithm and then see what 'classes' the algorithm has come up with.

So let's see a small example about clustering : This is a capture from WEKA just before the clustering process :

I have produced a training file which essentially contains the 'buzzwords' of financial news : barrel, recession, Yen, Euro, ECB, price, consumer, etc. The file is then analyzed by K-means algorithm to extract clusters of the same 'buzzwords'. Each cluster is assigned a number so each news header ultimately falls onto one cluster number.

After running the K-Means algorithm i ended up with 16 clusters. Let's see two instances that K-Means decided that they should fall under cluster '6' :

Instance_number : 130.0

Fear
Decrease
US
Economy
Futures

and

Instance_number : 174.0

Fear
Decrease
US
Price
Oil
Banking
Recession

So the first instance is about fears for the US Economy which results in US Futures dropping and the second instance must be -something about- a decrease of Oil prices and Banking stocks because of the fear of US recession. Not bad at all...

But not so fast : Clustering presents a lot of problems later in the process. Remember that what we are after, is to combine text mining and data mining together to better understand how the markets react. Should one use classification or clustering? There are many more things to take under consideration and for obvious reasons i cannot disclose all the details of such a project...but i am hoping to give to the interested reader a good enough introduction on the subject.

Capturing the Financial Facts

2008-11-14T10:32:00.006+02:00

So far, we have seen the data mining part on analyzing the financial markets and some of the problems that arise during such an analysis : Data have to be collected and pre-processed accordingly. There are dangers of over-fitting and the analyst must make sure that the model(s) created have the expected quality. The analyst has also to choose relevant attributes with which the analysis will be performed and how the training of the algorithms will be made.

The markets react to financial news and there is no question about this. Of course there are other factors that make people buy or sell : For example if a stock price has hit a support or resistance level then some investors are going to either buy or sell when such a price level is reached. Investors are also going to buy or sell when specific technical indicators such as MACD or oscillators show the signals to do so. Even when bad news are out, markets after an -unknown- number of consecutive drops will go up by an -unknown- percentage and vice-versa.

People that are involved with Machine Learning know that the representation of the problem at hand is of high importance...so first we are going to see how financial news can be represented in a helpful way for the analysis.

We have to see with what we are dealing here. To do this, we have to analyze and categorize accordingly the financial information as this is created. Financial News can be news about a number of things :

1) The number of jobless claims in US is higher than last year.
2) Automotive company's XYZ sales were dropped by 15%
3) Oil prices hit -yet- another record high
4) The dollar is dropping

....and the list goes on.

So the first problem arises : Should we categorize the information according to its content and present it to the algorithms? We could do that by having a boolean field for each type of news on our training file and set it accordingly to TRUE or FALSE values. By using this method we could easily reach thousands of input fields, since for the "jobless claims" news type we could have the following variants :

-A specific country for the jobless claim report (not only the US, it could be any country)

-Jobless claims could be higher than expected or higher than last year or the highest in the last decade.

It is easy to see that this gets way too fast out of control. Perhaps a better solution would be to try to create clusters of (more or less) the same news. The idea of clustering the financial news might seem an interesting one and an analyst could define a number of clusters -say he is after 100- and let the clustering process categorize accordingly all the news. But is clustering the solution? More on this on the next post...

Model Testing

2008-11-12T10:41:00.008+02:00

Once a model has been created (such as the decision tree for our example), the analyst is required to test the model. During model testing, an analyst performs specific tests that show the actual predictive power of a model.

Many methods can be used for model testing, depending on the problem. For our example and since the available volume of data is sufficiently large, the model training and testing methodology i used was as follows :

1) 50% of data were used for model training
2) 25% of data were used for model validation - fine tuning
3) 25% were used for testing of the model.

In other words, 75% of the data were used for training the algorithm and assessing the impact that changes on algorithm parameters have on the accuracy of the model. For a decision tree algorithm (and depending on the type of decision tree used) an analyst might try different settings for splitting criteria and/or number of minimum cases per branch, etc.

Unfortunately, numerous times an analyst finds that the predicted accuracy of the model given during training - model validation phases (ie steps 1 and 2 shown above) is in no way representative when the model is tested on unseen cases ( Step 3).

During my analysis, numerous models were showing an estimated accuracy of 85% or more but when they were presented on actual data, the accuracy was dropping down to 50-53%, suggesting that overfitting was present. Consequently, the use of these biased models to predict new cases would have detrimental effects in actual stock trading.

When all models are built, the analyst should choose a model (when there is a requirement to use only one model) according to :

1) (Statistically significant) best accuracy.
2) Misclassification costs, if these are not taken into account during the model building process.

On the next post we will see how text mining may help us in making better predictions for the markets.

Decision Tree Interpretation

2008-10-30T16:34:00.009+02:00

On the previous post i went through some basic steps required for predicting the price changes of a specific stock of the Greek stock exchange market. As a result of this process, the following decision tree was generated :

To interpret a decision tree, the analyst starts from the root of the tree and reads through it until a leaf node is reached. For example a rule that can be extracted from the decision tree above is the following:

"IF aseStockExchange > 0.360 AND aseStockExchange > 1.985 THEN price>+2"

The rule above can be found by starting from the root of the tree, moving on the left branch and then continuing to the right sub-branch. In the same way an analyst is able to find the rest of the rules identified by the decision tree.

When using decision tree learners or rule extractors, analysts record the precision and recall of a rule which are not shown in the decision tree above. However, for matters of simplicity i will omit this information and describe the insights provided from the analysis. Decision Trees possess the two following qualities :

1) They provide easy model interpretation

and

2) They show us the relevant importance of the variables

When confronted with many variables, analysts usually start by building a decision tree and then using the variables which the decision tree algorithm has selected with other methods that suffer from the complexity of many variables, such as neural networks. However, decision trees perform worse when the problem at hand is not linearly separable. For the purpose of our example though, a decision tree 'explains' the behavior of the stock nicely.

It should be noted that during the Feature Selection analysis of our stock example we have found that features 'aseStockExchange' and 'DAX' are important. Other features such as 'xaaPersonalHouseProducts' were flagged as important from the Feature Selection algorithm and were not used in the decision tree. Different feature selection methods produce different results (and one might say that this is not very assuring) but usually most methods produce a common feature subset that is of high predictive value.

The importance of the attributes can be seen from the level that they appear on the decision tree (the higher the level, the better is the prediction power of the attribute). So in our example, the 'aseStockExchange' feature is the most important (since it is the attribute with which the decision tree starts) and less important attributes seem to be 'xaaLeisure' and 'xaaBenefit'.

Insights from a Decision Tree

2008-10-16T00:22:00.017+03:00

Assuming that an analyst has made all necessary pre-processing tasks prior to the data mining phase, we are ready to deploy analytical methods such as decision tree learners that can classify unseen cases. For the goal of stock prediction we assume that we have the following data collected :

The column named as XAACLASS is the target column that we wish to classify. Essentially here we have the following classes :

-price change percentage greater than 2%
-price change percentage less than -2%
-price change percentage greater than 0% and +2% inclusive
-price change percentage between -2% inclusive and 0% inclusive

In other words, each line shows us the state of the stock we wish to predict, that occurs given the rest of the market indices (such as realTimeFTSE, realTimeDAX, etc).

So, let us assume that we are ready to build such a model. However, we have to decide the time window that our predictions will be made for...do we wish to predict what the stock price change will be 2 hours ahead? How about 1 day ahead?

Before dealing with this issue, i wanted to see how good a predictive model is by predicting the stock price percentage change right now, based on the current market conditions. Here is a decision tree that is created from such data:

More to come on the next post where the model seen above will be explained in detail. Until then please read the post from this blog about the same problem. If you can, read Fooled By Randomness also...

So...What's important??

2008-10-10T00:23:00.009+03:00

A step of a Knowledge Discovery Process is to perform what is known as Feature Selection, which essentially is the identification of a subset of features with high predictive value.

Feature selection can potentially help in increasing the accuracy of prediction models. Methods such as Naive Bayes can perform better when presented with a subset of selected features, rather than the whole feature set (because of feature redundancy).

Even if feature selection does not prove to help too much, it is important to know the predictive power of each feature. There are numerous methods to do this and -as normally is the case- there is no universally better method to perform an optimal feature selection. The following is a representation of all available Feature Selection methods in WEKA:

Let us stick to our example with stocks, to make things more clear. Suppose that i would like to know which features seem to be important for predicting the behavior of a stock. For our example we will try to find out about how the stock of NBG reacts.

By using a feature selection method we extract the following information :

The feature selection method above shows us how many times each attribute was selected during a 10-fold cross validation. We can see that some attributes are used more times than other attributes during each cross validation . For example :

realTimeDax
aseStockExchangeIndex
xaaPersonalHouseProducts
xaaTechnology
bankAgrotiki
bankAlpha
bankPiraeus
bankEuro

are present in all 10 folds of our cross-validation and hence the 10(100%) entry. xaaFinancialServices index has been selected fewer times (8 out of 10) and hence the 8(80%) entry. Other features never appear to any of the cross validation folds.

Of course feature selection does not stop here and there are many ways to enhance the process. Data Mining is both an art and a science. However for our purpose, we were able to identify those attributes that seem to be important in the prediction of the NBG stock. We immediately see for example that DAX index and the Athens Stock Exchange Index are two important features, plus the stocks of four specific banks. Other methods of feature selection produce weights that essentially rank the importance of each attribute for class prediction.

Always know your data!

2008-10-06T15:59:00.006+03:00

Before rushing in analyzing and predicting the Financial Markets (and actually anything else) it is essential that we get an idea about the data at hand. So after data collection (ie getting values of different market indices) i wanted to understand first what is going on to the markets. And a correlation Matrix tells us just that. Let's see what happens on the Greek Stock Exchange :

By looking at the matrix we can immediately see some interesting things :

1) There is a high correlation (=0.847) between DAX index and the Greek stock exchange index (marked as aseStockExchangeIndex)

2) The Insurance index sector (xaaInsurance) and the Media sector (xaaMedia) have a low correlation with the aseStockExchangeIndex. Consider the following scatter chart that shows the poor correlation between Insurance sector stocks and the aseStockExchangeIndex :

Those two facts alone can help significantly in trading: For example if an investor's trading decision is heavily based on aseStockExchangeIndex then the investor should also keep a close look on the DAX Index as opposed to other European indices (such as FTSE,CAC40,etc).

A lot of problems later in the analysis can be prevented if one pays attention to the "Data Understanding" phase. Plus, we also get an insight as to what kind of results should we expect from the learning algorithms.

Predicting the Financial markets

2008-09-25T13:46:00.004+03:00

After a very long break i decided to start writing again. It is very interesting to see that people are still answering on my questionnaire (see Links area) and every week i see more answers coming in....but more data is always welcome!

Since February i have been involved with yet another Data & Text Mining Application, namely predicting the financial markets by using financial - world news (the Text Mining side) and key financial indices (The Data Mining side). There are numerous blogs and entries i have found about this problem. One blog example is neural market trends and also a series of articles which i originally found in kdnuggets.

There is no question that such an application of Predictive Analytics in Financial Markets is interesting but also it can be (potentially) dangerous. However my experience so far on the subject has shown me that by a) getting a grip on the risk of a trading decision and b) using Predictive Analytics to make the trading decision, a user has more chances in making a successful trade.

On the next post, we will go through the data mining side of the problem.

Analyzing the Real Estate Market - Part 2

2008-02-11T13:08:00.001+02:00

In the previous part i listed the first steps required that can turn unstructured information of flat adverts for rent to a suitable form for further analysis of the Greek Real Estate Market.

Once the Information Extraction step is finished, the characteristics of each flat advert (price, square meters, type of heating, years old etc) are inserted into a database. Once flat adverts data are inserted, we are able to extract key information about price trends for specific areas of Athens such as Nea Smyrni. The following screen capture shows a portion of the records that exist in the database, after the information extraction phase :

With the advert data in place we are ready to deploy data mining algorithms that can reveal to us potentially useful patterns. For example, a classification analysis aimed in finding which characteristics are important to obtain a high renting price produces the following decision tree :

The decision tree depicted above essentially gives us the following information :

The most important characteristic for obtaining a high renting value (in terms of Euros per square meter) is the provision of a parking space with the flat.

If a flat provides a parking space, has a storage area and has up to two bed rooms then the flat obtains the highest renting rate, (ie 7.54 Euros per square meter)

If a flat does not provide a parking space but has at least one bedroom and is located at the fourth floor (or higher) then the flat obtains the highest renting rate per square meter.

Analyzing the Real Estate Market - Part 1

2008-01-06T16:27:00.001+02:00

Over the next days i will present an example of using Data Mining and Information Extraction techniques to analyze Real Estate in the Greek Market.

The problem is as follows : In a specific suburb of Athens in Greece (let's say Nea Smyrni) what are the key factors (or characteristics) that contribute to a high renting price of a flat? Which is more important? Having a parking space, or whether the house is less than 5 years old?

This piece of information is particularly valuable for flat owners, real estate investors and real estate agents (to name a few) according to my experience.

I really like this example of analysis given, because it shows the power of Information Extraction and Data Mining combined and the insight that these techniques can reveal.

In order to implement this analysis, the first required action is the collection of information. For this reason, special software collects flat adverts for rent from Greek websites. The next step is to extract each flat's information from each advert. Information extraction is used to extract these characteristics as shown below :

The goal of Information Extraction is to transform unstructured information to a form suitable for further analysis. More specifically, after the Information Extraction phase, the characteristics of each flat advert are inserted into a database. More on this on Part 2...

What People Digg More? - Part 3

2007-12-14T08:21:00.000+02:00

In this third -and final- part of the way that digg stories are analyzed, i will present an example of the co-occurrence table used to find statistically significant correlation of words.

In the previous part i outlined the way stories from digg are collected and how these stories are transformed in a way suitable for analysis. This last step of the analysis, is finding out about what subjects people seem to really like and which subjects are not 'digged' so much. To do that, a co-occurrence table is used which maps words to two categories : HighDiggs (=stories that are interesting) and LowDiggs (=stories that are not interesting).

This is an example of a word co-occurrence table from IBM's UI Modeler

The statistical significance between words and categories of interestingness is denoted by colors.The more intense the color, the more higher the affinity between the word and the category.

In the example table above we see that people are interested (and therefore 'digged' more) in :

1) Stories that have pictures
2) US President George W Bush
3) Apple Leopard
4) Ron Paul (not shown in table)

On the other hand people do not 'digg' stories about :

1) Microsoft (..!)
2) Blogs (not shown in table)

What people Digg More? - Part 2

2007-11-06T11:55:00.001+02:00

After getting some e-mails requesting more details about the way i analyze diggs, here are the details of the process :

First of all, obviously some coding is necessary to implement software that sifts through digg and records the number of diggs of the story as well as the time that the story has been out. The software is also responsible for selecting all stories that have been out for 10-11 days and for calculating the diggs_per_minute metric

where :

Diggs_per_minute= total_diggs / total_minutes

During the analysis it appeared that the diggs_per_minute metric was not normally distributed since its skewness (positive) was found to be 2.795.

After applying log transformation, skewness dropped to 0.534 having a mean value of -2.878 :

The next step is to create a text file, as follows :

Notice that there is a 'highdiggs' or 'lowdiggs' word at the end of each line (story). If the diggs_per_minute metric for each story exceeds the threshold value -2.878 then 'highdiggs' is appended at the end of the line, otherwise 'lowdiggs' is added.

The last step of the analysis is to use a co-occurrence matrix to see which words are associated with high digg and low digg stories. A chi-square test is used to test for statistical significance of word co-occurrences.

For the last part of the analysis i use a tool called Unstructured Information Modeler from IBM.

What people Digg More?

2007-10-16T11:31:00.001+03:00

Due to too much work i wasn't able to write to this blog as much as i wanted. Although people continue to answer the questionnaire (over 400!) i wasn't able to make any other analysis so far that will shed some light on the patterns that emerge from living our lives.

However, i feel that i should write something about my new ventures on text mining. The question that came up to my mind was simple :

"What stories people tend to digg more?"

So i collected all stories on digg and for each story the number of diggs and the time that the story has been around was recorded. By dividing the number of diggs by the total minutes the story has been out, you get a "Diggs_per_Minute" score which essentially designates which stories are "hot" and which are not.

After the preliminary analysis i immediately found out that it is essential to use data from a specific time period and not just everything. If you think about it, a story should be out for quite a while (say 10 days) so that you are able to get a good estimate of the "Diggs_per_Minute" variable. Stories that have been out for less than 2 days tend to have a much greater score of Diggs per Minute than newer stories.

So the process is as follows: Diggs from stories that have been out for 10-11 days are collected. I then use text mining techniques to find out what words the stories with many diggs have in common. Don't you think that marketing people would love to know this information?

First Results for Most Digged stories :

1) Stories that have pictures tend to be digged more

2) Having the phrase "Digg this if you....."

3) Specific Companies / technologies etc (e.g Apple and Ipod)

That's all for now but i will come back with more.

First Results Out : Phobias

2007-09-14T15:15:00.001+03:00

Well over 300 people have submitted and described their lives on the questionnaire. I must admit that i couldn't wait for this time to come so that i can start finding out about patterns that emanate from our lives.

As explained in another post regarding classification analysis, the target variable in my first attempt is predicting class "Phobias". Simply put, what are the common characteristics of people having phobias?

It seems that people that have answered > 2 on the good looking scale question are less likely by 85% to have phobias. Does this make sense to you?

This is just one example of how interesting patterns may arise from analyzing submitted data. If you already have submitted the questionnaire why not asking your friends to do it?. The questionnaire can be reached on the following URL :

http://lifeanalytics.org/MainSurvey

LifeAnalytics : Over a month online

2007-08-27T12:02:00.001+03:00

Time sure passes by very quickly...it is over one month now that this blog has started. Over 200 people have submitted their answers to the questionnaire so far...Over half of visitors originate from the US but also people from Europe (especially UK, Germany and the Netherlands) are also producing many hits. A Big Thanks to kdnuggets (see links area) -the best site for data mining news- for listing this blog on its pages.

The truth is that with over 60 questions, we need more people to fill the survey, so if you are reading this and you haven't filled the questionnaire yet, you can submit your answers here :

http://lifeanalytics.org/MainSurvey

Very shortly, i will make a new post explaining what types of analysis can also be made -apart from classification and clustering- once we have enough data...

By public demand, here is more information

2007-07-26T14:26:00.000+03:00

Already the first answers are coming in. A Big Thanks to everyone that already have submitted their answers to our questionnaire. I get quite a few e-mails asking me how i am going to use the results, so i feel it is time for more explanation.

The process of Data Mining consists of the following steps (simplified..) :

1) Data Collection
2) Data Preparation
3) Analysis
4) Application of results

Currently we are on step (1), collecting the data. Although questions on your e-mails sent to me are for step (4), i feel that it is important to talk about step (3) as well :

By deploying specialized algorithms, we try to find common characteristics of a specific class in data (also known as classification) . By class we mean a category. For "happiness" we have two possible categories of people : Those that are generally happy and those that are not. In the same manner several other classes of people exist .

Once we decide on which class to analyze (say married/divorced) , now it is time to perform the analysis with the goal of creating a predictive model. Once a -reasonably accurate- model is created, we are ready to predict new, unseen cases of people. That essentially means that when new users submit the questionnaire, they will be given a score (= a percentage) of the probability of getting a divorce. Note that "divorce" is just one class ; several other predictions can be made for other classes too, once a predictive model has been created for each class . But that's not all. Not only we are able to predict new cases, but -with specific algorithms- we are able to find out what is important on an outcome (ie getting a divorce) and also the importance of each parameter (where each parameter is a question on the questionnaire).

In our example rule :

IF AGE >31 AND AGE<=40 AND NUM_OF_CHILDREN = 0 THEN DIVORCE="TRUE"

Not only we realize that age and the number of children is important in getting a divorce, but also we are able to know the relevant importance of each parameter. For example, a model may tell us that the most important factor in getting a divorce is first of all having children and the second factor in importance is the age.

Of course there are other kinds of analysis which do not search for common characteristics but seek to find associations between variables. Another analysis type finds homogeneous groups through clustering, but we will leave examples of these types of analysis for another post.

Questionnaire is Ready

2007-07-20T09:15:00.001+03:00

Although i thought it will take a lot of time, the questionnaire is ready. I used a great tool to create the web questionnaire called websurveytoolbox. By using it, i managed having the questionnaire ready in less than 2 days. All data and questions of the questionnaire are saved on a MySQL database and the tool automatically creates jsp pages with the questions. I also registered the domain lifeanalytics.org although there is not much to see there for now, except that it hosts the necessary code and database for the questionnaire.

To fill the questionnaire please visit:

http://lifeanalytics.org/MainSurvey

I counted the time needed to fill out the questionnaire, so assuming you know "Everyday English" it shouldn't take more than 5 minutes to complete it. The questionnaire comprises of all kinds of questions, having the (tough) aim of describing one's life as best as possible in terms of facts, personal decisions and points of view.

As a last note, i would like to stress out that no personal data are asked, not even your e-mail. If you feel that this effort is worth it, please let other people know. You can even digg it to spread out the story ;-)

Life, Uncertainty and Mathematics

2007-07-18T20:21:00.002+03:00

One of the immediate questions that may arise is "How on earth are you going to predict if someone is going to get a divorce?".

So, what essentially i am trying to do here is to model life and its uncertainties with Mathematics...Now, can this be possible?

Certainly there are hundreds of factors that could play a role in getting a divorce. A questionnaire of a 100 -or less- questions cannot capture the facts of a person's life. But my goal is to just give it a try and see how it goes. Perhaps tens or even hundreds of thousands of answers may be able to give us a clue as to what is happening.

Each rule extracted (see previous post about what i mean by rules), will be tested for its statistical validity through chi-square tests and making adjustments through Bonferroni correction. Several other techniques will be used to assess the quality of the extracted models. In other words, if there is something there, we will find it.

Once a model is produced and is reasonably accurate, we will be ready to predict unseen cases. In other words -and continuing our divorce example- if a model is 80% correct in predicting whether someone will get a divorce, then anyone that fills the questionnaire at the end of the process, will also find out about the probability of getting a divorce. More importantly : Why he or she, is likely to get one.

LifeAnalytics blog has started

2007-07-17T14:53:00.002+03:00

I finally made the decision to start LifeAnalytics blog. Hopefully, many people will find useful the findings from the research on the patterns that emerge by just living.

What is LifeAnalytics? Simply put, i will be using analytical techniques (especially classification and associations discovery), also known as Data Mining to understand key facts about a person's life : For example ,what are the common characteristics of people that are divorced? What factors play an important role in having an increased risk for getting a divorce?

Of course, getting a divorce is one probability in someone who is married. Several other facets and facts compose our lives.... as an example consider the following life facts :

- Having a good marriage
- Being happy about work
- Having phobias
- Having an above-average salary
- Being Depressed

The goal then, is to look at all of those probable outcomes in one's life and try to extract "rules" that increase (or decrease) the probability of experiencing the above facts. In order to do this, thousands of people must somehow describe their lives and their character idiosyncrasies by submitting a questionnaire.

Example : By analyzing thousands of people's life facts (submitted via questionnaire), we may extract the following rule :

IF AGE >31 AND AGE<=40 AND NUM_OF_CHILDREN = 0 THEN DIVORCE="TRUE"

In other words, the example rule above says that People between 32 and 40 years old without children have increased probabilities (say for example 82%) in getting a divorce. Findings and conclusions like the example shown above will be given to anyone interested -free of charge of course- from this blog.

Think about it. Living our life creates "data" and along with the "data" of thousands of others we may find some really interesting answers.

Stay tuned, the journey to this kind of knowledge has -hopefully- just begun...