While stocks are considered as a fundamental financial equipment and has a significant role in the economy of a country. It is really important to find out the major factors that contributes to the trend of stock market. We are technologically advanced and have various medium to collect news about anything in the universe. The purpose of this report to show how news headlines affect the stock market trends.
The author have collected news headlines of Apple Company for a month and did a sentiment analysis and data visualisation of those news to correlate the ups and downs of trade data for a particular period. All the information are provided by Yahoo Finance, the site for most up to date business related news. Processing of data is done in R software, which is developed to solve the problems in computational finance and have many packages for data visualisation.
People are only using fundamental analysis and Technical analysis to predict the price of their stocks, but most of them are really unware about the dependency of sentiment analysis for market trends. More precisely, the stock market’s fluctuations are analysed and predicted in order to retrieve knowledge that could guide traders on when to buy and sell.
Stock market plays a remarkable part in the economy of a country so it is very vital to identify the factors which has an effect on stock market trends. A talk about a product or a business news have a greater influence on the markets, but it is really unknown how accurately it affects the market movements. This project visualize the relation of news sentiments and stock market trends. R language’s available functions and algorithms helps to facilitate rapid data analysis and visualization and it very easy to interface with new data sources and other computing environment in the financial industry. There are more than 70 packages in R that are presumably developed directly solve the problems in computational finance.
People are getting their news from different sources like traditional print, radio, television and social media. People depends on news to know what is happening in the surroundings and even used for to take a smarter choices like when and where to buy stuffs or which people to elect into office. When it comes to business mind, people are always curious about the industry which they are going to invest. Before taking a good decision, they will look for the current industry trends, corporate news, best practice company’s features and other related information about corporate industry. In short, news has a pivotal role in all fields.
Now a days stocks are considered as the pivotal financial equipment and it allows a person to own a share of a public corporation so that he can profit from earning’s growth and dividends. Google Finance and Yahoo Finance are the two most popular site to get current financial news, market data for various companies and their shares, updated news and trends. So it is very easy for everyone to know about stock specific data in every second. It also allows us to download the data in CSV format also.
For this project author have collected news headlines of ‘Apple’ from Yahoo Finance for a month. In the present scenario, succession of good news about a company may lead investors to overreact in market and it also boosts the investor’s self-confidence and reducing risk mentality. Sentiment analysis of such news helps investors more to predict the stock market trends. R software provides good functionality for sentiment analysis and time series plotting. Especially quantmod in R is designed to assist the quantitative trader in the development, testing and statistically based trading models. Charting with quantmod provide a better understanding and visualisation of trade data. This project correlate the news sentiments and trading data with the help of R language functionalities.
3. LITERATURE REVIEW
There are mainly two methods existing to predict stock market. They are fundament analysis and Technical analysis. Fundamental analysis is calculating the intrinsic values of company and Technical analysis is based on evaluating the company’s past market activity (J. G. Agrawal, Dr. V. S. Chourasia, Dr. A. K. Mittra “State-of-the-Art in Stock Prediction Tech-niques” April 2013) This two techniques are not enough to predict the market trends, there are various other economic and political factors are also taken into consideration. The recent and up to date political and economic information can be obtained from social media and from news reporting agencies. Nowadays people are interested to express their idea and atti-tude in Twitter. Anshul Mittal and Arpit Goel (2009) investigated the causative relation be-tween public mood from twitter and DJIA values. But there is no straight association be-tween the people who invest in stocks and who use twitter more frequently. Sang Chung and Sandy Liu (Dec 2011) stated that twitter sentiment score only forecast stock market trend if the sentiment score is moving towards positive but not negative. There is no significant pre-dictive power for twitter sentiment score. The above stated conflicts lead me to find a static solution for stock market trend prediction that is the correlation between News headlines sen-timent score and stock market data. The motivation lead to do this are, everyone are con-scious about the company which they are going to invest and online news sources which are preferably free of charge. Methodology used for the analysis is also simple, with the help of statistical computing software ‘ R’ ,Yahoo finance and some functionalities of Excel.
4. FINANCIAL BACKGROUND
History of Stock market
In the 20th century, stock market investor’s activities were very different from what exists on today, only few people purchase individual stock. On that day, the main source of information was printed on paper. History of stock prices were available in library in the form of big books that were also several months old. If a person interested in some stocks he probably wait for one more month to check how they were going. When he was ready to buy a stock, he immediately call his stockbroker and have live conversation about what he wanted to buy. Depending on what he decided to buy, the stock broker place the order within the hour but might not be executed until the next business day. Stockbroker got paid commission for every stock they sold and very stock they bought.
By the late 1980s, personal computer had become very powerful and a serious tool. As a result people started to use PCs in their homes and the emergence of internet, stockbrokers turned into online brokers and communicated via internet and execute stock transactions. Over the last 200 years stocks are considered as the best return on money than any other kind of investment. In case of stocks historical performance is not a model for anticipating performance and it is very true about the risk associated with stock market. If we identify the risk and have a better knowledge about the company which we are going to invest, then will have an excellent chance of doing well in the future.
Why Stock market is so important?
The primary function of stock market is to support the growth of industry and commerce in the country. Rising stock market is a sign of developing industrial sector and a growing economy of a country .It will also affect the pension funds because most of the private pension funds will invest in the stock market. Movement in the stock market can have an economic impact on everyday people. A collapse in the share prices have widespread economic depression. Stock market crash on 1929 showing as a key factor in causing the great depression of 1930.Stock market makes the stock as a liquid asset unlike real estate investment.
4.1 STATE OF THE ART OF STOCK MARKET TRENDS PREDICTION
Stock Market prediction is the art of estimating the future value of companies’ trade or other financial instrument that traded on exchange. More specifically, stock markets movements are analysed or predicted in order to guide investors the best time to buy or sell the trade.
Fundamental analysis of stock prediction
Fundamental Analysis involves the in-depth analysis of companies past performance, It involves understanding the company apparently in terms of its product sales, man power quality,infrastructure,profitability on investment. It also consider revenues,earnings,future growth, return on equity, profit margins, social acceptance and other data which determines the companies underlying values and potential growth for future. The concept of stock valuation can be understood by knowing some ratios. They are Price to book ratio (P/B), price to earnings ratio (P/E), debt to equity ratio, dividend yield, returns on equity etc. Each of the ratios has their own importance and none of them are inferior or superior to each other. By combining these methods of valuation, we can get a better view of the stocks worth.
Advantages of Fundamental Analysis
• It is a systematic approach and its capability to forecast changes before they published on the chart.
• High level method for long term stability and growth
Disadvantages of Fundamental Analysis
• It is very difficult to implement and all this knowledge for the purpose of automation and interpretation of this information may be subjective.
• It is not possible to time the market effectively using fundamental analysis.
Technical analysis of stock prediction
Technical Analysis is the science or skill of forecasting the future financial stock trends based on an evaluation of past price movements. Like nature events forecasting, technical analysis does not ends in hundred percent prediction result about the future. Instead, technical analysis may help traders to think what is exactly going to happen to prices over time.
Fig 4.1.1 Technical analysis of Stocks of IBM 
Fig 4.1.2 Technical analysis of Stocks of Google
Technical analysis looks for pattern and indicators on stock charts that will determine a stock future performance. It focus on ups, downs, slope, fluctuations and other factors affecting a stock’s price movement. Technical analysts are not worried about the companies’ reputation. This analysis believe that market trend moves directed by frequently changing reactions of investors in response to different influences.
Advantages of Technical Analysis
• It is cast off by majority of the leading stock traders
• It is used to assess stocks for shorter period of time
Disadvantages of Technical Analysis
• Regardless its widespread use, it is highly criticized and is subjective also .
• Different individuals and predict or interpret the chart differently
From this two analysis methods, we came to a conclusion that existing techniques are not safe for prediction of stock market trends as well as price of different stocks. Still there exist a gap between technologies and user requirement for a safe and accurate stock prediction system.
5. HARDWARE / SOFTWARE REQUIREMENTS
Name Version Type
7 or above
Financial Data Website
3.1.3 or above
Programming Language For Statistical Computing And Graphics
Table 4.1 Requirement specification of Hardware/Software
Fig 6.1 Schematic Workflow of News Headlines Sentiment Analysis to Predict Stock Market Trends
6.1 NEWS HEADLINES COLLECTION
While collecting News Headlines it is very important to select the trade of a relevant organisation. The author have selected the most valuable and profitable company in the world, that is nothing but APPLE. It beat the whole market and running a successful business. Apple is considered as the most valuable in design and uniqueness in their brand. It has greater revenue and rapidly growing net income. Apple’s growth rate is extra ordinary while compare to other brands, in case of buying any Apple gadgets customers only look for the brand name. They are not concerned about other factors and have trust on the company products. It is a greater advantage to become first in the market and beat others. The stock for Apple is always volatile. The author have depend on Yahoo Finance to obtain the news feed. It helps to obtain the leading and breaking business news. We can get the RSS feed of Yahoo Finance on a worksheet with the help of Excel functionality.
Steps to obtain Yahoo Finance RSS feed on an Excel worksheet
1. Right click on the RSS icon and copy the link
Create an XML Map in a workbook using the RSS link where we want to create the RSS feed table
2. On the Developer Tab in an Excel sheet click on Source.
3. in the opened XML Task pane, click on XML Maps button
4. In the dialogue box click Add to create a new XML Map
5. Click in the appeared File Name box and copy the RSS link
6. Click Open and click OK, if a message appears
7. In the XML Source Task pane, find the item selection and click on item heading to select it.
8. Drag title from the item section to the worksheet space
9. This create a table on the work sheet with the heading from the item section
10. To see the list items, right click on a cell below headings and click Refresh Data under Developer Tab.
12/02/2016 Apple Music gains subscribers at a fast clip
11/02/2016 “Error 53”: lawyers are considering class action lawsuit against Apple
10/02/2016 Apple Pay: acceptance grows
09/02/2016 Apple Music now in over 110 countries
08/02/2016 Adapted Apple\’s movie trailer app for the giant iPad
05/02/2016 Apple to US Supreme Court: design dispute with Samsung earned no renewed clarification
04/02/2016 Media: Apple contract manufacturer Foxconn good opportunities in Sharp takeover
03/02/2016 Privacy in fitness trackers: researchers praise Apple
02/02/2016 iPhone: Apple restricts express exchange an apparently
01/02/2016 Alphabet Passes Apple as World\’s Most Valuable Public Company
29/01/2016 Own Exclusive content: Apple knocks again in Hollywood
28/01/2016 How To Fix Safari On iPhone After Crash
27/01/2016 Apple loses luster on fears that \’wow\’ days over
26/01/2016 Apple\’s iPhone success may be near peak
25/01/2016 Apple stuffs captive portal vulnerability in iOS – by year
22/01/2016 Apple hires virtual reality luminary
21/01/2016 iOS music app: users complain of repeated Apple Music & Cinema
20/01/2016 Apple TV: TV service is slow in coming
19/01/2016 Patent dispute: Apple obtains a sales ban against Samsung in the US
18/01/2016 iOS 9.3: No night mode for older devices
15/01/2016 Apple\’s Advantage Over Android Embarrasses Google
14/01/2016 Office 2016 for Mac: security fix and new features
13/01/2016 Critical vulnerabilities in Office for Mac
12/01/2016 iOS 9.3: Apple announces very useful numerous innovations
09/01/2016 German customs can “smart button” into the land
08/01/2016 Apple buys specialists for emotion recognition
07/01/2016 Reports of iPhone production cut: Apple\’s stock more pressure
06/01/2016 Apple\’s software shops: billion sales with apps over Christmas
05/01/2016 Trademark applications: Apple plans two, three, many beats
04/01/2016 Apple\’s stock first since 2008 over the year in the red
03/01/2016 8 ways Apple may delight business users in 2016
02/01/2016 Wall Street out of love for Apple?
01/01/2016 Taxes on phones and tablets: Apple more expensive iPhones and iPads
Table 6.1 Excel RSS feed table of Yahoo Finance’s News Headlines
6.2 SENTIMENT ANALYSIS OF NEWS HEADLINES USING R
Sentiment Analysis is the scientific study of people’s opinion, attitude and emotions towards an entity. This entity can be product, issues, organisation, topic and so on. Since 2002 researches are taking place actively in sentiment analysis or opinion mining. Sentiment is simply meaning the positive or negative feeling implies in an opinion and some opinion doesn’t contain any impact i.e. a neutral feedback. Suggestions or feedback are so important to business and organisations because they always want to know how their products are liked by the public. In the same way we can predict stock market trends by using news headlines and it is correlated to each other.
R software is a very good software for computing and data analysis, it is free, open source and highly extensible R software provides a wide variety of statistical techniques like linear, non-linear, time series analysis, classification, correlation and graphical visualisation facilities for data analysis. It is developed at Bell Laboratories by John Chambers and his colleagues. The packages in R are helpful for rapid data analysis and visualisation techniques.
Here author is using simple sentiment algorithms used by Jeffrey Breen to determine the scores of News headlines of Apple company (AAPL) from January 1st 2016 to February 12th 2016.Opinion lexicon used by him is based on the research papers of Hu and Liu.This algorithm assigns a score by counting the “positive” and “negative” occurrences in a statement.Hu and Liu published an “opinion lexicon” which almost contain 6800 English words separated as positive and negative and also allowed to download. Main strength of R is the facility to load and scan data. It is very flexible to load the downloaded positive and negative words and store in the local machine and import using scan function.
1. Loading and scanning of Positive and negative words by Ha and Liu
Now we will score the News headlines of APPL using Jeffrey Breen sentiment algorithm. This can be available from http://jeffreybreen.wordpress.com/2011/07/04/twitter-text-mining-r-slides/. ‘Plyr’ package in R helps to split data sets apart into smaller subsets and apply methods to parted subsets and combine the results.‘Stringr’ package used for text processing.
2. ‘score .sentiment’ Function
3. Testing of ‘score.sentiment’ function with sample data
Here author tested the ‘sentiment.score’ function with some sample data.
Sl.No Text Score Context
1 You are too good and Excellent 2 Positive
2 I am very Sorry, your services are very bad and worst -3 Negative
3 I don’t know 0 Neutral
Table 5.2.1 Sentiment score and interpretation of sample data
Here author have tested three different sentences and while reading we can get the clear context or sentiment of those three sentences and it replicated the same resulted with sentiment. Score function. Score ‘2’ means it is highly positive, ‘-1’ replicate it is a negative sentiment or a negative opinion and ‘0’ means it has no impact and it’s a neutral sentence.
passing sentences vector (News Headlines) to ‘lapy’ method. This method takes each headline sentence and pass on the function along with Positive and Negative words and combines the result. Function ‘gsub’ helps to handle the replacement like gsub (pattern, replacement, x).Convert all sentences to lower case by using the function ‘tolower’ and convert the sentences to words by split methods. Finally scores of all text will stored in ‘result’ variable.
Date News Score
12/02/2016 Apple Music gains subscribers at a fast clip 2
11/02/2016 “Error 53”: lawyers are considering class action lawsuit against Apple -1
10/02/2016 Apple Pay: acceptance grows 0
09/02/2016 Apple Music now in over 110 countries 0
08/02/2016 adapted Apple\’s movie trailer app for the giant iPad 0
05/02/2016 Apple to US Supreme Court: design dispute with Samsung earned no renewed clarifition 1
04/02/2016 Media: Apple contract manufacturer Foxconn good opportunities in Sharp takeover 2
03/02/2016 Privacy in fitness trackers: researchers praise Apple 0
02/02/2016 iPhone: Apple restricts express exchange an apparently 0
01/02/2016 Alphabet Passes Apple as World\’s Most Valuable Public Company 1
29/01/2016 Own Exclusive content: Apple knocks again in Hollywood 0
28/01/2016 Apple loses luster on fears that \’wow\’ days over 0
27/01/2016 How it possible to Fix Safari On iPhone After Crash -1
26/01/2016 Apple\’s iPhone success may be near peak 1
25/01/2016 Apple stuffs captive portal vulnerability in iOS – by year 0
22/01/2016 Apple hires virtual reality luminary 0
21/01/2016 iOS music app: users complain of repeated Apple Music & Cinema 0
20/01/2016 Apple TV: TV service is slow in coming 0
19/01/2016 Patent dispute: Apple obtains a sales ban against Samsung in the US 0
18/01/2016 iOS 9.3: No night mode for older devices 0
15/01/2016 Apple\’s Advantage Over Android Embarrasses Google 0
14/01/2016 Office 2016 for Mac: security fix and new features 0
13/01/2016 Critical vulnerabilities in Office for Mac -1
12/01/2016 iOS 9.3: Apple announces very useful numerous innovations 1
09/01/2016 German customs can “smart button” into the land 1
08/01/2016 Apple buys specialists for emotion recognition 0
07/01/2016 Reports of iPhone production cut: Apple\’s stock more pressure 0
06/01/2016 Apple\’s software shops: billion sales with apps over Christmas 0
05/01/2016 Trademark applications: Apple plans two, three, many beats 0
04/01/2016 Apple\’s stock first since 2008 over the year in the red 0
03/01/2016 8 ways Apple may delight business users in 2016 1
02/01/2016 Wall Street out of love for Apple? 1
01/01/2016 Taxes on phones and tablets: Apple more expensive iPhones and iPads -1
Table 6.2.2 Sentiment score of AAPL News Headlines
4. Histogram of AAPL News Headlines Sentiment Score
Histogram provides a visual representation of the distribution of a data sets. The shape of a histogram gives the most informative data and a great way to get know about data.R software has an extensive visualization technique and ‘hist’ function in R computes the histogram of given data
Fig 6.2 Histogram of AAPL Sentiment score
From the histogram it is clear that most of the sentiment score has value ‘0’ that means those headlines has neutral impact. Second position lies between the value ‘1’and ‘2’ and those are positive sentiment and least is the negative sentiment.
6.3 COLLECTING TRADE DATA OF AAPL
In Yahoo Finance we can download the stock information to a CSV file and also it support many additional features. It helps all business people to make a better analysis and decision on their stocks prices. The download URL is in the form
[FILENAME]-Name of the file to save
[TICKER]-The stock information to download
[TAGS]-Ticker symbol of a company
In our case, we need the stock information of AAPL for a month particularly from 01-01-2016 to 12-02-2016.So the URL will become like this
If the sample data is in comma separated value (csv) format, we can read the data with a function called ‘read.csv’ in R
6.4 TRADE ANALYSIS USING R
Quandmod in R
Quandmod is an extraordinary package in R, it is designed in such a way to assist the quantitative trader in development, testing and deployment of statistically based trading models. It provides a very suitable function to downloading any kind of financial data from the web. The function is called ‘getSymbols’ .The argument of this function is a character vector specifying the names of the vector to be specified. It is used to retrieve stock data. Data can originate in different number of locations, in the below example we are looking only for AAPL stock data.
The output will be an OHLC time series Open, High, Low and close prices for the symbol AAPL and it also contains the traded volume and the closing price adjusted for splits and dividends. The data object is in ‘xts’ format i.e. extensible time series object.
In R, it is possible to add many technical indicators to a stock chart to analyse in a better way. The function ‘chartSeries’ retrieves a large amount of trade data like the date, open and close price and volume of trading for each day. It is a primary function for drawing a chart in quantmod.
Fig 6.4.1 Stock data of AAPL
We can also add many technical indicators to the graph like ‘Bollinger Bands’ using the command ‘addBBands()’this amounts to a line indicating moving average and two lines a standard deviation above and below this moving average. Moving Average Convergence Divergence can be added by addMACD() and also we can customize the chart by changing theme, colour ,background, type of charts like candle chart, bar chart, line chart etc.
Compare to traditional bar chart, many traders prefers to have candlestick charts to interpret easily and to visualise the movements in markets.
Fig 6.4.2 Candlestick chart of APPL
Data Visualization of AAPL Sentiment Scores
Fig 7.1 AAPL Headlines and its Sentiment Score
X-axis represents the dates and Y-axis represents the range of sentiment score. It has the sentiment score of AAPL news headlines from 1st January 2016 to 12th February 2016.When moving the mouse pointer over the graph ‘dyRangeSelector()’displays the exact sentiment score of particular date and it is straightforward interface for panning and zooming. The digraph function is an amazing functionality of R which provides an interactive time series visualisation. It has a maximum score of ‘2’ on 4th February and minimum score ‘-1’ on several days.
Trade Data Visualisation
Same functionality of ‘dygraphs’ can also be used to visualise trade data of AAPL.It provides time series visualisation of data.
Fig 7.2 APPL Closing Prices
This is a graph against AAPL Closing prices and Dates. Closing price refers to the last price at which a stocks trades during a regular trading session. This graph has several ups and down and it states that the trade data is dynamic. Mining microblogging data to predict stock market behaviour is getting popular topic for researchers.22nd January has the highest closing price of $100.87264 and 27th January has the lowest closing price of $ 92.91581.If we analyse these two graphs closely we can identify that how news headlines have an effect on stock market trends. The ups and downs of News sentiment score and the closing pricing of AAPL are almost same for the period from 1st January 2016 to 12th February 2016.This can be clearly represent in the below diagram.
Visual Correlation of News Headlines Sentiment Score and Trade data
Fig 7.3 Ups and Downs of AAPL Headlines and its Sentiment Score
Fig 7.4 Ups and Downs of APPL Closing Prices
In both of the above graphs, there is a strong visualisation correlation between sentiment score and trade data. Similarity in the shape and trends of the curve indicates strong relation between the two. Green star indicates the ups and red star indicates the downs. By visualizing the dates exactly, both graph indicates a win on 12th January 2016.That means APPL has delivered good news on 12th January 2016 and its sentiment score is positive one and closing price raised from 98.53 to 99.96 on the same day. Another strong evidence is on January 27th AAPL delivered a bad news and has a score of negative one, closing price of same day decreased from 99.99 to 93.42.
Statistical Correlation of News Headlines Sentiment Score and AAPL Trade data
Fig 7.5 Statistical correlation of AAPL News Headlines and AAPL Trade data
In the above correlation graph, it is clear that closing prices with high rang are concentrated on positive score. It is marked in ‘green’ colour. Closing prices with less range focus on the negative sentiment score and marked as ‘red’ colour. Now it is easy to make a conclusion that more than 60 percent of the stock market moves depends on news headlines and its trustable for the investors to rely on market news.
8. CONCLUSION AND FUTURE SCOPE
News Headlines sentiment can also be used along with other techniques to provide a very strong indicator of stock market trends as the results shows a strong visual correlation between the two. News has a direct impact on stock market and it can easily change a good day into a bad day and vice versa. Negative news has more reaching effect to investor sentiment and it stops normal people to buy stocks. Sentiment of the market is an important factor and news about company results, government policies, political interest, inclusion or exclusion from indexes, mergers and acquisitions etc. affect stock market. Pattern results in this project shows a post news drift in stock market movements. News can also be considered as a motivational factor for price movements. Sentiments score can predict movements of stock quite accurately.R can be used as a potential computational software for data analysis and visualisation
If we collect and analyse news for larger period, we would get better results and it easier to predict the trending pattern, but it sometimes difficult too to obtain an older news archive. Yahoo Finance is an aggregated source for financial news and it is used by major industrialists and financialists. If we develop an application which automatically calculate the news headlines sentiment score of major companies or organisations it may help the business men to realise where to invest or when to buy or sell their shares. Many researches are also taking place based on natural language processing or mining microblogging data to forecast stock market behaviour.
9. OVERALL PROJECT TIMELINE AND MILESTONES ACHIEVED
Fig 9 .1 Overall Project Timeline along with Milestones achieved
 P.G.Preethi, V. Uma, Ajit kumar,“Temporal Sentiment Analysis and Causal Rules Extraction from Tweets for Event Prediction”, International Conference on Intelligent Computing, Communication & Convergence (ICCC-2014) Conference Organized by Interscience Institute of Management and Technology, Bhubaneswar, Odisha, India
 J. G. Agrawal, Dr. V. S. Chourasia, Dr. A. K. Mittra “State-of-the-Art in Stock Prediction Techniques” International Journal of Advanced Research in Electrical, Electronics and In-strumentation Engineering Vol. 2, Issue 4, April 2013
 Anshul Mittal, Arpit Goel “Stock Prediction Using Twitter Sentiment Analysis” Stanford University
 Sang Chung, Sandy Liu, “Predicting Stock Market Fluctuations from Twitter, An analysis of the predictive powers of real-time social media”, Aldous Dec 12, 2011
 Suresh Kumar, K. K., Elango, N. M., “An Efficient Approach to Forecast Indian Stock Market Price and their Performance Analysis”, International Journal of Computer Application, Vol. 34, No.5, 2011.
 J H Maindonald, “Using R for Data Analysis and Graphics Introduction, Code and Com-mentary”, Centre for Mathematics and Its Applications, Australian National University.
...(download the rest of the essay above)