A framework for investigating search engines' stemming mechanisms: A case study on Bing

Abstract

Big data attracts the attention of governments and a lot of companies today. The developments in technology and the Internet make it one of the important sources of big data. It is easy to get lost in the enormous amount of information contained on the Internet if there were no search engines. Knowing how the search engines work will be helpful to access the desired information. This work aims to be a guide for accessing the right information and also to help to understand search engine stemming and indexing algorithm for interested parties. In this article, we have developed a framework that could be used to investigate the stemming mechanisms of search engines. Our framework also uses Word2vec to analyze semantic relations. We have used our framework to investigate the stemming algorithm of the search engine Bing for English language. In order to achieve that we have used this framework to select words, create queries, send them to Bing, and finally analyze the millions of returned results. We have discussed the results in the context of our article. The results indicate that our framework is useful for analyzing the stemming mechanisms of search engines.