Date of Publication :13th February 2018
Abstract: A web crawler is a software program or programmed script that browses the world extensive web in a systematic, automated manner. Web crawler peregrinates from web page to page via the making use of the graphical structure of the internet pages. Such programs are additionally kenned as robots, spiders, and worms. In this system explained further, Data mining algorithms were used to introduce intelligence into the crawler. A statistical analysis of the performance of intelligent crawler is presented in this work. While introducing crawler intelligence, data mining algorithm plays an important role. The main objective is to develop an intelligent crawler to serve the purpose of web-indexing which helps in gathering relevant information from over the Internet with the help of search engines. The proposed intelligent crawler must perform crawling in minimum time with a maximum number of results
Reference :
-
- https://en.wikipedia.org/wiki/Web_crawler
- AbhirajDarshakar, Crawler intelligence with Machine Learning and Data Mining integration, Pune Institute of Computer Technology, Katraj, Pune, India (ICCCA2015) ISBN:978-1-4799-8890-7/15/$31.00 ©2015 IEEE 849
- Shruti Sharma and Parul Gupta, The Anatomy of Web Crawlers ISBN:978-1-4799-8890-7/15/$31.00 ©2015 IEEE
- Cho, J. and Garcia-Molina, H. 2003. Estimating frequency of change.ACM Transactions on Internet Technology 3, 3 (August).
- Cho J and Hector Garcia-Molina, “The evolution of the Web and implications for an incremental crawler”, Prc. Of VLDB Conf., 2000.
- Xiang Peisu, TianKe and Huang Qinzhen, A Framework of Deep Web Crawler.
- JUNGHOO C, HECTOR GM, and LAWRENCE P. Efficient crawling through URLordering. Proceedings of the Seventh
- MirelaPirnau, Considerations on the functions and importance of a web crawler, ECAI 2015 - International Conference – 7th Edition Electronics, Computers, and Artificial Intelligence 978-1-4673-6647-/15/$31.00©2015 IEEE