ScraCOVID-19: Digital content information platform through Scraping and NoSQL storage

Authors

DOI:

https://doi.org/10.17981/ingecuc.16.2.2020.18

Keywords:

data analysis, NoSQL Database, digital communication, web page, information extraction

Abstract

Introduction— Keeping the community informed about the recent pandemic caused by COVID-19 has become a necessity, making the use of reliable communication channels accurate and evidence-based information indispensable.

Objectives— His work has as main objective to create ScraCOVID-19 on a connected digital content web platform to access updated news quickly. As a case study, four digital media are managed with national license. The news is presented in a summarized way to allow readers, depending on their interest, to read the news through some filters such as: unemployment, education, abuse, corruption and discrimination.

Methodology— ScraCOVID-19 is created from the Scraping extraction technique, using BeautifulSoup, a library that allows information in HTML format to be extracted from various websites, using the Python programming language. Results: As a result, a categorization model is described that extracts useful information to classify information into categories by referring to the URL.

Conclusions— It is concluded that, from extraction techniques used in conjunction with unstructured data storage tools, information is obtained from different web pages and all the data collected on the same dynamically generated web is managed.

Downloads

Download data is not yet available.

References

A. Landers, R. N., Brusso, R. C., Cavanaugh, K. J. & A. B. Collmus, “A primer on theory-driven web scraping: Automatic extraction of big data from the Internet for use in psychological research,” Psychol. Mhetods, vol. 21, no. 4, 475–492, 2016. https://doi.org/10.1037/met0000081

R. S. Chaulagain, S. Pandey, S. R. Basnet & S. Shakya, “Cloud Based Web Scraping for Big Data Applications,” presented at 2nd IEEE International Conference on Smart Cloud, SmartCloud, NY, USA, 3-5 Nov. 2017, pp. 138–143. https://doi.org/10.1109/SmartCloud.2017.28

E. Uzun, “A Novel Web Scraping Approach Using the Additional Information Obtained from Web Pages,” IEEE Access, vol. 8, pp. 61726–61740, 2020. https://doi.org/10.1109/ACCESS.2020.2984503

AMI, “AMI en los medios de comunicación,” ami.org, 2020. https://ami.org.co/ami-en-los-medios-de-comunicacion/.

ASOMEDIOS, “Medios Digitales,” asomedios.com, 2020. http://www.asomedios.com/medios-digitales/

S. C. M. de S Sirisuriya, “A Comparative Study on Web Scraping,” presented at 8th International Research Conference IRS, KDU, RML, LK, 27-28 Aug. 2015, pp. 135–140. Available from http://ir.kdu.ac.lk/bitstream/handle/345/1051/com-059.pdf?sequence=1&isAllowed=y

N. R. Haddaway, “The Use of Web-scraping Software in Searching for Grey Literature,” Grey J, vol. 11, no. 3, pp. 186–190, 2015.

L. Citra, Meiliana & A. Chandra, “Social media web scraping using social media developers API and regex,” Procedia Comput Sci, vol. 157, pp. 444–449, 2019. https://doi.org/10.1016/j.procs.2019.08.237

A. Josi, L. A. Abdillah & Suryayusra, “Penerapan teknik web scraping pada mesin pencari artikel ilmiah,” SISFO, vol. 5, pp. 1–6, 2014. Available: https://arxiv.org/abs/1410.5777

D. M. Thomas & S. Mathur, “Data Analysis by Web Scraping using Python,” presented at 3rd International Conference on Electronics and Communication and Aerospace Technology, ICECA 2019, CJB, IN, 12-14 Jun. 2019, pp. 450–454. https://doi.org/10.1109/ICECA.2019.8822022

D. K. Mahto & L. Singh, “A dive into Web Scraper world,” presented at 3rd International Conference on Computing for Sustainable Global Development, INDIACom 2016, New DEL, IN, 16-18 Mar. 2016, pp. 689–693.

R. Mitchell, Web Scraping with Python: Collecting More Data from the Modern Web. Sebastopol, USA: O'Reilly Media, 2018.

R. Diouf, E. N. Sarr, O. Sall, B. Birregah, M. Bousso & S. N. Mbaye, “Web Scraping: State-of-the-Art and Areas of Application,” presented at 2019 IEEE International Conference on Big Data, Big Data 2019, LA, USA, 9-12 Dec. 2019, pp. 6040–6042. https://doi.org/10.1109/BigData47090.2019.9005594

U. Baskaran & K. Ramanujam, “Automated scraping of structured data records from health discussion forums using semantic analysis,” Inform Med Unlocked, vol. 10, pp. 149–158, 2018. https://doi.org/10.1016/j.imu.2018.01.003

I. Hui, “Shaping the Coast with Permits: Making the State Regulatory Permitting Process Transparent with Text Mining,” Coast Manag, vol. 45, no. 3, pp. 179–198, 2017. https://doi.org/10.1080/08920753.2017.1303694

M. Z. Kurdi, “Text Complexity Classification Based on Linguistic Information: Application to Intelligent Tutoring of ESL,” JDMDH, pp. 1–38, 2020. Available: https://arxiv.org/abs/2001.01863

L. Junjoewong, S. Sangnapachai & T. Sunetnanta, “ProCircle: A promotion platform using crowdsourcing and web data scraping technique,” presented at 7th ICT International Student Project Conference, ICT-ISPC 2018, Nakhon, TH, 11-13 Jul. 2018, pp. 1–5. https://doi.org/10.1109/ICT-ISPC.2018.8524003

E. N. Sarr, O. Sall & A. Diallo, “FactExtract: Automatic C ollection and A ggregation of A rticles and J ournalistic F actual Claims from Online Newspaper,” 5 International Conference on Social Networks Analysis, Management and Security, SNAMS, VA, ES, 15-18 Oct. 2018, pp. 336–341. https://doi.org/10.1109/SNAMS.2018.8554421

Alexa, “Top Sites in Colombia,” Amazom Company, 2020. https://www.alexa.com/topsites/countries/CO

J. Díez, “Aplicación para monitorización de precios para Android,” Trabajo grado, ETS, UPNA, PNA, ES, 2019. Disponible en https://hdl.handle.net/2454/33693

C. Lopezosa, L. Codina & C. Gonzalo-Penela, “Off-page SEO and link building: General strategies and authority transfer in the digital news media,” Prof Inf, vol. 28, no. 1, pp. 1–14, 2019. https://doi.org/10.3145/epi.2019.ene.07

R. Bahana, R. Adinugroho, F. L. Gaol, A. Trisetyarso, B. S. Abbas & W. Suparta, “Web crawler and back-end for news aggregator system (Noox project),” presented at 2017 IEEE International Conference on Cybernetics and Computational Intelligence, Cybernetics, HKT, TH, 20-22 Nov. 2018, pp. 56–61. https://doi.org/10.1109/CYBERNETICSCOM.2017.8311684

Downloads

Published

2020-10-19

How to Cite

Sánchez Paipilla, A. G., Durán Vaca, M. K., González Amarillo, A. M., & Ballesteros Ricaurte, J. A. (2020). ScraCOVID-19: Digital content information platform through Scraping and NoSQL storage. INGE CUC, 16(2), 229–237. https://doi.org/10.17981/ingecuc.16.2.2020.18

Issue

Vol. 16 No. 2 (2020): (Julio-Diciembre)

Section

In Press

License

Copyright (c) 2020 INGE CUC

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Published papers are the exclusive responsibility of their authors and do not necessary reflect the opinions of the editorial committee. 

INGE CUC Journal respects the moral rights of its authors, whom must cede the editorial committee the patrimonial rights of the published material. In turn, the authors inform that the current work is unpublished and has not been previously published. 

All articles are licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Licencia Creative Commons

Most read articles by the same author(s)