Web Scraping - a scientific database

Question

I am searching a scientific database for abstracts of papers containing the words project management. Here is the link:

For getting abstracts, I need to click on any paper and open a new page. How can I do that for 68 papers? I program in R and bash.

Have you read the "Web Technologies" task view on CRAN? Tells you all about web scraping in R. — Spacedman, Jun 30 '15 at 08:30
Use https://www.kimonolabs.com/ for scraping. Then just use its `csv` output. — Anton Tarasenko, Jun 30 '15 at 18:20

score 3 · Answer 1 · answered Jun 30 '15 at 15:04

try RSelenium. with phantomjs since the date is requested and filled in by ajax calls. so any static web scraping tools wont work.

I managed to get the list on the first page.

http://cran.r-project.org/web/packages/RSelenium/vignettes/RSelenium-headless.html

sample of what i managed to pull.

remove( mopub, m, run , rx, x , first1)
library(RSelenium)
pjs<- phantom( pjs_cmd="C:/Users/bhavin.patel/Downloads/phantomjs-2.0.0-    windows/bin/phantomjs.exe")
Sys.sleep(5)
remDr <- remoteDriver( browserName = 'PhantomJS')
dsurl <- "http://en.journals.sid.ir/SearchPaper.aspxstr=project%20management"
remDr$open()
remDr$navigate(dsurl)
allt3 <-remDr$findElements('id', 'Table3')
lapply( allt3 , FUN=function(dst){ dst$getElementText(); })

[[1]]
[[1]][[1]]
[1] " 1 :   EFFECTIVE FACTORS ON RURAL PEOPLE’S NON-PARTICIPATION OF     MAHABAD’S DAM CATCHMENT IN WATERSHED MANAGEMENT PROJECTS\nAuthor(s): RASOULIAZAR SOLEIMAN*,FEALY SAEID\nJournal: INTERNATIONAL JOURNAL OF AGRICULTURAL MANAGEMENT AND DEVELOPMENT (IJAMAD)\nNumber: MARCH 2015 , Volume  5 , Number  1 ; Page(s) 19 To 26.\nKeyword(s): NON-PARTICIPATION, CATCHMENT, WATERSHED MANAGEMENT, MAHABAD TOWNSHIP, IRAN\nReference(s):  (0)      Citation(s):  (0) FullText:"

Tomas Pazur · Answer 2 · 2015-07-01T07:56:44.510

Another workaround is to get the listing by POST requests using curl in bash.

You can get the curl post statement from Firebug ( Firefox F12 ) under Network , filter for XHR requests and copy the last statement which requests SearchPaper.aspx?str=project+management (right-click -> copy curl-adress).

In this post request statement you have to increase the parameter ctl00$ContentPlaceHolder1$txtPageNo to a desired pagination number (1-6 in this case).

Then parse the output to a static xml parsing tool to get your data.

Web Scraping - a scientific database

2 Answers2