python - Pandas: Repeat function for current keyword if except -

-

i have built web scraper. program enters searchterm searchbox , grabs results. pandas goes through spreadsheet line-by-line in column retrieve each searchterm.

sometimes page doesn't load properly, prompting refresh.

i need way repeat function , try same searchterm if fails. right now, if return, go on next line in spreadsheet.

import pandas pd selenium import webdriver selenium.webdriver.common.keys import keys  df = pd.read_csv(searchterms.csv, delimiter=",")  def scrape(searchterm):     #loads url     searchbox = driver.find_element_by_name("searchbox")     searchbox.clear()     searchbox.send_keys(searchterm)     print "searching %s ..." % searchterm      no_result = true     while no_result true:         try:             #find results, grab them             no_result = false         except:             #refresh page , above again current searchterm - how?             driver.refresh()     return pd.series([col1, col2])  df[["column 1", "column 2"]] = df["searchterm"].apply(scrape) #executes crawl each line in csv

the try except construct comes else clause. else block executed if goes ok. :

def scrape(searchterm):     #loads url      no_result = true     while no_result:         #find results, grab them         searchbox = driver.find_element_by_name("searchbox")         searchbox.clear()         try: #assumes exception thrown if there no results             searchbox.send_keys(searchterm)             print "searching %s ..." % searchterm         except:             #refresh page , above again current searchterm              driver.refresh()         else: # executed if no exceptions thrown             no_results = false             # .. post-processing code here      return pd.series([col1, col2])

(there finally block executed no matter what, useful cleanup tasks don't depend on success or failure of preceding code)

also, note empty except catches exceptions , never idea. i'm not familiar how selenium handles errors, when catching exceptions, should specify exception expecting handle. how, if unexpected exception occurs, code abort , you'll know bad happened.

that why should try keeping few lines possible within try block.


Comments

Post a Comment

Popular posts from this blog

yii2 - Yii 2 Running a Cron in the basic template -

-
i have seen instructions on running cron in advanced template, cant figure out how on basic template. have following controller in basic controllers folder. <?php namespace app\controllers; use yii\console\controller; /** * cron controller */ class croncontroller extends controller { public function actionindex() { echo "cron service runnning"; } public function actionmail($to) { echo "sending mail " . $to; } } i have navigated root of application , tried these comands yii cron php yii cron im getting unknown comand "cron" i know old question failed find actual answer this. if @ code yii exec file you'll notice requires /common/config/aliases.php file console/config/main.php. can naturally change these or copy on these advanced template. hope solves problem wanting same thing.
Read more

asp.net - 'System.Web.HttpContext' does not contain a definition for 'GetOwinContext' Mystery -

-
i realize question might seem trivial some, it's these types of things find myself fighting quite bit , want make sense of despite seeming losing battle in .net (for me anyway). so, if following: using system.web; ... applicationuser user = system.web.httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); that produces error in title , red getowincontext() , error cannot resolve symbol 'getowincontext()' however, if following (remove system.web in front of httpcontext ), works expected (or @ least no errors): using system.web; ... applicationuser user = httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); however, if (same line that's working using system.web commented out): //using system.web; ... applicationuser user = httpcontext.getowincontext().getusermana...
Read more

mercurial graft feature, can it copy? -

-
there new feature in mercurial 3, called 'graft' (graft local). job of moving change-set different branch. is there way "copy" change-set branch? i interested in in moving stuff qa branch production branch, still need code reside in qa. want copy. found , have tried few things, maybe it'll become clear more tinkering , reading... does have better way of 'cherry picking' change-sets move cross branch (or trunk)??? graft copies changesets, doesn't move them. hg graft (emphasis mine): this command uses mercurial's merge logic copy individual changes other branches without merging branches in history graph. known 'backporting' or 'cherry-picking' . default, graft copy user, date, , description source changesets.
Read more