python - Reduce Memory Usage Pandas -

-

i can't figure out way reduce memory usage program further. efficient implementation date:

columns = ['eventname', 'sessionid', "eventtime", "items", "currentpage", "browsertype"] df = pd.dataframe(columns=columns) l=[]  i, file in enumerate(glob.glob("*.log")):     print("going through log file #%s named %s..." % (i+1, file))     open(file) myfile:         l+=[json.loads(line) line in myfile]         tempdata = pd.dataframe(l)         column in tempdata.columns:             if not column in columns:                 try:                     tempdata.drop(column, axis=1, inplace=true)                 except valueerror:                     print ("oh no! we've got problem %s column! don't exist!" % (badcolumn))         l = []         df = df.append(tempdata, ignore_index = true)         # slow version, memory efficient         # length = len(df)         # length_temp = len(tempdata)         # in range(1, length_temp):         #     update_progress((i*100.0)/length_temp)         #     column in columns:         #         df.at[length+i, column] = tempdata.at[i, column]         tempdata = 0  print ("data frame initialized , filled! sorting...") df.sort(columns=["sessionid", "eventtime"], inplace = true) print ("done sorting... changing indices...") df.index = range(1, len(df)+1) print ("storing in pickles...") df.to_pickle('data.pkl')

basically, i'm reading json log files pandas dataframe, append function causing issue. creates 2 different objects in memory, causing huge memory usage. in addition, seems .to_pickle method of pandas huge memory hog, because biggest spike in memory when writing pickle. there easy way reduce memory? commented code job takes 100-1000x longer. i'm @ 45% memory usage @ max during .to_pickle part, 30% during reading of logs. more logs there are, higher number goes. help,

best, f

if need build dataframe pieces, more efficient construct list of component frames , combine them in 1 step using concat. see first approach below. 

# df = 10 rows of dummy data  in [10]: %%time     ...: dfs = []     ...: _ in xrange(1000):     ...:     dfs.append(df)     ...: df_concat = pd.concat(dfs, ignore_index=true)     ...:  wall time: 42 ms  in [11]: %%time     ...: df_append = pd.dataframe(columns=df.columns)     ...: _ in xrange(1000):     ...:     df_append = df_append.append(df, ignore_index=true)     ...:  wall time: 915 ms

Comments

Post a Comment

Popular posts from this blog

yii2 - Yii 2 Running a Cron in the basic template -

-
i have seen instructions on running cron in advanced template, cant figure out how on basic template. have following controller in basic controllers folder. <?php namespace app\controllers; use yii\console\controller; /** * cron controller */ class croncontroller extends controller { public function actionindex() { echo "cron service runnning"; } public function actionmail($to) { echo "sending mail " . $to; } } i have navigated root of application , tried these comands yii cron php yii cron im getting unknown comand "cron" i know old question failed find actual answer this. if @ code yii exec file you'll notice requires /common/config/aliases.php file console/config/main.php. can naturally change these or copy on these advanced template. hope solves problem wanting same thing.
Read more

asp.net - 'System.Web.HttpContext' does not contain a definition for 'GetOwinContext' Mystery -

-
i realize question might seem trivial some, it's these types of things find myself fighting quite bit , want make sense of despite seeming losing battle in .net (for me anyway). so, if following: using system.web; ... applicationuser user = system.web.httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); that produces error in title , red getowincontext() , error cannot resolve symbol 'getowincontext()' however, if following (remove system.web in front of httpcontext ), works expected (or @ least no errors): using system.web; ... applicationuser user = httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); however, if (same line that's working using system.web commented out): //using system.web; ... applicationuser user = httpcontext.getowincontext().getusermana...
Read more

mercurial graft feature, can it copy? -

-
there new feature in mercurial 3, called 'graft' (graft local). job of moving change-set different branch. is there way "copy" change-set branch? i interested in in moving stuff qa branch production branch, still need code reside in qa. want copy. found , have tried few things, maybe it'll become clear more tinkering , reading... does have better way of 'cherry picking' change-sets move cross branch (or trunk)??? graft copies changesets, doesn't move them. hg graft (emphasis mine): this command uses mercurial's merge logic copy individual changes other branches without merging branches in history graph. known 'backporting' or 'cherry-picking' . default, graft copy user, date, , description source changesets.
Read more