performance - Hive: Is there a better way to percentile rank a column? -

-

currently, percentile rank column in hive, using following. trying rank items in column percentile fall under, assigning value form 0 1 each item. code below assigns value 0 9, saying item char_percentile_rank of 0 in bottom 10% of items, , value of 9 in top 10% of items. there better way of doing this?

select item     , characteristic     , case when characteristic <= char_perc[0] 0         when characteristic <= char_perc[1] 1         when characteristic <= char_perc[2] 2         when characteristic <= char_perc[3] 3         when characteristic <= char_perc[4] 4         when characteristic <= char_perc[5] 5         when characteristic <= char_perc[6] 6         when characteristic <= char_perc[7] 7         when characteristic <= char_perc[8] 8         else 9       end char_percentile_rank (     select split(item_id,'-')[0] item         , split(item_id,'-')[1] characteristic         , char_perc     (         select collect_set(concat_ws('-',item,characteristic)) item_set             , percentile(bigint(characteristic),array(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)) char_perc         from(             select item                 , sum(characteristic) characteristic             table             group item         ) t1     ) t2     lateral view explode(item_set) explodetable item_id ) t3

note: had collect_set in order avoid self join, percentile function implicitly performs group by.

i've gathered percentile function horribly slow (at least in usage). perhaps better manually calculate percentile?

try removing 1 of derived tables

select item     , characteristic     , case when characteristic <= char_perc[0] 0         when characteristic <= char_perc[1] 1         when characteristic <= char_perc[2] 2         when characteristic <= char_perc[3] 3         when characteristic <= char_perc[4] 4         when characteristic <= char_perc[5] 5         when characteristic <= char_perc[6] 6         when characteristic <= char_perc[7] 7         when characteristic <= char_perc[8] 8         else 9       end char_percentile_rank (      select item, characteristic,          , percentile(bigint(characteristic),array(0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9)) on () char_perc       (        select item          , sum(characteristic) characteristic                     table        group item                  ) t1 ) t2

Comments

Post a Comment

Popular posts from this blog

yii2 - Yii 2 Running a Cron in the basic template -

-
i have seen instructions on running cron in advanced template, cant figure out how on basic template. have following controller in basic controllers folder. <?php namespace app\controllers; use yii\console\controller; /** * cron controller */ class croncontroller extends controller { public function actionindex() { echo "cron service runnning"; } public function actionmail($to) { echo "sending mail " . $to; } } i have navigated root of application , tried these comands yii cron php yii cron im getting unknown comand "cron" i know old question failed find actual answer this. if @ code yii exec file you'll notice requires /common/config/aliases.php file console/config/main.php. can naturally change these or copy on these advanced template. hope solves problem wanting same thing.
Read more

asp.net - 'System.Web.HttpContext' does not contain a definition for 'GetOwinContext' Mystery -

-
i realize question might seem trivial some, it's these types of things find myself fighting quite bit , want make sense of despite seeming losing battle in .net (for me anyway). so, if following: using system.web; ... applicationuser user = system.web.httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); that produces error in title , red getowincontext() , error cannot resolve symbol 'getowincontext()' however, if following (remove system.web in front of httpcontext ), works expected (or @ least no errors): using system.web; ... applicationuser user = httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); however, if (same line that's working using system.web commented out): //using system.web; ... applicationuser user = httpcontext.getowincontext().getusermana...
Read more

mercurial graft feature, can it copy? -

-
there new feature in mercurial 3, called 'graft' (graft local). job of moving change-set different branch. is there way "copy" change-set branch? i interested in in moving stuff qa branch production branch, still need code reside in qa. want copy. found , have tried few things, maybe it'll become clear more tinkering , reading... does have better way of 'cherry picking' change-sets move cross branch (or trunk)??? graft copies changesets, doesn't move them. hg graft (emphasis mine): this command uses mercurial's merge logic copy individual changes other branches without merging branches in history graph. known 'backporting' or 'cherry-picking' . default, graft copy user, date, , description source changesets.
Read more