parallel processing - PHP receive and respond to thousands of requests - do calculations behind the scene -

-

i working on php script that:

  • receives client request;
  • processes request via cpu-and-time-intensive-binary-computation
  • store result of computation mysql database
  • then respond client status 200 ok

problem: when there 10s of 1000s of requests coming in per second during peak hours: clients have wait long time receive status 200 ok.

flexibilities: script not need respond client result of computation. script not need respond status 200 ok based on success/failure of computation - computation may fail , that's okay. actual computation happen in parallel behind scene.

what tools / packages / libraries / strategies should used achieve kind of intensive request handling design on php? on php side or solvable apache side?

notes:

  • running apache, mysql, php, redis on ubuntu [ampru]
  • clients send request , receive status 200 ok right away.
  • clients not wait computation of request complete.
  • there no auto-scaling or load-balancing concept in place: it's single ampru server.
  • better if multiple computations can happen in parallel behind scenes

this classic use case queue. of tech-stack have listed, redis has support queues (check out php-resque library), or there other tools can used, such beanstalkd (a favourite of mine, pheanstalk php library), or amazon sqs. there number of other options, both self-hosted., or available services.

the website, or other mechanism receives data, , queue - returning 200 ok. back-end workers, simple cron-based system, or (better) multiple long-running (occasionally restarting clean-up) scripts, pull items queue , performs work, saving results.

i've literally run hundreds of millions of jobs this, through such systems. workers, if can reach queue , database servers, don't have run on same machines (i running dozens of workers across many servers).


Comments

Post a Comment

Popular posts from this blog

yii2 - Yii 2 Running a Cron in the basic template -

-
i have seen instructions on running cron in advanced template, cant figure out how on basic template. have following controller in basic controllers folder. <?php namespace app\controllers; use yii\console\controller; /** * cron controller */ class croncontroller extends controller { public function actionindex() { echo "cron service runnning"; } public function actionmail($to) { echo "sending mail " . $to; } } i have navigated root of application , tried these comands yii cron php yii cron im getting unknown comand "cron" i know old question failed find actual answer this. if @ code yii exec file you'll notice requires /common/config/aliases.php file console/config/main.php. can naturally change these or copy on these advanced template. hope solves problem wanting same thing.
Read more

asp.net - 'System.Web.HttpContext' does not contain a definition for 'GetOwinContext' Mystery -

-
i realize question might seem trivial some, it's these types of things find myself fighting quite bit , want make sense of despite seeming losing battle in .net (for me anyway). so, if following: using system.web; ... applicationuser user = system.web.httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); that produces error in title , red getowincontext() , error cannot resolve symbol 'getowincontext()' however, if following (remove system.web in front of httpcontext ), works expected (or @ least no errors): using system.web; ... applicationuser user = httpcontext.getowincontext().getusermanager<applicationusermanager>().findbyid(system.web.httpcontext.user.identity.getuserid()); however, if (same line that's working using system.web commented out): //using system.web; ... applicationuser user = httpcontext.getowincontext().getusermana...
Read more

mercurial graft feature, can it copy? -

-
there new feature in mercurial 3, called 'graft' (graft local). job of moving change-set different branch. is there way "copy" change-set branch? i interested in in moving stuff qa branch production branch, still need code reside in qa. want copy. found , have tried few things, maybe it'll become clear more tinkering , reading... does have better way of 'cherry picking' change-sets move cross branch (or trunk)??? graft copies changesets, doesn't move them. hg graft (emphasis mine): this command uses mercurial's merge logic copy individual changes other branches without merging branches in history graph. known 'backporting' or 'cherry-picking' . default, graft copy user, date, , description source changesets.
Read more