i've got rather large php web app gets products numerous others suppliers through api's, responding large xml parse. there 20 suppliers due rise further.
our current set uses multi curl make requests , takes 30-40 seconds complete , long. script runs in background whilst front end polls database looking results , displays them come in.
to improve process thinking of using job server run in background, each supplier request being separate job. we've seen beanstalkd , gearman being mentioned.
so looking in right direction, in, job server right way go? we're looking @ doing promotion may 200+ users searching 30 suppliers @ same time right choice needs scale if have load balance.
any advice great received.
you can use beanstalkd, can customize priority of jobs , ttr time-to-resolve, default 60 seconds, scenario must increase it. there nice admin console panel beanstalkd.
you should leverage multi curl calls, should use parallel requests. in order make use of keep-alive need maintain pool of curl handles , keep them warm. see high performance curl tips. need tune linux network stack.
if run in cloud, make sure use multiple micro machines rather 1 heavy machine throughput better when have multiple resources available.
