Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> It's simple and atomic

and almost certainly incorrect, gotta read at least https://www.pgcon.org/2016/schedule/attachments/414_queues-p... which discusses FOR UPDATE SKIP LOCKED



that's a nice read, but does it also apply to MySQL (InnoDB)?


I've done this with mysql. Never do one at a time if you have jobs per minute over 30. It won't scale. Instead have the job dispatcher reserve 100 at a time and then fire that off to a subprocess which will subsequently fire off a process for each job. A three layer approach makes it much easier to build out multiserver. Or if you don't want the headache just use SQS which is pretty much free under 1 million jobs.


Yeah it's very basic and limited.

However if I am about to use DB as a job queue for budget reasons, I'd make sure the job doesn't get too complicated.


For me it was a lot of small jobs.

I was able to get it up to 3500 jobs an hour and likely could have gone far past that but the load on the MySQL server was not reasonable


I was able to push PqQueuer to 25k jobs a second in my benchmarking script.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: