JMH and continuos integration

Mark Price Mark.Price at
Tue Nov 15 15:09:22 UTC 2016

Hi Leonardo,
at LMAX, we run a number of JMH benchmarks continuously. 

We also try to make sure that there is minimal OS jitter in the results [1], using cpu isolation, sched_setaffinity, and making sure that the benchmark thread gets its own CPU, with no contention from child/spawned threads.

We record the JSON output in a database, and generate charts based on these results. We have tried (unsuccessfully) to implement a pass/fail CI job that will automatically fail if there are regressions in performance. Unfortunately, we haven't got around to open-sourcing the component that persists results & generates charts.

We found that even with a very highly tuned system, there was enough inter-run noise that we would get false positives frequently. This led to broken-windows syndrome, so these days we just regularly eyeball the charts for any obvious signal. We record the timestamp and revisions associated with each test-run, so finding a culprit isn't too hard.

With warm-up iterations, measurement iterations and multiple forks, the feedback-loop can become longer than we'd like. Our micro-benchmarks currently take over an hour to run, though with more hardware we could run them in parallel to improve this. That's still not bad, but for comparison, our suite of ~11k acceptance tests only takes ~25mins...

I'd be interested to hear of any headway you make in this area.



----- On 11 Nov, 2016, at 00:12, Leonardo Gomes leonardo.f.gomes at wrote:

> Hi,
> I'm looking for feedback on how people use JMH as part of their development
> process.
> I've played a bit with and am
> wondering if people in this discussion list do use any sort of automation
> (Jenkins or other) to detect performance regression on their software, on a
> pull-request basis.
> I've seen JMH being used on different open-source projects, but the
> benchmarks seem to have been introduced mostly to compare different
> implementations (when introducing / validating changes) and not as part of
> CI.
> Any feedback would be appreciated.
> Thank you,
> Leonardo.

LMAX Exchange, Yellow Building, 1A Nicholas Road, London W11 4AN

Recognised by the most prestigious business and technology awards
2016 Best Trading & Execution, HFM US Technology Awards
2016, 2015, 2014, 2013 Best FX Trading Venue - ECN/MTF, WSL Institutional Trading Awards

2015 Winner, Deloitte UK Technology Fast 50
2015, 2014, 2013, One of the UK's fastest growing technology firms, The Sunday Times Tech Track 100
2015 Winner, Deloitte EMEA Technology Fast 500
2015, 2014, 2013 Best Margin Sector Platform, Profit & Loss Readers' Choice Awards


FX and CFDs are leveraged products that can result in losses exceeding your deposit. They are not suitable for everyone so please ensure you fully understand the risks involved.

This message and its attachments are confidential, may not be disclosed or used by any person other than the addressee and are intended only for the named recipient(s). This message is not intended for any recipient(s) who based on their nationality, place of business, domicile or for any other reason, is/are subject to local laws or regulations which prohibit the provision of such products and services. This message is subject to the following terms (, if you cannot access these, please notify us by replying to this email and we will send you the terms. If you are not the intended recipient, please notify the sender immediately and delete any copies of this message.

LMAX Exchange is the trading name of LMAX Limited. LMAX Limited operates a multilateral trading facility. LMAX Limited is authorised and regulated by the Financial Conduct Authority (firm registration number 509778) and is a company registered in England and Wales (number 6505809).

LMAX Hong Kong Limited is a wholly-owned subsidiary of LMAX Limited. LMAX Hong Kong is licensed by the Securities and Futures Commission in Hong Kong to conduct Type 3 (leveraged foreign exchange trading) regulated activity with CE Number BDV088.

More information about the jmh-dev mailing list