Gathering metrics for the Adoption programme
dalibor.topic at oracle.com
Tue Feb 18 11:06:30 PST 2014
On 05.02.2014 18:00, Martijn Verburg wrote:
> Hi all,
> **Important - please remember to sign-up to the
> adoption-discuss at openjdk.java.net mailing list and reply to this thread
> there please**
> We'd like to measure & report to the OpenJDK governing board some
> quantifiable metrics on the impact and effectiveness of the Adoption
> programme. These metrics will also be useful to the group in terms of
> helping us determine whether we are fulfilling our mission statement and
Thanks for kicking off the discussion on this item, Martijn!
To set some context, here's what the Bylaws require the Group Lead to do:
"The obligation, once per quarter, to publish a written report
summarizing the recent activities of the Group."
As there is no specific format required, we have some liberty in terms
of what information we want to report, where to report it, and so on.
Let's start with the 'where' and 'when' parts first.
While Martijn's e-mail says 'report to the OpenJDK governing board',
that's actually not what the Bylaws ask for, and not what the Governing
Board asked for (see
for the actual request:
"Unlike other groups/projects, keeping to a schedule
of quarterly reports seems appropriate."). So I would suggest that
* the quarterly report is compiled on the Group's Wiki,
* starting in April 2014, and
* published by the Group Lead as a finished copy to this mailing list,
allowing followup and discussion by all Participants
which includes Governing Board Members, of course, should they chose to
participate on this mailing list.
Now, on to the 'what' question.
What activities are there to report about for Groups? Again, looking at
the Bylaws we find:
"A Group is a collection of Participants who engage in open conversation
about a common interest. "
"Groups may have web content and one or more mailing lists. Groups do
not have code repositories of their own but they may sponsor Projects,
so based on that, the metrics that we should be looking for are around
conversations, the Group's web content and mailing lists.
If the Adoption Group were to sponsor Projects which publish quarterly
report themselves, then such reports could be referenced by the
quarterly report of this Group, but compiling such reports would be the
responsibility of their respective Project Leads, as defined in
http://openjdk.java.net/bylaws#project-lead - and not this Group's task.
Some further exploration of notable activities in the Bylaws can be
found in http://openjdk.java.net/bylaws#_5
"A Member of a Group has write access to the Group’s web content and
"A Member of a Group may nominate any Contributor to be a new Member of
"A Group’s Lead has:
The authority to sponsor Projects;
The obligation to act as a contact point for the Group and to look
after the Group’s mailing lists and web content; and
The obligation, once per quarter, to publish a written report
summarizing the recent activities of the Group."
"A Group’s Lead may delegate selected obligations, but not authorities,
to other of that Group’s Members as desired."
So further Group activities to measure and report could be
* web content updates and changes (like Wiki spring cleaning)
* Group Member nominations and other Group membership changes
* sponsoring of Projects
* publication of written quarterly reports
Another thing to keep in mind is that just because something could be
measured, it may not actually be useful information. For example, while
a quarterly report could include a word count of e-mails sent to this
mailing list in a given period, it would provide no actually useful
information for the OpenJDK Community or this Group to do so. ;)
So instead of thinking of the things that we could measure, I think it's
important to apply the concept of 'Datensparsamkeit'
(http://martinfowler.com/bliki/Datensparsamkeit.html for some mildly
related thoughts) and ask what we really have to measure, and then go
Another important consideration from my point of view is that
information included in the reports should either be publicly available
or compilable from publicly available sources. As any such reports would
be compiled by people, and "people are people", as a wise philosopher
once said, exclusively relying on publicly available or compilable
information would allow errors to be detected and corrected prior to the
Finally, an important consideration is that a Group Lead could only
measure those activities directly taking place within OpenJDK, as those
would be the activities they would be responsible in some way or another
for coordinating and directing, as defined in the Bylaws.
That doesn't mean that other information would not be interesting to
reference in quarterly reports, and if someone compiles quarterly
updates on, say, edits of OpenJDK pages on Wikipedia, then that may be
interesting information to reference, but it should not be something
that is expected to be tracked and reported on an ongoing basis as an
activity of this Group.
That was a lot to read. Now on to the specifics!
> Initial Onboarding:
> * Number of mailing list users
That number is not publicly available. As the mailing list moderator, I
could find out, and report on that, but I don't think that it's an
actually useful number, and besides, it's trivial to game.
A more important number is the number of users who posted to the mailing
lists in a quarter, I think. It's not that trivial to compile, but since
the archives are publicly available, it can be publicly verified, by
looking at the archives:
for February (I believe the current count is 13).
The reason why the number is interesting is that activity on the mailing
list is a usual way for new Contributors to make their mark on the
Group, and through lasting contributions become Group Members
themselves. So in a way, that number, when offset against the number of
Group Members, gives us an very rough idea of whether the Group is
attracting fresh ideas outside its core membership.
> * Message volume on the mailing list
That is public information available on the page cited above - "
Messages: 63" it says, and should be at least 64 by the time this e-mail
I think that's a useful number to track, though I don't think that its
resolution as such is very useful. If mailing list traffic goes from 216
e-mails in a month to 252 in the next month, is it relevant? Probably
not very much. If it goes to 2045? It may be.
I don't have a better resolution other than "none, few, dozens,
hundreds" to propose though - and that seems a bit arbitrary. So let's
stick with reporting the number as is.
> * Number of new developers successfully building OpenJDK / Downloads of
> OpenJDK development virtual machine
I don't think that the former number could be tracked, as there is no
'callback' mechanism in the build. Nor should there be one, I think.
The latter number is a no op for this Group, as Groups do not produce
artifacts you can download - that's something Projects do.
> * Number of unique visitors to Adoption page/wiki
I'm not aware of any public information on that, or if it exists, to
begin with. I don't think that's actually very useful information as it
is trivial to game, Datensparsamkeit aside.
On the other hand, since Adoption Group Members can edit the wiki, I
think information on the number of edits, new pages and active editors
in a quarter could be useful information to collect and track, as that
would be a measure of how well the Group is utilizing that part of its
Looking at the page information menu entry point in the wiki, I don't
see any further items on which information is available that would be
> ** Perhaps measure impact on overall OpenJDK visits
I'm not aware of any public source of such information, either, as above.
> Betterev is a proposed patch workflow & build farm system. It will act as
> community clearing house for new contributors and as a prototype for
> OpenJDK infrastructure.
I think that if a Betterrev Project were to be established in OpenJDK,
then its quarterly report could contain all kinds of interesting data on
its activities, but I don't think a Project's data belongs in a Group's
quarterly report. A reference to a Project's quarterly report would be
perfectly fine, though.
I'd relabel that category 'QA Outreach' to differentiate it from ongoing
OpenJDK development focused QA efforts which some Projects may have.
This is a great example of contributions users can make to improve the
JDK without having to be or become experts in the implementation themselves.
> * Number of FOSS projects involved in testing OpenJDK nightly builds
I think that this is an important metric, though, but not at that
granularity - basically, I think the metric we should generally care
about is the number of FOSS projects approached by this Group's Members
providing active feedback on OpenJDK builds, whether their own, or from
someone else, whether those are nightly, weekly, monthly, or release
builds, and the volume of such feedback.
So - an example of great information to report would be: P1 issues filed
and fixed in JDK Bug System based on this Group's QA outreach efforts in
a given quarter. I think that would be an a lot more interesting list
than a line like '18 projects' in a column on a report or '19' the next
Assuming the QA outreach effort was tracked on our wiki, along with the
reported issues, such a list should not be very hard to compile and verify.
> * Number of bugs/issues reviewed by adoption group for quality control
Technically, Reviewer is a Project-specific role, rather than
If we redefine this item to mean something like 'major issues this Group
assisted with' then I think we have something more interesting to
report, which may not be as neatly quantifiable as a number, but is
probably more interesting as a report on activity than a line that says
'reviewed 10 issues this quarter'. ;)
> Longterm Onboarding:
> * Number of 'graduates' from Adoption programme into official OpenJDK
> contributors, authors, committers and reviewers.
This could be partly derived from the OpenJDK census at
http://openjdk.java.net/census . I don't think that tracking Contributors
is that useful, as being a Contributor does not imply having made a
code contribution or ever getting around to making one.
On the other hand, you need at least two code contributions to a Project
to become an Author on a Project, so that's interesting information to
track, per se.
But ... a challenge here is the connection between the Adoption Group
membership and Project members.
A way to do it is to look at the progress in roles of Group Members of
this Group in different Projects. But becoming a Group Member requires
continuous contributions to a Group over a longer period of time:
"As a rough guide, a Contributor should be regularly active in a Group
or its sponsored Projects for at least a year before being nominated to
be a Group Member."
In other words, the set of this Group Members, whose 'graduations' we
could try to measure, is not going to change a lot over the next twelve
And so, while this is an interesting measurement overall, as the name
says 'long term', it's not something I think we should include in our
initial quarterly reports. Over the long term, in 12 months, or so, it
would be interesting to look at the additional roles this Group's
Members have been voted into in other OpenJDK Groups and Projects, but
it seems premature to me to report on it in the Group's first year of
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: +494089091214 <tel:+494089091214> | Mobile: +491737185961
ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg
ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603
Geschäftsführer: Jürgen Kunz
Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Astrid Kepper, Val Maher
<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment
More information about the adoption-discuss