We need more keywords, captain!

Brian Goetz brian.goetz at oracle.com
Tue Jan 8 17:35:48 UTC 2019

When discussing this today at our compiler meeting, we realized a few 
more places where the lack of keywords produce distortions we don't even 
notice.  In expression switch, we settled on `break value` as the way to 
provide a value for a switch expression when the shorthand (`case L -> 
e`) doesn't suffice, but this was painful for everyone.  It's painful 
for users because there's now work required to disambiguate whether 
`break foo` is a labeled break or a value break; it was even more 
painful to specify, because a new form of abrupt completion had to be 
threaded through the spec.

Being able to call this something like `break-with v` (or some other 
derived keyword) would have made this all a lot simpler. (BTW, we can 
still do this, since expression-switch is still in preview.)

Moral of the story: even just a few minutes of brainstorming led us to 
several applications of this approach that we hadn't seen a few days ago.

On 1/8/2019 10:22 AM, Brian Goetz wrote:
> This document proposes a possible move that will buy us some breathing 
> room in the perpetual problem where the keyword-management tail wags 
> the programming-model dog.
> ## We need more keywords, captain!
> Java has a fixed set of _keywords_ (JLS 3.9) which are not allowed to
> be used as identifiers.  This set has remained quite stable over the
> years (for good reason), with the exceptions of `assert` added in 1.4,
> `enum` added in 5, and `_` added in 9.  In addition, there are also
> several _reserved identifiers_ (`true`, `false`, and `null`) which
> behave almost like keywords.
> Over time, as the language evolves, language designers face a
> challenge; the set of keywords imagined in version 1.0 are rarely
> suitable for expressing all the things we might ever want our language
> to express.  We have several tools at our disposal for addressing this
> problem:
>  - Eminent domain.  Take words that were previously identifiers, and
>    turn them into keywords, as we did with `assert` in 1.4.
>  - Recycle.  Repurpose an existing keyword for something that it was
>    never really meant for (such as using `default` for annotation
>    values or default methods).
>  - Do without.  Find a way to pick a syntax that doesn't require a
>    new keyword, such as using `@interface` for annotations instead of
>    `annotation` -- or don't do the feature at all.
>  - Smoke and mirrors.  Create the illusion of context-dependent
>    keywords through various linguistic heroics (restricted keywords,
>    reserved type names.)
> In any given situation, all of these options are on the table -- but
> most of the time, none of these options are very good.  The lack of
> reasonable options for extending the syntax of the language threatens
> to become a significant impediment to language evolution.
> #### Why not "just" make new keywords?
> While it may be legal for us to declare `i` to be a keyword in a
> future version of Java, this would likely break every program in the
> world,  since `i` is used so commonly as an identifier.  (When the
> `assert` keyword was added in 1.4, it broke every testing framework.)
> The cost of remediating the effect of such incompatible changes varies
> as well; invalidating a name choice for a local variable has a local
> fix,  but invalidating the name of a public type or an interface
> method might well be fatal.
> Additionally, the keywords we're likely to want to reclaim are often
> those that are popular as identifiers (e.g., `value`, `var`,
> `method`), making such fatal collisions more likely.  In some cases,
> if the keyword candidate in question is sufficiently rarely used as an
> identifier, we might still opt to take that source-compatibility hit
> -- but names that are less likely to collide (e.g.,
> `usually_but_not_always_final`) are likely not the ones we want in our
> language. Realistically, this is unlikely to be a well we can go to
> very often, and the bar must be very high.
> #### Why not "just" live with the keywords we have?
> Reusing keywords in multiple contexts has ample precedent in
> programming languages, including Java.  (For example, we (ab)use `final`
> for "not mutable", "not overridable", and "not extensible".)
> Sometimes, using an existing keyword in a new context is natural and
> sensible, but usually it's not our first choice.  Over time, as the
> range of demands we place on our keyword set expands, this may well
> descend into the ridiculous; no one wants to use `null final` as a way
> of negating finality.  (While one might think such things are too
> ridiculous to consider, note that we received serious-seeming
> suggestions during JEP 325 to use `new switch` to describe a switch
> with different semantics.  Presumably to be followed by `new new
> switch` in ten years.)
> Of course, one way to live without making new keywords is to stop
> evolving the language entirely.  While there are some who think this
> is a fine idea, doing so because of the lack of available tokens would
> be a silly reason. We are convinced that Java has a long life ahead of
> it, and developers are excited about new features that enable to them
> to write more expressive and reliable code.
> #### Why not "just" make contextual keywords?
> At first glance, contextual keywords (and their friends, such as
> reserved type identifiers) may appear to be a magic wand; they let us
> create the illusion of adding new keywords without breaking existing
> programs.  But the positive track record of contextual keywords hides
> a great deal of complexity and distortion.
> Each grammar position is its own story; contextual keywords that might
> be used as modifiers (e.g., `readonly`) have different ambiguity
> considerations than those that might be use in code (e.g., a `matches`
> expression).  The process of selecting a contextual keyword is not a
> simple matter of adding it to the grammar; each one requires an
> analysis of potential current and future interactions.  Similarly,
> each token we try to repurpose may have its own special
> considerations;  for example, we could justify the use of `var` as a
> reserved type name  because because the naming conventions are so
> broadly adhered to.  Finally, the use of contextual keywords in
> certain  syntactic positions can create additional considerations for
> extending the syntax later.
> Contextual keywords create complexity for specifications, compilers,
> and IDEs.  With one or two special cases, we can often deal well
> enough, but if special cases were to become more pervasive, this would
> likely result in more significant maintenance costs or bug tail. While
> it is easy to dismiss this as “not my problem”, in reality, this is
> everybody’s problem. IDEs often have to guess whether a use of a
> contextual keyword is a keyword or identifier, and it may not have
> enough information to make a good guess until it’s seen more input.
> This results in worse user highlighting, auto-completion, and
> refactoring abilities — or worse.  These problems quickly become
> everyone's problems.
> So, while contextual keywords are one of the tools in our toolbox,
> they should also be used sparingly.
> #### Why is this a problem?
> Aside from the obvious consequences of these problems (clunky syntax,
> complexity, bugs), there is a more insidious hidden cost --
> distortion.  The accidental details of keyword management pose a
> constant risk of distortion in language design.
> One could consider the choice to use `@interface` instead of
> `annotation` for annotations to be a distortion; having a descriptive
> name rather than a funky combination of punctuation and keyword would
> surely have made it easier for people to become familiar with
> annotations.
> In another example, the set of modifiers (`public`, `private`,
> `static`, `final`, etc) is not complete; there is no way to say “not
> final” or “not static”. This, in turn, means that we cannot create
> features where variables or classes are `final` by default, or members
> are `static` by default, because there’s no way to denote the desire
> to opt out of it.  While there may be reasons to justify a locally
> suboptimal default anyway (such as global consistency), we want to
> make these choices deliberately, not have them made for us by the
> accidental details of keyword management. Choosing to leave out a
> feature for reasons of simplicity is fine; leaving it out because we
> don't have a way to denote the obvious semantics is not.
> It may not be obvious from the outside, but this is a constant problem
> in evolving the language, and an ongoing tax that we all pay, directly
> or indirectly.
> ## We need a new source of keyword candidates
> Every time we confront this problem, the overwhelming tendency is to
> punt and pick one of the bad options, because the problem only comes
> along every once in a while.  But, with the features in the pipeline, I
> expect it will continue to come along with some frequency, and I’d
> rather get ahead of it. Given that all of these current options are
> problematic, and there is not even a least-problematic move that
> applies across all situations, my inclination is to try to expand the
> set of lexical forms that can be used as keywords.
> As a not-serious example, take the convention that we’ve used for
> experimental features, where we prefix provisional keywords in
> prototypes with two underscores, as we did with `__ByValue` in the
> Valhalla prototype. (We commonly do this in feature proposals and
> prototypes, mostly to signify “this keyword is a placeholder for a
> syntax decision to be made later”, but also because it permits a
> simple implementation that is unlikely to collide with existing code.)
> We could, for example, carve out the space of identifiers that begin
> with underscore as being reserved for keywords. Of course, this isn’t
> so pretty, and it also means we'd have a mix of underscore and
> non-underscore keywords, so it’s not a serious suggestion, as much as
> an example of the sort of move we are looking for.
> But I do have a serious suggestion: allow _hyphenated_ keywords where
> one or more of the terms are already keywords or reserved identifiers.
> Unlike restricted keywords, this creates much less trouble for
> parsing, as (for example) `non-null` cannot be confused for a
> subtraction expression, and the lexer can always tell with fixed
> lookahead whether `a-b` is three tokens or one. This gives us a lot
> more room for creating new, less-conflicting keywords. And these new
> keywords are likely to be good names, too, as many of the missing
> concepts we want to add describe their relationship to existing
> language constructs -- such as `non-null`.
> Here’s some examples where this approach might yield credible
> candidates. (Note: none of these are being proposed here; this is
> merely an illustrative list of examples of how this mechanism could
> form keywords that might, in some particular possible future, be
> useful and better than the alternatives we have now.)
>   - `non-null`
>   - `non-final`
>   - `package-private` (the default accessibility for class members, 
> currently not denotable)
>   - `public-read` (publicly readable, privately writable)
>   - `null-checked`
>   - `type-static` (a concept needed in Valhalla, which is static 
> relative to a particular specialization of a class, rather than the 
> class itself)
>   - `default-value`
>   - `eventually-final` (what the `@Stable` annotation currently suggests)
>   - `semi-final` (an alternative to `sealed`)
>   - `exhaustive-switch` (opting into exhaustiveness checking for 
> statement
>     switches)
>   - `enum-class`, `annotation-class`, `record-class` (we might have 
> chosen these
>      as an alternative to `enum` and `@interface`, had we had the option)
>   - `this-class` (to describe the class literal for the current class)
>   - `this-return` (a common request is a way to mark a setter or 
> builder method
>     as returning its receiver)
> (Again, the point is not to debate the merits of any of these specific
> examples; the point is merely to illustrate what we might be able to do
> with such a mechanism.)
> Having this as an option doesn't mean we can't also use the other
> approaches when they are suitable; it just means we have more, and
> likely less fraught, options with which to make better decisions.
> There are likely to be other lexical schemes by which new keywords can
> be created without impinging on existing code; this one seems credible
> and reasonably parsable by both machines and humans.
> #### "But that's ugly"
> Invariably, some percentage of readers will have an immediate and
> visceral reaction to this idea.  Let's stipulate for the record that
> some people will find this ugly.  (At least, at first.  Many such
> reactions are possibly-transient (see what I did there?) responses
> to unfamiliarity.)

More information about the amber-spec-experts mailing list