Propose adopting the Rust RFC process

[calebamiles]

I believe it is time to adopt a new development process given the
current scale of Kubernetes. The inadequacy of our current process
is particularly apparent within SIGs with cross project concerns
such as SIG Release, SIG Testing, SIG Architecture, and SIG PM
so additional process around proposing changes to Kubernetes is
suggested.

[derekwaynecarr]

I think this is clearer than the kube feature process at first glance, can we do a walk through for how a sample feature already implemented would have gone in this flow? Ideally, a feature that has gone from alpha->stable?

[calebamiles]

That sounds like a great idea, @derekwaynecarr. I can try to put up a more concrete example in the next day or so time permitting. Perhaps we could choose some of the work that has come out of SIG Node?

[smarterclayton]

One more item:

• Ensure community participants are able to successfully drive changes to completion across one or more releases
  • Ensure stakeholders are adequately represented throughout

Which may overlap with above and below, but I think that the goal of the development process is to manage change across releases, not just within them, and releases are vehicles for features (rather than the other way around)

[jdumars]

+1

[bgrant0607]

More items in the email thread:
https://groups.google.com/forum/#!topic/kubernetes-sig-architecture/t-1EqeEoLPA

[calebamiles]

I agree entirely @smarterclayton that releases are vehicles for features and not the other way around. I was trying to motivate a process which would tie together the continuous work of proposing, scoping, and implementing features, with how the results of that work would filter into user or developer facing documentation so that people can actually consume the new features.

[ghost]

I hate to throw a spanner in the works so late in the process, but I have a concern/suggestion. It seems that we're trying to address two mostly independent sets of concerns here:

1. Product and technical integrity

   1. Do we want this proposed thing (feature addition, enhancement, defect correction etc) to be applied to Kubernetes?
   2. If so, what exactly should it look like to users?
   3. What should the internal implementation look like?
   4. How do we ensure that it can be designed and implemented as quickly/efficiently as possible?

2. Progress tracking and communication.

   1. When will users be able to use it.
   2. How should users use it.

For in-SIG things, the answers to the above are fairly straightforward. An individual SIG makes most of the decisions in relative isolation, and communicates via their meetings, notes, roadmap, release notes etc.

For cross-SIG things, it's clearly more complicated, and that's what we're trying to address here.

I would suggest that we first focus squarely on answering 1.1-1.4 above, and only then try to answer 2.1-2.2 within the above context.

[bgrant0607]

for reference: https://groups.google.com/d/msg/kubernetes-sig-architecture/t-1EqeEoLPA/B_LCuU5xBAAJ

[jdumars]

A foundational concern for this proposal would be what constitutes a "change" that is in bounds for the process. In the current workflow, I believe much of our struggle comes from arbitrary boundaries between features, enhancements, defect corrections, and technical debt elimination. One person's bug fix is another person's feature. Brian Grant and others have been fairly prescriptive about these definitions, but the implementation of a proscriptive policy will force the issue. Because open collaboration contributions usually follow the path of least resistance, it means that an increasing amount of work will bypass the process over time. And, if the idea of the RFC process is to seed the entirety of feature and architecture backlogs, having it out of alignment with the actual code could present a serious risk. In order to address this, extensive referential automation, documentation, and human attention may be required. We will also need a very clear knife edge between all of the types of "changes" I mentioned above, as well as easy access to decision-making guidance. If you study the pitfalls of implementing technical change management, you quickly learn how such processes can lead to disaster either through subversion or dropping velocity precipitously. None of this is a disagreement with the premise of this PR, only illuminating known antipatterns.

[calebamiles]

I definitely think that's a good point about work bypassing any process in order to follow the path of least resistance. I am hoping that SIGs will find the process useful enough to push back against exceptions on their own accord. Once we've decomposed the monorepo I think there will be a natural desire at the SIG level to enforce some sort of structure around change management in order to manage their own exposure to risk

[jbeda]

Practically, any work that significantly crosses SIG boundaries (and touches some special areas like APIs) will require coordination and this process. As we move to associating code with SIGs, it'll become more more apparent when this happens.

Even though an individual person may have the authority to approve changes that cut across SIGs, we expect those folks to use their power wisely. Instead of approving a big PR that hits a lot of areas, those are the folks that would suggest that a proposal be generated.

[calebamiles]

@jbeda I feel that the production of release notes is a special case of work significantly crossing a SIG boundary. In my limited sampling of feature issues and pull requests there is a huge burden placed on our tech writers to explain changes in a format consumable by people besides the developers of a feature. There is also a large burden placed on the PM Group and the Release Team to actually determine what went into the release. in my opinion this is a huge risk for the future success of the project because we need users to feel that it's important to keep their clusters up to date, however, if we can't produce a compelling narrative which encourages people to upgrade their critical infrastructure we're in bad shape as a project

[jdumars]

I might qualify this statement and say it's difficult to know the chain of custody of any change in Kubernetes. That chain currently includes all program-level SIGs, sponsor SIGs, reviewers, approvers, automation, and eventually a release team. In the scope of this proposal, you add conceptual stewardship on top of it all. It's critical that even a one-line change has a defined lifecycle. We also need visibility of all the custodial handoffs so we know when things stall, or reach a bottleneck in the value stream. That will allow traceability, and the ability to focus attention on areas of the RFC and release process that need it the most. (My magic 8 Ball says architectural review will be an early bottleneck)

[jbeda]

The "one line change" thing is scaring me, to be honest. There will be a significant number of bug fixes and enhancements that are purely contained within a single SIG and shouldn't have to activate this process.

[calebamiles]

I think that's up to the SIG to decide as part of their chartering document. I do not agree that there will be a significant number of enhancements that fall within a single SIG that would not benefit from a more formalized process that will allow SIG Docs, SIG PM to communicate the importance of those enhancements beyond the SIG. I'm sure @radhikapc or @Bradamant3can can explain the challenge of communicating what went into the release and why it's important far better than I can.

[jdumars]

I would go one step further and say the current process is insufficient for the scale of the project. This is a serious risk that needs attention before 1.9 ships.

[jbeda]

This is great context to put in this PR, but not in this document. 2 years from now "our current mechanism" will mean something different.

[jdumars]

I agree on the ubiquity of Git. That said, a VCS is not a project management tool, nor a cogent way of managing an architectural catalog/backlog. An insistence on finding ways to "make Git work" for various processes is going to lead to suboptimal results. We need to be explicit about its role in the ecosystem, as well as identify critical tooling gaps from the program management perspective.

[jdumars]

Have there been issues in the other communities using this pattern with reviewer availability and bandwidth?

[calebamiles]

I can't speak directly to how effective the PEP Editor role has been in their community but perhaps @jbeda could help investigate the effectiveness of the role as he's expressed a good deal of admiration for the PEP process in our conversations.

[jbeda]

The key line here is "PEP editors don't pass judgment on PEPs. They merely do the administrative & editorial part (which is generally a low volume task)." This is more of an ombudsman role and is hopefully something that a wider group of folks can help with.

[jdumars]

Would there be an effort to retroactively build the architectural catalog? Or, would we just pick a day and start asking for RFCs for net new changes? How would we snapshot the current state sufficient to provide context for all future RFCs enacted against it?

[calebamiles]

I think we would pick a day. Perhaps SIGs could go through what's currently in design-proposals and prune out non implemented designs but I think we'd probably archive the old design documentation that had no active work around its implementation.

[jdumars]

Caleb, this is an excellent proposal. Thank you for being so thorough and thoughtful, as always.

[bgrant0607]

cc @kubernetes/sig-architecture-proposals @kubernetes/sig-release-proposals @kubernetes/sig-testing-proposals

Hmm. There are no sig-pm teams.

[bgrant0607]

I think this is at a good place to checkpoint and iterate further, so I'm going to merge.
/lgtm

[justaugustus]

/remove-sig pm
/sig architecture
/area enhancements