pkg directory should not be recommended for use

[theckman]

The general consensus of the Go community has been converging on the idea that the pkg/ directory is a useless abstraction / indirection. Instead, people are encouraged to either put the packages in the top-level of the repository, or to create a repository structure that structures things based on their primary functionality of domain in the business logic.

So while in your example you may encourage people to have the following layout:

pkg/
  server/
    handlers/
  poller/
    config/
    manager/

We instead would prefer:

server/
  handlers/
poller/
  config/
  manager/

This may seem innocuous, but because this project has asserted itself as some sort of Go standards body what is being recommended here is being taken seriously. In the Slack workspace, where there are over 30k registered users, we quite often need to correct people who are using pkg/ and have them ignore the recommendations here. Some of the things recommended here are good, and I'd like to be able to use them as a resource. That said, considering the current pkg/ recommendation we're hard-pressed to use this as a resource because we have to tell people to ignore certain parts.

[rburmorrison]

@theckman Interesting! I'll certainly consider this adopting this. What is the general reason that using the pkg/ directory is frowned upon, may I ask? Is it due to long import paths? I tend to like it because I can name my root folder what I want as to not conflict with my other github projects, and keep the actual package name separate within the pkg/ directory. I also like that I can keep configuration files and READMEs separate from my source code. But it is certainly a long import path. Also, do you tend to agree with the cmd/ directory, as it is similar to the usage of the pkg/ directory? Thanks!

[kcq]

Can you be a bit more specific about the 'we' part. Who exactly are you talking about. Is there an official community consensus around this (controversial) pattern? Is there a consensus from the official go dev team? At least, the last interaction around it on the gophers slack channel is limited mostly to you saying don't use the 'pkg' pattern :-) There might be other threads and discussions I didn't see and that's why I'm asking.

This repo attempts to capture the existing or emerging patterns in the Go ecosystem / project community. The 'pkg' pattern is used in a number of high profile and popular projects. Some of those projects might not be great examples others should follow, but it would be a bit suspicious to say that all projects leveraging the 'pkg' pattern are bad examples :-) Of course, it's not something that the majority of projects out there do and it appears there are some strong feelings against it, but i'm not sure if there's a unanimous agreement that it's a bad pattern. I definitely agree that it's a confusing pattern for new go devs.

I'll be happy to add more context and the necessary references to the emerging community recommendations.

[theckman]

@rburmorrison I'll start by saying I'm a proponent of cmd/, especially when you have multiple binaries being built. It gives you a good starting point to understand how the library is imported / consumed. 👍

For pkg/, there's a few reasons why I think it's started to fall out of favor. The first being that any packages not encapsulated within an internal directory are considered publicly consumable. You can import them, and the compiler will allow you to build the software. And so whether or not it's included in pkg people can consume it. If you don't want to expose it, you should absolutely use the internal package convention to do so. This will cause the compiler to fail to build because it's an internal package.

The second reason kind of has to do with long paths, but it's also that pkg doesn't provide a useful grouping for me. If I want to see where the source code lives, I've tools that can easily walk the tree and tell me that. Even then I generally don't use them, because I'm either already familiar with the library or I'm reading code that uses the library and deducing where the code is based on import statements.

[theckman]

@kcq I'm speaking in terms of what has become a general consensus in the Go Slack and Forums areas. The Go authors themselves have been absent in providing guidance on this topic, rightfully so.

There are discussions beyond ones involving me (based on the interactions I've seen), and my opinion was shifted from liking pkg/ to not after having become involved in the Go community itself. I think I shifted to this opinion well-before the date of the first commit of this repo.

Library code that's safe to use by external applications (e.g., /pkg/mypubliclib).

I think one issue I have with explanations like this, is that the internal/ directory is where code that's unsafe for use by external applications is meant to go. And so if it's outside of an internal directory, the author is saying it's safe to use as an external application no matter the top-level directory. It may be better to say it's one location where people sometimes put library code, and make no mention of the external consumption part?

[nineinchnick]

I don't know if a Tweet can/should be used as source but Brad Fitzpatrick states pkg is legacy: https://twitter.com/bradfitz/status/1039512487538970624
There is definitely some momentum though.

[kcq]

@nineinchnick yes, it's a 'legacy' pattern for Go itself. Brad's tweet provides a nice background about the origins of the pattern :)

[hron84]

@theckman While I'm not a Go developer, I do know a little bit about development in general terms. You said:

think one issue I have with explanations like this, is that the internal/ directory is where code that's unsafe for use by external applications is meant to go. And so if it's outside of an internal directory, the author is saying it's safe to use as an external application no matter the top-level directory.

The problem is you made a direct conclusion that if something is not in internal directory then it's safe to use by external application/project which is not neccessarily true. Think about cmd directory, while the proposal says only importing things and launching things should go here, things could be more difficult and this code is still not meant to public use/re-use.

I personally like the idea of pkg directory, even if I come from Java/Ruby world, because if something is in this directory it clearly statest that code is intended for public re-use and if something is located at different, it does not. Note, I emphasized word public because if some projects are share code partially internally this not directly means that code is also meant/intended to public reuse. So, there could be a gray zone and pkg directory makes things more clear. If something is in this directory, use it freely. If not, expect some consequences (not neccessarily legal).

[theckman]

@hron84 In Go you can put it at the top level of a package so all things inside of that repository can use it. If you are looking for code reuse across repos, that means you expect it to not be internal and consumed externally. If you consider that, I don't think my statement is a stretch.

[rakyll]

pkg is not a canonical practice to organize "importable" packages. One side effect of representing it here is that I hear a lot from others that I should reorg my packages as suggested as here.

it clearly statest that code is intended for public re-use

I don't think there is a widely accepted rule for this. This is the first time I am hearing this intention and I personally know very few people who thinks "pkg" makes a package intended for public use. Any importable package is for public use. Anything else goes into internal. Note that internal package is not just a convention, it is supported by the Go tool.

[egorse]

/cc @peterbourgon @thockin

[peterbourgon]

The pkg subdirectory is a way to sequester Go source files into well-known locations in a repository. That is, all Go files are guaranteed to live in ./{cmd,pkg,internal}/*, with cmd being for all package main, pkg for reusable packages, and internal for non-reusable packages. This property is useful on its own, but it is especially useful in repositories that have lots of other types of files, for example build and hack scripts, config files, static assets, or source code written in other languages.

This value proposition seems clear and unambiguous to me. I don't understand why some Go programmers are so vocal about squashing it. I don't think it is mandatory, and I wouldn't suggest advertising it that way, but it's clearly useful and I would push back against (or at least ignore) anyone who suggested it not be used.

[theckman]

@peterbourgon It's not clear and unambiguous to me, because it's honestly not clear what value it adds beyond cognitive overhead and unnecessary abstractions. To me it's just noise. I don't really care what directories my Go source code is in. What makes you care?

My editor has jump to definition, and I have other tooling I can use to discover files based on their declared types. I also have an easy way to open any file in my current project using a simple keyboard shortcut. It's been superseded by better tooling, functionality, editor integrations, and assumptions by the Go toolkit, and we should retire it as a relic of times past

I also have issue with this quote in the README:

pkg/: Library code that's ok to use by external applications

No, anything that's NOT in internal is ok to use by external applications.

[peterbourgon]

I don't really care what directories my Go source code is in. What makes you care?

If I'm looking for Go code, I know where to look. Equally importantly, I know where not to look: is hack/ a poorly-named package, or a collection of scripts? With pkg I know for sure. Even more concretely, when I run commands that target Go source files or source trees, I can point them in the right place, without paying the costs of tree walking or stat calling on my (potentially very large) tree.

For example: go test ./{cmd,pkg,internal}/... versus go test ./... can be a huge difference in wall-clock time on repos with lots of non-Go-source files.

[pkg has] been superseded by better tooling, functionality, editor integrations, and assumptions by the Go toolkit, and we should retire it as a relic of times past.

You are affirming the consequent.

[theckman]

Hmm, that makes me less convinced. If I'm looking for Go code I generally have an entry point. A function, a file, some string to search for, etc. I use my editor's tools to do the work beyond that. It seems like a pretty inefficient workflow to just go spelunking in repos looking for Go files without any other way to drill-down to what you want to find.

If someone wants to keep all the go code together, I'm 👍 on a top-level go directory. I'm 👎 on a list of arbitrary locations which may or may not have Go code, and which may or may not also indicate the public nature of those packages.