Goapp v1.0

This is an opinionated guideline to structure a Go web application/service (could be extended for any type of application). My opinions were formed over a span of 9+ years building web applications/services with Go, trying to implement DDD (Domain Driven Development) & Clean Architecture. This guideline works for 1.4+ (i.e. since introduction of the special 'internal' directory).

P.S: This guideline is not directly applicable for an independent package, as their primary use of such a package is to be consumed in other applications. In such cases, having most or all of the package code in the root is probably the best way of doing it.

The structure is explained based on a note taking web application (with hardly any features implemented 🤭).

Table of contents

1. Directory structure
2. Configs package
3. API package
4. Users (would be common for all such business logic / domain units, 'usernotes' being similar to users) package.
5. Testing
6. pkg package
   • 6.1. datastore
   • 6.2. logger
7. HTTP server
   • 7.1. templates
8. lib
9. vendor
10. docker
11. schemas
12. main.go
13. Error handling
14. Dependency flow
15. Integrating with ELK APM
16. Note

Directory structure

├── cmd
│   ├── server
│   │   ├── grpc
│   │   │   └── grpc.go
│   │   └── http
│   │       ├── handlers.go
│   │       ├── handlers_usernotes.go
│   │       ├── handlers_users.go
│   │       ├── http.go
│   │       └── web
│   │           └── templates
│   │               └── index.html
│   └── subscribers
│       └── kafka
│           └── kafka.go
├── docker
│   ├── docker-compose.yml
│   └── Dockerfile
├── go.mod
├── go.sum
├── internal
│   ├── api
│   │   ├── api.go
│   │   ├── usernotes.go
│   │   └── users.go
│   ├── configs
│   │   └── configs.go
│   ├── pkg
│   │   ├── apm
│   │   │   ├── apm.go
│   │   │   ├── grpc.go
│   │   │   ├── http.go
│   │   │   ├── meter.go
│   │   │   ├── prometheus.go
│   │   │   └── tracer.go
│   │   ├── logger
│   │   │   ├── default.go
│   │   │   └── logger.go
│   │   ├── postgres
│   │   │   └── postgres.go
│   │   └── sysignals
│   │       └── sysignals.go
│   ├── usernotes
│   │   ├── store_postgres.go
│   │   └── usernotes.go
│   └── users
│       ├── store_postgres.go
│       └── users.go
├── lib
│   └── goapp
│       ├── goapp.go
│       ├── go.mod
│       └── go.sum
├── LICENSE
├── main.go
├── inits.go
├── shutdown.go
├── README.md
└── schemas
    ├── functions.sql
    ├── user_notes.sql
    └── users.sql

internal

"internal" is a special directory name in Go, wherein any exported name/entity can only be consumed within its immediate parent or any other packages within internal directory.

internal/configs

Creating a dedicated configs package might seem like an overkill, but it makes things easier. In the app, you see the HTTP configs are hardcoded and returned. Later you decide to change to consume from env variables. All you do is update the configs package. And further down the line, maybe you decide to introduce something like etcd, then you define the dependency in Configs and update the functions accordingly. This is yet another separation of concern package, to try and keep main tidy.

internal/api

The API package is supposed to have all the APIs exposed by the application. A dedicated API package is created to standardize the functionality, when there are different kinds of services running. e.g. an HTTP & a gRPC server, a Kafka & Pubsub subscriber etc. In such cases, the respective "handler" functions would inturn call api.<Method name>. This gives a guarantee that all your APIs behave exactly the same without any accidental inconsistencies across different I/O methods. It also helps consolidate which functionalities are expcted to be exposed outside of the application via API. There could be a variety of exported functions in the domain packages, which are not meant to communicate with anything outside the application rather to be used among other domain packages.

But remember, middleware handling is still at the internal/server layer. e.g. access log, authentication etc. Even though this can be brought to the api package, it doesn't make much sense because middleware are mostly dependent on the server/handler implementation. e.g. HTTP method, path etc.

internal/users

Users package is where all your actual user related business logic is implemented. e.g. Create a user after cleaning up the input, validation, and then store it inside a persistent datastore.

The store_postgres.go in this package is where you write all the direct interactions with the datastore. There's an interface which is unique to the users package. It is used to handle dependency injection as well as dependency inversion elegantly. The file naming convention I follow is to have the word store in the beggining, suffixed with _<db name>. Though I think it's also ok name it based on a logical group, e.g. store_registration, store_login etc. Especially when there's a lot of database/storage related to code to be crammed into a single file.

NewService/New function is created in each package, which initializes and returns the respective package's feature implementor. In case of users package, it's the Users struct. The name 'NewService' makes sense in most cases, and just reduces the burden of thinking of a good name for such scenarios. The Users struct here holds all the dependencies required for implementing features provided by users package.

internal/users_test

There's quite a lot of discussions about achieveing and maintaining 100% test coverage or not. 100% coverage sounds very nice, but might not always be practical or at times not even possible. What I like doing is, writing unit test for your core business logic, in this case 'Sanitize', 'Validate' etc are my business logic.

It is important for us to understand the purpose of unit tests. The sole purpose of unit test is unironically "test the purpose of the unit/function". It is not to check the implementation, how it's done, how much time it took, how efficient it is etc. The sole purpose is to validate "what it does". This is why you see a lot of unit tests will have hardcoded values, because those are reliable/verified human input which we validate against.

Once you develop the habit of writing unit tests for pure functions and get the hang of it. You automatically start breaking down big functions into smaller testable functions/units (this is the best outcome, and what we'd love to have). When you layer your application, the datastore is ideally just a utility (implementation detail in Clean Architecture parlance), and if you can implement your business logic with pure functions alone, not dependent on such utlities, that'd be perfect! Though in most cases you'd have dependencies like database, queue, cache etc. But to keep things as pure as possible, we bridge the gap using Go interfaces. Refer to store.go, the domain package functions are oblivious to the underlying technology (RDBMS, NoSQL, CSV etc.).

Always writing the entire business logic within the app is not necessary or sometimes extremely difficult, rather make use of features provided by databases and other tools. e.g. Database can do joins, sort etc. Though when using such features, it's best that the function signature hints at this. e.g. GetUserNotes(ctx, userID) []Note is a name which hints at the joining of User and Note. This way, if we decide to switch database which does not support join, we still know the expected behaviour from the data store function.

integration tests

In case of writing integration tests, i.e. when you make API calls from outside the app to test functionality, I prefer using actual running instances of dependencies instead of mocks. Especially in case of databases, or any such easy to use dependency. Though if the dependency is an external service's APIs, mocks are probably the best available option.

internal/usernotes

Similar to the users package, 'usernotes' handles all business logic related to user's notes.

internal/pkg

pkg directory contains all the packages which are to be consumed across multiple packages within the project. For instance the postgres package will be consumed by both users and usernotes package. It's the destination for utility packages, and nothing to do with the features of your main application.

internal/pkg/postgres

The postgres package initializes pgxpool.Pool and returns a new instance. Though a seemingly redundant package only for initialization, it's useful to do all the default configuration which we want standardized across the application. An example is to wrap the driver, or functions for APM. The screenshots below show how APM can help us monitor our application.

IMPORTANT: the screenshots are severely outdated, though the point stands that you get very indepth details of how your application is performing.

internal/pkg/logger

I usually define the logging interface as well as the package, in a private repository (internal to your company e.g. vcs.yourcompany.io/gopkgs/logger), and is used across all services. Logging interface helps you to easily switch between different logging libraries, as all your apps would be using the interface you defined (interface segregation principle from SOLID). Though here I'm making it part of the application itself as it has fewer chances of going wrong when trying to cater to a larger audience.

Logging might sound trivial but there are a few questions around it:

1. Should it be made a dependency of all packages, or can it be global?

Logging just like any other dependency, is a dependency. And in most cases it's better to write packages (code in general) which have as few dependencies as practically possible. This is a general principle, fewer dependencies make a lot of things easier like maintainability, testing, porting, refactoring, etc. Creating singleton globals bring in restrictions, also it's a dependency nevertheless. Global instances have another issue, it doesn't give you flexibility when you need varyi