Konubinix' opinionated web of thoughts

How to Structure/Organize a Multi Technologies Repo?

Fleeting

I honestly don’t know what is a monorepo. Therefore I will avoid calling that a monorepo, even though I intuitively guess there might be similarities.

Imagine you are working on a project where all components are put in the same repository. For instance, imagine the classical scenario;

  • a frontend, in javascript,
  • a backend, in go,
  • an IAM, like keycloak, with home made plugins written in java,
  • a vault, with home made plugins written in go,

Intuitively, the project layout should look like:

  • frontend
    • package.json
    • …
  • backend
    • go.mod
    • …
  • iam
    • plugins
      • plugin1
        • pom.xml
        • …
  • vault
    • plugin1
      • go.mod
      • …

Chances are that people from different background will contribute to this project. Some javascript guys will likely work in the frontend, while go guys will work in the backend or vault plugins.

Also, it is likely that these component might be related somehow. For instance, the frontend might consume an openapi document generated from the backend. Also, the vault and the backend might have shared libraries.

Organising the repository to deal with shared libraries

When two co-located projects have a need for a shared library, it is common to look for a common ancestor folder to put such dependency.

For example, imagine the following layout.

  • a
    • b1
      • c
        • d
    • b2
      • e

Imagine that d and e share the need for a common library, it is intuitive to put this library in a.

  • a
    • common
    • b1
      • c
        • d -> ../../../common
    • b2
      • e -> ../../common

In case of the organisation per component, the only common ancestor is the repository itself.

Also, each technology comes with its own conventions about how to deal with shared libraries development. I kinda like the one of go, with internal/ pkg/ and vendor/ that makes things quite clear.

So I suggest to put those in /internal or /pkg depending on whether those shared libraries are meant to be published.

The layout would then look like.

  • frontend
    • package.json
    • …
  • backend
    • go.mod -> ../internal
    • …
  • iam
    • plugins
      • plugin1
        • pom.xml
        • …
  • internal
  • vault
    • plugin1
      • go.mod -> ../../internal
      • …

Of course, choosing the way of doing of one technology might make some people angry. An attempt at fixing this would be to organise per technology rather than component.

  • js
    • frontend
      • package.json
      • …
  • go
    • internal
    • backend
      • go.mod -> ../internal
      • …
    • vault
      • plugin1
        • go.mod -> ../../internal
        • …
  • java
    • iam
      • plugins
        • plugin1
          • pom.xml
          • …

This would work, but somehow people (including me) tend to dislike this and prefer the component based organisation.

building the docker images

Another challenge is to decide how to build a docker image of each component, provided that the docker context needs to be in a folder that contain all the dependencies.

In the following example, building a docker image for the backend needs the following parts.

  • frontend
    • package.json
    • …
  • backend
    • go.mod -> ../internal
    • …
  • iam
    • plugins
      • plugin1
        • pom.xml
        • …
  • internal
  • vault
    • plugin1
      • go.mod -> ../../internal
      • …

To keep things kiss and have a similar look and feel in the docker images construction than the repository, I suggest to use the repository as docker context. I tend to like when a Dockerfile is in located in the expected docker context folder, hence I recommend putting the dockerfiles in the root repository, like this :

  • frontend
    • package.json
    • …
  • backend.Dockerfile
  • backend
    • go.mod -> ../internal
    • …
  • iam
    • plugins
      • plugin1
        • pom.xml
        • …
  • internal
  • vault
    • plugin1
      • go.mod -> ../../internal
      • …

Sharing a common docker configuration

It is likely that you will want to have similarities in the construction of the docker images.

To do so, you could either try to build base images and make those images depend on the base images. That would require dancing with the docker registry so as to have base images in your local docker cache as well as in the remote team registry.

I prefer using a tool like DockerMake to build several images using the same template.

Because it allows to print the dockerfile instead of directly building the docker image, you can even use it along with tools using dockerfile like tilt.

I then recommend replacing the dockerfiles by a single DockerMake.yml document at the root.

  • DockerMake.yml
  • frontend
    • package.json
    • …
  • backend
    • go.mod -> ../internal
    • …
  • iam
    • plugins
      • plugin1
        • pom.xml
        • …
  • internal
  • vault
    • plugin1
      • go.mod -> ../../internal
      • …

Organising the artifact transportation

Now, in case you want some artifact from one component to be available in another, I think you can

  1. either make images depend on each other,
  2. or create images with the same dockerfile content (using DockerMake to help factorizing the code),
  3. or build artifacts locally and put them in some registry so that they can be downloaded when creating images
    1. either a shared registry
    2. or directly commit them into the code (using git-lfs)

I tend to prefer the third option, because chances are that you need the artifact in your development anyway. Also, because we tend to move to container based development, chances are that you local setup is exactly the same as the one in the continuous integration, and then you can simply generate the artifact yourself.

Also, because tools like tilt help ensuring that your artifacts remain up to date, you have no excuse to forget to build and commit the artifact.