Skip to content

Building in GitLab

To learn how to use containers and Docker on your local machine, refer to our tutorial section.

We use our own installation of GitLab for Source Code Management, Continuous Integration automation, containers registry and other development lifecycle tasks. It fully uses Nautilus Cluster resources, which provides our users unlimited storage and fast builds. All data from our GitLab except container images are backed up nightly to Google storage, which means there's almost zero chance that you might lose your code in our repository.

Step 1: Create a Git repo

  1. To use our GItLab installation, register at https://gitlab.nrp-nautilus.io
  2. Use GitLab for storing your code like any git repository. Here's GitLab basics guide.
  3. Create a new project in your GitLab account

Step 2: Use Containers Registry

What makes GitLab especially useful for kubernetes cluster in integration with Containers Registry. You can store your containers directly in our cluster and avoid slow downloads from DockerHub (although you're still free to do that as well).

If you wish to use our registry, in your https://gitlab.nrp-nautilus.io project go to Packages & Registries -> Container Registry menu and read instructions on how to use one.

Step 3: Continuous Integration automation

To fully unleash the GitLab powers, introduce yourself to Continuous Integration automation

  1. Create the .gitlab-ci.yml file in your project, see Quick start guide. The runners are already configured.
    There's a list of CI templates available for most common languages.
  2. If you need to build your Dockerfile and create a container from it, adjust this .gitlab-ci.yml template:
image: gcr.io/kaniko-project/executor:debug

stages:
  - build-and-push

build-and-push-job:
  stage: build-and-push
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --cache=true --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:8} --destination $CI_REGISTRY_IMAGE:latest

More advanced example

  1. Go to CI / CD -> Jobs tab to see in amazement your job running and image being uploaded to your registry.
  2. From the Packages -> Containers Registry tab get the URL of your image to be included in your pod definition:
spec:
  containers:
  - name: my-container
    image: gitlab-registry.nrp-nautilus.io/<your_group>/<your_project>:<optional_tag>

Using docker instead of kaniko

While kaniko container builder is the easy to use userspace builder, it fails to build some complex images, like the CUDA ones provided by NVIDIA. If you believe you need the original docker to build your image, you can use the provided sysbox Docker-In-Docker runner by adding the sysbox tag to your CI job:

image: docker:git

default:
  tags:
    - sysbox

services:
  - name: docker:dind
variables:
  DOCKER_HOST: tcp://docker:2376/
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_TLS_VERIFY: 1
  DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"

build-jupyter-base:
  before_script:
    - until docker info; do sleep 1; done
    - docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab-registry.nrp-nautilus.io
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest

Build better containers

Make yourself familiar with Docker containers best practices

Use multi-stage builds when necessary

Use S3 to store large files collections and access those during builds

Refer to S3 documentation

Other development information

Check out this guide from the Netherlands eScience Center for best practices in developing academic code.