Building in GitLab
To learn how to use containers and Docker on your local machine, refer to our tutorial section.
We use our own installation of GitLab for Source Code Management, Continuous Integration automation, containers registry and other development lifecycle tasks. It fully uses Nautilus Cluster resources, which provides our users unlimited storage and fast builds. All data from our GitLab except container images are backed up nightly to Google storage, which means there's almost zero chance that you might lose your code in our repository.
Step 1: Create a Git repo
- To use our GItLab installation, register at https://gitlab.nrp-nautilus.io
- Use GitLab for storing your code like any git repository. Here's GitLab basics guide.
- Create a new project in your GitLab account
Step 2: Use Containers Registry
What makes GitLab especially useful for kubernetes cluster in integration with Containers Registry. You can store your containers directly in our cluster and avoid slow downloads from DockerHub (although you're still free to do that as well).
If you wish to use our registry, in your https://gitlab.nrp-nautilus.io project go to Packages & Registries -> Container Registry
menu and read instructions on how to use one.
Step 3: Continuous Integration automation
To fully unleash the GitLab powers, introduce yourself to Continuous Integration automation
- Create the
.gitlab-ci.yml
file in your project, see Quick start guide. The runners are already configured.
There's a list of CI templates available for most common languages. - If you need to build your Dockerfile and create a container from it, adjust this
.gitlab-ci.yml
template:
image: gcr.io/kaniko-project/executor:debug
stages:
- build-and-push
build-and-push-job:
stage: build-and-push
script:
- echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
- /kaniko/executor --cache=true --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:${CI_COMMIT_SHA:0:8} --destination $CI_REGISTRY_IMAGE:latest
- Go to
CI / CD -> Jobs
tab to see in amazement your job running and image being uploaded to your registry. - From the
Packages -> Containers Registry
tab get the URL of your image to be included in your pod definition:
spec:
containers:
- name: my-container
image: gitlab-registry.nrp-nautilus.io/<your_group>/<your_project>:<optional_tag>
Using docker instead of kaniko
While kaniko container builder is the easy to use userspace builder, it fails to build some complex images, like the CUDA ones provided by NVIDIA. If you believe you need the original docker to build your image, you can use the provided sysbox Docker-In-Docker runner by adding the sysbox tag to your CI job:
image: docker:git
default:
tags:
- sysbox
services:
- name: docker:dind
variables:
DOCKER_HOST: tcp://docker:2376/
DOCKER_TLS_CERTDIR: "/certs"
DOCKER_TLS_VERIFY: 1
DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
build-jupyter-base:
before_script:
- until docker info; do sleep 1; done
- docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN gitlab-registry.nrp-nautilus.io
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
- docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA $CI_REGISTRY_IMAGE:latest
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
- docker push $CI_REGISTRY_IMAGE:latest
Build better containers
Make yourself familiar with Docker containers best practices
Use multi-stage builds when necessary
Use S3 to store large files collections and access those during builds
Refer to S3 documentation
Other development information
Check out this guide from the Netherlands eScience Center for best practices in developing academic code.