Docker Buildx Cross Compiling Python to Armhf: "Fatal Python Error: Pyinit_main: Can't Initialize Time"

The symptoms

When running

uname -a
docker version

Linux konixwork 5.8.0-2-amd64 #1 SMP Debian 5.8.10-1 (2020-09-19) x86_64 GNU/Linux
Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:55 2020
 OS/Arch:           linux/amd64
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:01:25 2020
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

ssh blueberry uname -a && ssh blueberry docker version

Linux blueberry 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:37:22 2019
 OS/Arch:           linux/arm
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:31:17 2019
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

The docker file is

FROM alpine
RUN apk add --update python3
CMD ["python3", "-c", "print('ok')"]

Building and running this on my desktop

docker buildx build ~/tmp/docker -t localhost:9692/testalpine --push 2>&1

time="2021-01-22T13:41:48+01:00" level=warning msg="invalid non-bool value for BUILDX_NO_DEFAULT_LOAD: "
#1 [internal] load build definition from Dockerfile
#1 DONE 0.0s

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 31B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 0.5s

#4 [1/2] FROM docker.io/library/alpine@sha256:d9a7354e3845ea8466bb00b22224d...
#4 resolve docker.io/library/alpine@sha256:d9a7354e3845ea8466bb00b22224d9116b183e594527fb5b6c3d30bc01a20378 0.0s done
#4 DONE 0.0s

#5 [2/2] RUN apk add --update python3
#5 CACHED

#6 exporting to image
#6 exporting layers done
#6 exporting manifest sha256:f65439ea3152cecd80544dc064a638f137addb7304e5aaa98e4d9543214121ac 0.0s done
#6 exporting config sha256:4260c9e0fe4cad647f64e860bed99d248fc7b637dd7ee07df695575b5224a269 0.0s done
#6 pushing layers 0.0s done
#6 pushing manifest for localhost:9692/testalpine:latest done
#6 DONE 0.1s

docker run --rm localhost:9692/testalpine

ok

When running it on the raspberry pi 3B+

ssh blueberry mkdir -p docker && \
scp ~/tmp/docker/Dockerfile blueberry:docker/ && \
ssh blueberry ls docker

Dockerfile

ssh blueberry docker build -t test ./docker

Sending build context to Docker daemon  2.048kB
Step 1/3 : FROM alpine
 ---> 7e4bece93b3e
Step 2/3 : RUN apk add --update python3
 ---> Using cache
 ---> 9d8dcef4c292
Step 3/3 : CMD ["python3", "-c", "print('ok')"]
 ---> Running in 9c79e94c140b
Removing intermediate container 9c79e94c140b
 ---> 39c5199bceff
Successfully built 39c5199bceff
Successfully tagged test:latest

ssh blueberry docker run --rm test

ok

Then, now let’s try to cross compile the image.

First, setup the qemu layer.

docker run --rm --privileged fkrull/qemu-user-static enable && \
    docker buildx create \
                   --name builder \
                   --driver-opt network=host && \
    docker buildx use builder && \
    docker buildx inspect --bootstrap

enabling qemu-aarch64 ... skipping
enabling qemu-alpha ... skipping
enabling qemu-arm ... skipping
enabling qemu-armeb ... skipping
enabling qemu-cris ... skipping
enabling qemu-m68k ... skipping
enabling qemu-microblaze ... skipping
enabling qemu-mips ... skipping
enabling qemu-mips64 ... skipping
enabling qemu-mips64el ... skipping
enabling qemu-mipsel ... skipping
enabling qemu-ppc ... skipping
enabling qemu-ppc64 ... skipping
enabling qemu-ppc64abi32 ... skipping
enabling qemu-ppc64le ... skipping
enabling qemu-s390x ... skipping
enabling qemu-sh4 ... skipping
enabling qemu-sh4eb ... skipping
enabling qemu-sparc ... skipping
enabling qemu-sparc32plus ... skipping
enabling qemu-sparc64 ... skipping
builder
Name:   builder
Driver: docker-container

Nodes:
Name:      builder0
Endpoint:  unix:///var/run/docker.sock
Status:    running
Platforms: linux/amd64, linux/arm64, linux/ppc64le, linux/s390x, linux/386, linux/arm/v7, linux/arm/v6

Ok, let’s cross compile it for armv7

docker buildx build ~/tmp/docker --platform linux/arm/v7 -t localhost:9692/testalpine --push 2>&1

time="2021-01-22T13:47:42+01:00" level=warning msg="invalid non-bool value for BUILDX_NO_DEFAULT_LOAD: "
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 115B done
#1 DONE 0.2s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.2s

#3 [internal] load metadata for docker.io/library/alpine:latest
#3 DONE 2.6s

#4 [1/2] FROM docker.io/library/alpine@sha256:d9a7354e3845ea8466bb00b22224d...
#4 resolve docker.io/library/alpine@sha256:d9a7354e3845ea8466bb00b22224d9116b183e594527fb5b6c3d30bc01a20378 0.0s done
#4 DONE 0.1s

#4 [1/2] FROM docker.io/library/alpine@sha256:d9a7354e3845ea8466bb00b22224d...
#4 sha256:0f34ce5da94097b8c334f6b2065a010aced9855c3532e4462e9bd1b0a4c4b3f6 0B / 2.42MB 0.2s
#4 sha256:0f34ce5da94097b8c334f6b2065a010aced9855c3532e4462e9bd1b0a4c4b3f6 1.05MB / 2.42MB 0.3s
#4 sha256:0f34ce5da94097b8c334f6b2065a010aced9855c3532e4462e9bd1b0a4c4b3f6 2.42MB / 2.42MB 0.4s done
#4 extracting sha256:0f34ce5da94097b8c334f6b2065a010aced9855c3532e4462e9bd1b0a4c4b3f6 0.1s done
#4 DONE 0.5s

#5 [2/2] RUN apk add --update python3
#5 0.114 qemu: Unsupported syscall: 397
#5 0.114 qemu: Unsupported syscall: 397
#5 0.114 qemu: Unsupported syscall: 397
#5 0.114 qemu: Unsupported syscall: 397
#5 0.114 qemu: Unsupported syscall: 397
#5 0.119 qemu: Unsupported syscall: 397
#5 0.119 qemu: Unsupported syscall: 397
#5 0.120 qemu: Unsupported syscall: 397
#5 0.120 qemu: Unsupported syscall: 397
#5 0.134 qemu: Unsupported syscall: 397
#5 0.134 qemu: Unsupported syscall: 397
#5 0.135 qemu: Unsupported syscall: 397
#5 0.140 qemu: Unsupported syscall: 397
#5 0.141 qemu: Unsupported syscall: 403
#5 0.142 qemu: Unsupported syscall: 397
#5 0.142 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/armv7/APKINDEX.tar.gz
#5 0.144 qemu: Unsupported syscall: 403
[repeated 42 times]
#5 1.019 qemu: Unsupported syscall: 397
#5 1.019 fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/armv7/APKINDEX.tar.gz
#5 1.859 (1/11) Installing libbz2 (1.0.8-r1)
#5 1.884 qemu: Unsupported syscall: 397
#5 1.884 qemu: Unsupported syscall: 397
#5 1.887 qemu: Unsupported syscall: 397
#5 1.887 qemu: Unsupported syscall: 397
#5 1.887 (2/11) Installing libgcc (10.2.1_pre1-r3)
#5 1.912 qemu: Unsupported syscall: 397
#5 1.912 (3/11) Installing expat (2.2.10-r1)
#5 1.932 qemu: Unsupported syscall: 397
#5 1.936 qemu: Unsupported syscall: 397
#5 1.936 qemu: Unsupported syscall: 397
#5 1.936 qemu: Unsupported syscall: 397
#5 1.936 (4/11) Installing libffi (3.3-r2)
#5 1.980 qemu: Unsupported syscall: 397
#5 1.980 qemu: Unsupported syscall: 397
#5 1.980 (5/11) Installing gdbm (1.19-r0)
#5 2.006 qemu: Unsupported syscall: 397
#5 2.006 qemu: Unsupported syscall: 397
#5 2.006 qemu: Unsupported syscall: 397
#5 2.006 qemu: Unsupported syscall: 397
#5 2.006 qemu: Unsupported syscall: 397
#5 2.006 qemu: Unsupported syscall: 397
#5 2.006 qemu: Unsupported syscall: 397
#5 2.007 (6/11) Installing xz-libs (5.2.5-r0)
#5 2.035 qemu: Unsupported syscall: 397
#5 2.036 qemu: Unsupported syscall: 397
#5 2.036 (7/11) Installing ncurses-terminfo-base (6.2_p20210109-r0)
#5 2.055 qemu: Unsupported syscall: 397
[repeated 54 times]
#5 2.061 (8/11) Installing ncurses-libs (6.2_p20210109-r0)
#5 2.099 qemu: Unsupported syscall: 397
#5 2.099 qemu: Unsupported syscall: 397
#5 2.099 qemu: Unsupported syscall: 397
#5 2.099 qemu: Unsupported syscall: 397
#5 2.099 qemu: Unsupported syscall: 397
#5 2.099 qemu: Unsupported syscall: 397
#5 2.099 qemu: Unsupported syscall: 397
#5 2.099 qemu: Unsupported syscall: 397
#5 2.099 (9/11) Installing readline (8.1.0-r0)
#5 2.131 qemu: Unsupported syscall: 397
#5 2.131 qemu: Unsupported syscall: 397
#5 2.131 qemu: Unsupported syscall: 397
#5 2.131 (10/11) Installing sqlite-libs (3.34.1-r0)
#5 2.199 qemu: Unsupported syscall: 397
#5 2.199 qemu: Unsupported syscall: 397
#5 2.199 (11/11) Installing python3 (3.8.7-r0)
#5 2.228 qemu: Unsupported syscall: 397
[repeated 2692 times]
#5 3.877 Executing busybox-1.32.1-r0.trigger
#5 3.878 qemu: Unsupported syscall: 397
[repeated 120 times]
#5 3.908 qemu: Unsupported syscall: 403
#5 3.909 OK: 48 MiB in 25 packages
#5 DONE 5.0s

#6 exporting to image
#6 exporting layers
#6 exporting layers 2.0s done
#6 exporting manifest sha256:5ee08abf8fbd46cf84bb5936939c0218c2830ac996dbc3cb7786ae0be6bc4548 0.0s done
#6 exporting config sha256:f4e4e9f213a521e319686083dd1b46883ca87797b945dfc805d3a9cb3d3d2dbd 0.0s done
#6 pushing layers
#6 pushing layers 4.3s done
#6 pushing manifest for localhost:9692/testalpine:latest
#6 pushing manifest for localhost:9692/testalpine:latest 0.3s done
#6 DONE 6.7s

And then run it on the raspberry pi.

ssh blueberry docker run --rm localhost:9692/testalpine

Fatal Python error: pyinit_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0x76f44390 (most recent call first):
<no Python frame>

The image works well, only running python causes this error.

ssh blueberry docker run --rm localhost:9692/testalpine ls

bin
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var

Analysis

We can see that qemu cannot run the following syscalls

Unsupported syscall: 397
Unsupported syscall: 403

If I remind correctly, it already complained in the past but it was still working. According to https://fedora.juszkiewicz.com.pl/syscalls.html, 397 is statx and 403 is clock_gettime64 in arm.

Also, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=951012 appears to say that 403 is related to seccomp.

Looking for this particular error on the web, I get few results.

https://github.com/linuxserver/docker-papermerge/issues/4

You need a updated version of libseccomp2, 2.4.3-1 or higher. It is currently not available for Raspberry PI OS, but it is for Ubuntu. You can look into downloading the Deb from Ubuntus repository.

Interesting, it also talks about seccomp.

Bug#951012: buster-kernel 5.5-armhf-seccomp: syscall 403: I looked into this, this is is new time64 syscall, I’ll be going ahead and whitelist all new time64 syscalls in 1.9.10.

403: clock_gettime64 404: clock_settime64 405: clock_adjtime64 406: clock_getres_time64 407: clock_nanosleep_time64 408: timer_gettime64 409: timer_settime64 410: timerfd_gettime64 411: timerfd_settime64 412: utimensat_time64 413: pselect6_time64 414: ppoll_time64
Bug#951012: buster-kernel 5.5-armhf-seccomp: syscall 403: > Of course, feel free to whitelist them in your apt.conf, by setting > > APT::Sandbox::Seccomp::Allow { “clock_gettime64”; <other syscalls> } > > as I don’t think this will get cherry-picked into stable releases.

It sounds like a good hypothesis, seccomp prevents the syscall clock_gettime64 to run and python in alpine now wants to make use of it. It does not answer the question why it work when built locally though.

May be this has something to do with the fact my host is running linux 5x while the raspberri is running linux 4x.

Seccomp security profiles for Docker | Docker Documentation

You can pass unconfined to run a container without the default seccomp profile.

$ docker run –rm -it –security-opt seccomp=unconfined debian:jessie \ unshare –map-root-user –user sh -c whoami

Let’s try this.

ssh blueberry docker run --security-opt seccomp=unconfined --rm localhost:9692/testalpine 2>&1

ok

Ok, so it looks like docker in the raspberry pi prevented the syscall 403 to run.

Let’s try to use the default docker profile to see if it is indeed the syscall 403.

ssh blueberry wget https://raw.githubusercontent.com/moby/moby/master/profiles/seccomp/default.json && \
    ssh blueberry docker run --security-opt seccomp=default.json --rm localhost:9692/testalpine 2>&1

ok

And we can see in this profile that clock_gettime64 is indeed allowed.

Now, let’s try to remove this syscall and see what happens.

ssh blueberry sed -i '/clock_gettime64/d' default.json && \
    ssh blueberry docker run --security-opt seccomp=default.json --rm localhost:9692/testalpine

Fatal Python error: pyinit_main: can't initialize time
Python runtime state: core initialized
PermissionError: [Errno 1] Operation not permitted

Current thread 0x76fd5390 (most recent call first):
<no Python frame>

We get the error, as expected.

Conclusion

My working hypothesis is now.

If I build the docker image in the raspberry pi

it is using linux 4
then it does not use the syscall clock_gettime64
the image can be run Successfully

If I build the image in my laptop

it is using linux 5
it uses clock_gettime64
the rasperry pi try to use it
it fails because docker does not allow it

It does not tell why the syscall clock_gettime64 that appears to work in raspberry pi is not used in linux prior to 5.

So I guess I can use a –security-opt profile for now and wait for the kernels to be in sync.

Or I can update libseccomp in my raspberry pi as suggested in After Update Docker starts only with privileged: true.

Konubinix' opinionated web of thoughts