Why Do CI Pipelines Fail For S390x Builds?

I've spent a lot of late nights tinkering with odd architectures, and the short story is: if you want true s390x (IBM Z / LinuxONE) hardware in the cloud, IBM is the real, production-ready option. IBM Cloud exposes LinuxONE and z Systems resources—both bare-metal and virtualized offerings that run on s390x silicon. There's also the 'LinuxONE Community Cloud', which is great if you're experimenting or teaching, because it gives developers time on real mainframe hardware without the full enterprise procurement dance. Outside of IBM's own public cloud, you'll find a handful of specialized managed service providers and system integrators (think the folks who historically supported mainframes) who will host s390x guests or provide z/VM access on dedicated hardware. Names change thanks to mergers and spinoffs, but searching for managed LinuxONE or z/VM hosting usually surfaces options like Kyndryl partners or regional IBM partners who do rent time on mainframe systems. If you don't strictly need physical s390x hardware, a practical alternative is emulation: you can run s390x under QEMU on ordinary x86 VMs from AWS, GCP, or Azure for development and CI. It’s slower but surprisingly workable for builds and tests, and a lot of open-source projects publish multi-arch s390x images on Docker Hub. So for production-grade s390x VMs, go IBM Cloud or a mainframe hosting partner; for dev, consider 'LinuxONE Community Cloud' or QEMU emulation on common clouds.

Building Go for s390x is way easier than I used to expect — once you know the tiny set of knobs to flip. I’ve cross-compiled a couple of small services for IBM Z boxes and the trickiest part was simply remembering to disable cgo unless I had a proper cross-GCC toolchain. Practically, for a pure Go program the canonical command I use is very simple: set GOOS=linux and GOARCH=s390x, and turn off CGO so the build doesn’t try to invoke a C compiler. For example: GOOS=linux GOARCH=s390x CGO_ENABLED=0 go build -o myprog_s390x ./... That produces an s390x ELF binary you can check with file myprog_s390x. If you need smaller binaries I usually add ldflags like -ldflags='-s -w'. If your project uses cgo (native libs), you’ll need a cross-compiler for s390x and to set CC appropriately (e.g. CC=s390x-linux-gnu-gcc), but the package names and toolchain installation vary by distro. When I couldn’t access hardware I tested with qemu (qemu-system-s390x for full systems, or register qemu-user in binfmt_misc) to sanity-check startup. I also sometimes use Docker buildx or CI (GitHub Actions) to cross-build images, but for pure Go binaries the env-variable approach is the fastest way to get a working s390x binary on an x86 machine. If you run into weird syscalls or platform-specific bugs, running the binary on a real s390x VM or CI runner usually tells you what to fix.

I’m often torn between geeky delight and pragmatic analysis when comparing s390x to x86_64, and honestly the differences read like two different design philosophies trying to solve the same problems. On paper, s390x (the IBM Z 64-bit architecture) is built for massive, predictable throughput, top-tier reliability, and hardware-assisted services: think built-in crypto, compression, and I/O plumbing that shine in transaction-heavy environments. That pays off in real-world workloads like large-scale OLTP, mainframe-hosted JVM applications, and legacy enterprise stacks where consistent latency, hardware offloads (zIIP-like processors), and crazy dense virtualization are the priorities. Benchmarks you hear about often favor s390x for throughput-per-chassis and for workloads that leverage those special features and the mainframe’s I/O subsystem; it’s also built to keep the lights on with near-zero interruptions, which changes how you measure “performance” compared to raw speed. By contrast, x86_64 CPUs from Intel and AMD are the everyman champions: higher clock speeds, aggressive single-thread boosts, and a monstrous software ecosystem tuned for them. For single-threaded tasks, developer tooling, desktop-like responsiveness, and the vast majority of open-source binaries, x86_64 usually feels faster and is far easier to optimize for. The compilers, libraries, and prebuilt packages are more mature and more frequently tuned for these chips, which translates to better out-of-the-box performance for many workloads. If you’re running microservices, cloud-native stacks, or latency-insensitive batch jobs, x86_64 gives you flexibility, cheaper entry costs, and a huge talent pool. Power efficiency per core and raw FLOPS at consumer prices also often lean in x86_64’s favor, especially at smaller scales. When I’m actually tuning systems, I think about practical trade-offs: if I need predictable 24/7 transaction processing with hardware crypto and great virtualization density, I’ll favor s390x; if I need rapid scaling, a broad toolchain, and cheap instances, x86_64 wins. Porting code to s390x means paying attention to endianness, recompiling with architecture flags, and sometimes rethinking assumptions about atomic operations or third-party binaries. On the flip side, s390x’s specialty engines and massive memory bandwidth can make it surprisingly efficient per transaction, even if its per-thread peak may not match the highest-clocked x86 cores. Honestly, the best choice often comes down to workload characteristics, ecosystem needs, and cost model — not a simple “better-or-worse” verdict — so I tend to prototype both where possible and measure real transactions rather than relying on synthetic numbers. I’ve had projects where a JVM app moved to s390x and suddenly cryptographic-heavy endpoints got cheaper and faster thanks to on-chip crypto, and I’ve also seen microservice farms on x86_64 scale out at way lower upfront cost. If you’re curious, try running your critical path on each architecture in a constrained test and look at latency distributions, throughput under contention, and operational overhead — that’s where the truth lives.

Honestly, if you're digging into s390x support today, the landscape is surprisingly tidy compared to other niche architectures. In plain terms: the big mainstream distributions offer official support, because IBM Z and LinuxONE are widely used in enterprise settings. The names you should know: Debian (official s390x port with regular images and repos), Fedora (s390x is an official Fedora architecture with regular composes), openSUSE/Leap and Tumbleweed (plus SUSE Linux Enterprise which is the commercial offering) and Red Hat Enterprise Linux (RHEL) all provide official builds for s390x. Canonical also ships Ubuntu images for IBM Z (s390x) for supported releases. Gentoo has maintained s390x support too, though its workflow is source-based rather than binary-focused. These are the ones you can reasonably point to as officially supported by their projects or vendors. Beyond that, some distributions provide community or experimental s390x images — Alpine and certain RHEL rebuilds or downstreams may have builds contributed by their communities, and projects like Rocky or AlmaLinux occasionally have community efforts, but their s390x coverage is more hit-or-miss and varies by release. If you need production stability, stick with Debian, Fedora, SUSE/SLES, Ubuntu, RHEL, or Gentoo depending on your preferred model (binary vs source). For getting started, look for images labeled 's390x' on each distro's download or cloud image pages, and check release notes for kernel and z/VM compatibility. I'm always tickled by how resilient these platforms are on mainframe iron — it's a different vibe from desktop Linux, but super solid if you need uptime.

I've been using QEMU for s390x work for years, and honestly, for most development tasks it's impressively dependable. For bringing up kernels, testing initramfs changes, and iterating on system services, QEMU will save you endless time: fast cycles with snapshots, easy serial logs, and straightforward debugging with gdb. The system emulation supports the common channel-attached (CCW) devices and block/network backends well enough to boot mainstream distributions, run systemd, and validate functionality without needing iron in the room. That said, reliability depends on what you mean by "reliable." If you need functional correctness—does the kernel boot, do filesystems mount, do userspace services run—QEMU is solid. If you need hardware-accurate behavior, cycle-exact timing, or access to specialized on-chip accelerators (cryptographic units, proprietary telemetry, or mainframe-specific features), QEMU's TCG emulation will fall short. KVM on real IBM Z hardware is the path for performance parity and hardware feature exposure, but of course that requires access to real machines. My usual workflow is to iterate fast in QEMU, use lightweight reproducible images, write tests that run in that environment, then smoke-test on actual hardware before merging big changes. For everyday development it's a huge productivity boost, but I always treat the emulator as the first step, not the final authority.

When I dive into s390x support, I tend to look at two things: how mature a feature is in upstream mainline, and what enterprise distributions have backported. Historically, s390x has been part of the kernel for a long time (the s390/s390x tree matured through the 2.6 and 3.x eras), but the real message is that modern LTS kernels are where you'll find the best, most polished support for contemporary mainframe features. If you want concrete guidance: pick a modern long-term-stable kernel — think 5.10, 5.15, or 6.1 — or newer 6.x kernels if you need bleeding-edge fixes. Those LTS lines collect important fixes for KVM on s390x, DASD/CCW improvements, zfcp (Fibre Channel) robustness, zcrypt and CPACF crypto support, and paravirtual I/O enhancements. Enterprise distros (RHEL, SLES, Ubuntu LTS) often backport features into their kernel trees, so a distribution-provided LTS kernel can be the safest route for production while still giving you modern hardware support. Practically, if I’m deploying to a z15/z16 or running heavy KVM workloads, I’ll test on the latest upstream stable or a 6.x kernel to catch recently merged performance and crypto improvements, then switch to the distribution LTS that includes those backports for production. Also check kernel config options (look for s390, CCW, DASD, zcrypt-related flags) and read the s390-specific changelogs in the kernel git to verify feature flags you rely on.

Whenever I need s390x images I treat Docker Hub like a little scavenger hunt — it’s oddly satisfying when you find exactly the manifest you need. I’ll usually start at hub.docker.com, search for the image name (for example 'ubuntu', 'alpine', or whatever project you care about) and open the Tags view. Click a tag and look for the 'Supported architectures' section: if the repository publishes a manifest list, Docker Hub will show whether 's390x' (aka IBM Z) is included. That visual check saves a lot of time before attempting a pull. If I want to be 100% sure from the command line I run a few quick checks: docker pull --platform=linux/s390x IMAGE:TAG to try pulling the s390x variant (Docker will error if it doesn’t exist), or docker manifest inspect IMAGE:TAG | jq '.' to inspect the manifest list and see which platforms are present. For more advanced work I use docker buildx imagetools inspect IMAGE:TAG or skopeo inspect docker://IMAGE:TAG — those return the manifest and platform info reliably. If an image doesn’t include s390x you’ll either need to find a different image, look for a vendor that publishes s390x builds, or build one yourself with buildx and qemu emulation. A few practical tips from my experiments: official images like 'ubuntu', 'debian', 'alpine' and many OpenJDK variants frequently include s390x builds, but not every tag/version will. Some community or vendor images explicitly add a '-s390x' suffix in their tag names, though relying on manifest lists is safer. If you’re running on non‑Z hardware and testing, remember to enable qemu (multiarch/qemu-user-static) or use a CI with actual s390x runners. Happy hunting — once you get the hang of manifest inspection it becomes second nature and saves many wasted pulls.

Okay, let's get hands-on — I love digging into this kind of system-level tuning. Running PostgreSQL on s390x (IBM Z) gives you a beast of a platform if you respect a few hardware and kernel quirks, so I usually start by getting a solid baseline: capture CPU, memory, IO, and PostgreSQL stats during representative workloads (iostat, sar, vmstat, pg_stat_activity, pg_stat_statements). Knowing whether your I/O is zFCP-backed storage, NVMe, or something virtualized under z/VM makes a huge difference to what follows. For PostgreSQL parameters, I lean on a few rules that work well on large-memory s390x hosts: set shared_buffers to a conservative chunk (I often start around 25% of RAM and iterate), effective_cache_size to 50–75% depending on how much the OS will cache, and tune work_mem per-connection carefully to avoid memory explosions. Increase maintenance_work_mem for faster VACUUM/CREATE INDEX operations, and push max_wal_size up to reduce checkpoint storms — paired with checkpoint_completion_target around 0.7 to smooth writes. Autovacuum needs love here: lower autovacuum_vacuum_scale_factor and raise autovacuum_max_workers if you have many DBs and heavy churn. On the kernel and storage side, check THP and either disable Transparent Huge Pages or move to explicit hugepages depending on your latency profile — THP can introduce pauses. Adjust vm.swappiness (10 or lower), vm.dirty_background_ratio/dirty_ratio and vm.dirty_expire_centisecs to tune writeback behavior. Use a modern I/O scheduler appropriate for your device (noop or mq-deadline for SSDs, test with fio). Mount data volumes with noatime and consider XFS for large DBs. If you control the build, enabling architecture-optimized compiler flags for s390x can help, and watch out for endianness when using custom binary formats or extensions. Finally, add connection pooling (pgbouncer), replicate with streaming replication for read-scaling, and automate monitoring and alerting — once you have metrics, incremental tuning becomes much less scary.