How's life without Docker?

G

Gab1bb0

Hi!

Lately I'm thinking to get more into BSD systems and one of the first steps I want to make is to migrate my relatively simple self hosting setup (a single machine with few applications running on it) from Ubuntu to BSD.

I was tempted to migrate this machine many times in the last years but something always stopped me: FreeBSD doesn't support Docker. But lately I started to wonder whether Docker in this context is actually helping me or its more something I use just because everyone does that and I read countless articles praising it. I'm no longer sure that Docker is that helpful to me in this context because while Docker makes things easy in some regards it surely complicates stuff in other cases. Sure, having a very repeatable configuration you can just copy-paste to have your service automatically configured is very nice especially at scale. On the other hand when you work with Docker you have to give up significant control over your environment and when things break, doesn't work as expected or you want significant customization (a silly example might be wanting to install mods on your Minecraft server) immutable containers seems to be much more cumbersome to service than jails or simple rc services where you can open a shell, type commands and edit files without worrying about losing these modifications on the next service restart.

So the question I'd like to ask to BSD community is: when you're on BSD do you miss Docker and in its absence what's your favorite way to spin up and manage services on a server?
 
Gab1bb0 said:
So the question I'd like to ask to BSD community is: when you're on BSD do you miss Docker and in its absence what's your favorite way to spin up and manage services on a server?
Click to expand...
Can't miss out on something I never actually used ;)

Of course that's not saying that BSD doesn't support containers; there are Jails, there's Bhyve, etc. But even though I've read about Docker in the past I never felt the urge to look deeper into it. I just don't see the point.
 
When I was learning Linux in Centos 6, all online guides said to host services on Docker. Tried it once and I hated it. Never used it, never had no call to. Life is much better with jails.
 
It seems to me that OP is asking a sincere question, so it should be answered in a serious way, besides of jokes. Unfortunately, I do not have much experience myself, I have only been maintaining a few of WWW publications, a mail server and a GNU repository on FreeBSD for a few years. HTTP requests are handled there by Hiawatha, Textpattern for WWW publishing, Sendmail, SASL and Cyrus for email, Nextcloud for file storage and sharing, media viewers and players, maps, chat, etc. Backed by MySQL, SQLite and Redis databases, with Adminer and Webmin GUI, PHP-FPM… Everything works coherently and robustly, without Docker, Jail, Bhyve. Maybe I should jail Nextcloud, but for now ZFS with BEctl is enough for me.
 
I used docker professionally. I prefer "units" that have a more complete OS installed, mainly for debugging purposes. And as you say, heavy configuration is also not docker's strong point. Service separation via jails works better for me, or complete virtual machines. In general I also tend to put more into a single "unit" so that I don't have to update a million ones when security patches come out.

So between one Unix, complete and properly configured with everything I need for debugging (and profiling) and a bunch of crippled semi-VMs I pick the former.
 
Jails are the way to go. I keep my setups in scripts and key/value config files, so I really don't have to do much other than change a few variables if at all to (re)make them.

I use the default jail.conf method for my jails and they: spin up when the machine boots, shutdown properly when the machine halts, and I really don't think about them. I think of my jails as separate mini servers (each with their own IP and whatnot) so when I need to do something, I just ssh into them and do what needs to be done (read: never really).

Updating jails is easy-ish: For a new release, I create a custom userland base. Destroy all my old jails, and remake them with my setup scripts against the new-er userland. ...And if you zip-em-up, you can ship/sent to other physical servers to spin up there too.

Jails are great.
 
Gab1bb0 said:
So the question I'd like to ask to BSD community is: when you're on BSD do you miss Docker and in its absence what's your favorite way to spin up and manage services on a server?
Click to expand...

I've never used Docker at home (and only at work to provide en older OS environment for builds).

I want to debug low level issues. For that I work directly on the real OS (first choice) and using VirtualBox (second choice). I don't want to waste time dealing with deficiencies in the container or virtualisation system.
 
What are the applications you are running?

Some apps are only available via docker, some say they are, but can be tweaked to work with BSD, while other are fully available with `pkg install`
 
victort said:
What are the applications you are running?

Some apps are only available via docker, some say they are, but can be tweaked to work with BSD, while other are fully available with `pkg install`
Click to expand...
At the moment I'm running just three applications: Nextcloud, Vaultwarden and RustDesk: the first is written in PHP and the other ones are written in Rust and ported to FreeBSD so I have no issues with software availability.
I mostly made this topic to hear about other people's experience with BSD jails or bare metal compared to Docker on Linux and eventually to collect some tips & tricks to get started with this new approach after years of Docker.
drhowarddrfine said:
Does one ask Linux users how life is without Jails?

They have their way of doing things and we have ours
Click to expand...
I hope I haven't come off as offensive with my post. Mine was a genuine question and desire to get other people's viewpoint before eventually switching my OS and approach to server management. I know that BSD has jails, casually used them in many occasions and loved them. On the other hand Docker's hub filled with turnkey containers which allows you to spin up a service with a single command (as long as you don't need something excessively complex/custom) always felt like an huge productivity booster to me, partly because I was conditioned by all of the people/articles who praised Docker/Kubernetes as the ultimate way to manage servers.
But how I said I noticed how down the road debugging and servicing a running service based on Docker feels more cumbersome compared to Jails/VM/bare metal in most occasions so I'm reevaluating the actual productivity boost you get from Docker and I'll very likely to switch in the next months, JohnK gave me a great practical approach to do that 😃
 
Gab1bb0 said:
JohnK gave me a great practical approach to do that 😃
Click to expand...
How about a practical application as well?

First you need to set up an environment for jails. This script (cobbled together, just now, without testing) should set up your server for getting jails up and running.
Code: 
#!/bin/sh

: ${ZROOT:="zroot/jails"}
: ${ZROOT_MOUNT:="/usr/local/jails"}                   # Location where jails will be created.
: ${ZROOT_MEDIA:="zroot/jails/media"}                  # media will contain the compressed files of the downloaded userlands.
: ${ZROOT_TEMPLATES:="zroot/jails/templates"}          # templates will contain the templates when using Thin Jails.
: ${ZROOT_CONTAINERS:="zroot/jails/containers"}        # containers will contain the jails.
: ${DEFAULT_ROUTER:="192.168.0.1"}                     # How this machine will reach the internet.
: ${DEFAULT_IP4:="192.168.0.250"}                      # This machine's IP4 address.
: ${DEFAULT_INTERFACE:="em0"}                          # This machine's interface.
: ${CLONED_INTERFACE:="bridge0"}                       # For VNET stuff.

jail_setup() {
        # Setup jail directories.
        zfs create -o mountpoint=${ZROOT_MOUNT} ${ZROOT}
        zfs create ${ZROOT_MEDIA}
        zfs create ${ZROOT_TEMPLATES}
        zfs create ${ZROOT_CONTAINERS}

        # Get a userland.
        if [ ! -f ${ZROOT_MOUNT}/media/14.2-RELEASE-base.txz ]; then
                echo "Fetching jail base to ${ZROOT_MOUNT}/media"
                fetch https://download.freebsd.org/ftp/releases/amd64/amd64/14.2-RELEASE/base.txz -o ${ZROOT_MOUNT}/media/14.2-RELEASE-base.txz
        fi

        # Set up a bridge for VNET.
        echo "Setting up ${CLONED_INTERFACE}"
        sysrc -f /etc/rc.conf cloned_interfaces="${CLONED_INTERFACE}"
        sysrc -f /etc/rc.conf "ifconfig_${DEFAULT_INTERFACE}=-rxcsum -txcsum -lro -vlanmtu -vlanhwcsum -vlanhwfilter -vlanhwtag up"
        sysrc -f /etc/rc.conf "ifconfig_${CLONED_INTERFACE}=addm ${DEFAULT_INTERFACE} up"
        sysrc -f /etc/rc.conf "ifconfig_${CLONED_INTERFACE}_alias0=inet ${DEFAULT_IP4}/24"
        sysrc -f /boot/loader.conf if_bridge_load="YES"
        sysctl net.link.bridge.inherit_mac=1

        cat > /etc/jail.conf <<'EOF'
# DEFAULT OPTIONS
# (COMMON TO ALL JAILS)

# STARTUP/LOGGING
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail_console_${name}.log";

# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;
devfs_ruleset = 5;

# PATH/HOSTNAME
path = "/usr/local/jails/containers/${name}";
host.hostname = "${name}";
exec.clean;
mount.devfs;
devfs_ruleset = 5;

# PATH/HOSTNAME
path = "/usr/local/jails/containers/${name}";
host.hostname = "${name}";

# VNET/VIMAGE
vnet;
vnet.interface = "${epair}b";

# NETWORKS/INTERFACES
EOF
ip=$(echo ${DEFAULT_IP4} | awk -F"." '{print $1"."$2"."$3".\${id}/24;\""}')
{
        echo "\$ip             =   \"${ip}"
        echo "\$epair          =   \"epair\${id}\";"
        echo "\$gateway        =   \"${DEFAULT_ROUTER}\";"
        echo "\$bridge         =   \"${CLONED_INTERFACE}\";"
} >> /etc/jail.conf
        cat >> /etc/jail.conf <<'EOF'
# ADD TO bridge INTERFACE
exec.prestart   =   "/sbin/ifconfig ${epair} create up";
exec.prestart   +=  "/sbin/ifconfig ${epair}a up descr jail:${name}";
exec.prestart   +=  "/sbin/ifconfig ${bridge} addm ${epair}a up";
exec.start      +=  "/sbin/ifconfig ${epair}b ${ip} up";
exec.start      +=  "/sbin/route add default ${gateway}";
exec.poststop   =   "/sbin/ifconfig ${bridge} deletem ${epair}a";
exec.poststop   +=  "/sbin/ifconfig ${epair}a destroy";

.include "/etc/jail.conf.d/*.conf";
EOF
}

jail_setup "$@"

Then you need a way to create jails. I said I use scripts and key/value files to set up my jails, in the following I have a few examples in the example directory for you to play with. NOTE: There are many jail managers which you can use as well (the link below is not a jail manager; it only sets up a jail using the handbook method) but this is how *I* set up my jails on two (soon three) FreeBSD servers (you choose your method).

github.com

GitHub - JohnKaul/jcreate: Script to create a FreeBSD jail.conf style jail with setup script, mounts, copy-in abilities

Script to create a FreeBSD jail.conf style jail with setup script, mounts, copy-in abilities - JohnKaul/jcreate
github.com github.com

Here is a collection of scripts you can load into jails to set them up (my tool above is NOT needed to launch these scripts, just another resource for you).

github.com

GitHub - tschettervictor/bsd-apps: Collection of scripts to install popular applications inside FreeBSD jails

Collection of scripts to install popular applications inside FreeBSD jails - tschettervictor/bsd-apps
github.com github.com
 
Gab1bb0 said:
I hope I haven't come off as offensive with my post. Mine was a genuine question and desire to get other people's viewpoint before eventually switching my OS and approach to server management.
Click to expand...
Hey, its all good! I don't think anyone here is genuinely upset. Once you migrate to FreeBSD and understand the philosophy of the system, you will get some of the grouchiness too haha. I've started migrated my personal services and I am a happy and grouchy ^^
 
A bit of quick research:
A description of Docker at https://en.wikipedia.org/wiki/Docker_(software) tells me that Docker was probably meant to ameliorate dependency hell in Linux, almost like a Flatpack or a Snap...

And now for the astonishing part: The Wikipedia description does NOT mention ZFS, so it's not unreasonable to expect that Docker does NOT support ZFS... But it does.

And yet, the Wikipedia page shows May 12, 2025 as the timestamp of last edit. The links for Docker documentation do not show any dates...

Programmers are getting too lazy to properly document software and check all the boxes for criteria of useful documentation. If readers of this post get angry at this, please feel free to complain - AFTER bringing the documentation up to snuff. 😏
 
Using a jail manager like Bastille (not necessary, but easier) you can create your jail, set up your services, and export it to have a docker-like experience.

Or you can use a tool like Appjail to have an even more docker-like experience. It does everything from jails to images etc…
 
Gab1bb0 said:
At the moment I'm running just three applications: Nextcloud, Vaultwarden and RustDesk:😃
Click to expand...
I am quite familiar with Docker, been using it for years. It has it's advantages and disadvantages. I don't think I should trust things I randomly pull from Dockerhub, and the images end up chewing massive amounts of disk space if I'm not constantly pruning. Also the complexity introduced by the immutability + network namespacing is often unnecessary. Each container is it's own OS that needs updating, security scans, etc.

This is why, even on Linux, I prefer to use sandboxes to run services (using bubblewrap), and I use domain sockets instead of TCP ports to avoid needing network isolation. This works well for Nextcloud, where I can run many instances of Nextcloud for clients (each with it's own MySQL, redis, PHP-FPM, etc.) while getting the most out of my bare metal server, without the overhead of Docker. It also makes it easy to debug problems when one of the services goes down. If you are interested in the details, I've described the method here: https://tobykurien.com/simpler-linux-selfhosting/

I'm now trying to bring this same technique to FreeBSD by using jails. I much prefer a thin jail to a thick one, as I have less things to update and there's more efficient use of disk space. Since I spin up so many sandboxes (my current server has over 100 sandboxes), I don't like the typical use-case of making a jail.conf for each sandbox, etc. That's why I'm working on the sandbox script for FreeBSD.

Anyway, life is fine without Docker, although spinning up a Docker instance of something complex like Nextcloud AIO is also nice, if all you want to do is try the thing out :) Jails are flexible enough to support a similar use-case though.
 
Gab1bb0 said:
On the other hand Docker's hub filled with turnkey containers which allows you to spin up a service with a single command (as long as you don't need something excessively complex/custom) always felt like an huge productivity booster to me, partly because I was conditioned by all of the people/articles who praised Docker/Kubernetes as the ultimate way to manage servers.
Click to expand...

Maybe I am always doing excessive things, but I find the need for customization with most docker things. It is a lot less fun that way.

Gab1bb0 said:
But how I said I noticed how down the road debugging and servicing a running service based on Docker feels more cumbersome compared to Jails/VM/bare metal in most occasions so I'm reevaluating the actual productivity boost you get from Docker and I'll very likely to switch in the next months, JohnK gave me a great practical approach to do that 😃
Click to expand...

Exactly.
 
You must log in or register to reply here.
Back
Top
OSZAR »