To be slightly more constructive:
The original UNIX kernel required that everything you wanted to do with the hardware be there right at the start, and it has to be initialized right at the start too.
The current Linux kernel does this as well, unless you use systemd, however the 'generic' kernel doesn't require that everything is correct. Instead it asks each driver in turn whether it works and, if it doesn't, throws it away and reclaims the memory. This means that a generic kernel works with everything but takes a long time to boot.
Around the time the ARM was first developed, in the 1980s, the concept of a "microkernel" became popular, probably as a reaction to the increasing complexity of the UNIX kernel. Microkernels don't do anything that doesn't need to be done to run the first user process, so they boot very fast but the user process might then have to wait a bit to get hold of the hardware it needs. RISC OS is an example of such a kernel, and it boots very fast.
You can achieve the same thing with Linux by putting everything
into modules and only modprobing the stuff you need. Linux starts to look a lot like RISC OS at this point
But, as others have said by implication, the normal issue people encounter is that the user system is itself bloated; it does all manner of things that aren't necessary. This is why people quote boot times of more than 5 seconds; all that time is spent doing things that probably aren't necessary, like setting the time (a very slow operation.)
For that reason the first thing to do is to deconstruct the user boot process; everything that happens after process 1 is started. In some sense that means building your own operating system, but, in practice, if you are not providing a multi-user operating system and you care about the time-to-start then you have no choice but to do this. Anyway, it's really easy; both OpenRC and systemd can be effectively deconstructed by disabling things from the start. Debian probably isn't a good starting point - it *is* a multi-user OS so it expects to have to start up everything that any user might need.
For Linux systems busybox provides an almost canonical implementation; you don't need anything apart from busybox to get a running Linux based operating system.