why is hardware random number generator not set up by default?

[jahboater]

Isn't /dev/random deprecated now?
I thought they were advising /dev/urandom or the new getrandom() system call.

[Heater]

Yes. According to Theodore Ts'o on the Linux Kernel Crypto mailing list, /dev/random has been deprecated for a decade.

I just found this nice discussion of the mechanics of Linux random numbers:

https://www.2uo.de/myths-about-urandom/

[skypi]

Also you might want to add the "bs=10" argument to dd, otherwise you are taking lots of entropy for just a few bytes of output, and you can then skip the head bit.

Good point!

I'm quite partial to this circuit for generating random bits:

Cool!

Isn't /dev/random deprecated now?

https://unix.stackexchange.com/question ... ev-urandom

Interesting question, this post from 2016 does mention that, but highest ranked answer explains it a bit better. depends what distro!

seems a competition between CSPRNG randomness vs. the HWRNG randomness

It seems to me, after having rng-tools set up /dev/random gives a mix of outputs from CSPRNG and HWRNG, precisely the ratio I am not sure

but as I said previously, can do that manually and manually random select snippets from both devices /dev/urandom and /dev/hwrng and you dont need to set up rng-tools as the /dev/hwrng device does exist by default.

/dev/urandom is much faster than /dev/hwrng

just messing with dieharder tests, tick tock!

on a 3gb file from /dev/hwrng raspberry only weak on two dieharder tests, fails none

the "truerng" box (not the circuit above) fails tests and is weak in others looking at results posted in comments on the page linked earlier at scruss.com

running rngtest on same 3gb file is giving about 0.8 failures per 1000 tests, which is in line with results shown on the arch linux write up of rng-tools

======= test just run last night/this morning on a pi zer raspbian stretch

sudo dd if=/dev/hwrng iflag=fullblock count=3072 bs=1024k > random.pi
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB, 3.0 GiB) copied, 28286.3 s, 114 kB/s

dieharder -a -g 201 -f random.pi
#=============================================================================#
# dieharder version 3.31.1 Copyright 2003 Robert G. Brown #
#=============================================================================#
rng_name | filename |rands/second|
file_input_raw| random.pi| 1.16e+07 |
#=============================================================================#
test_name |ntup| tsamples |psamples| p-value |Assessment
#=============================================================================#
diehard_birthdays| 0| 100| 100|0.12817816| PASSED
diehard_operm5| 0| 1000000| 100|0.22048199| PASSED
diehard_rank_32x32| 0| 40000| 100|0.91056214| PASSED
diehard_rank_6x8| 0| 100000| 100|0.41087991| PASSED
diehard_bitstream| 0| 2097152| 100|0.27521249| PASSED
diehard_opso| 0| 2097152| 100|0.94962586| PASSED
diehard_oqso| 0| 2097152| 100|0.77127671| PASSED
diehard_dna| 0| 2097152| 100|0.82050944| PASSED
diehard_count_1s_str| 0| 256000| 100|0.01849413| PASSED
# The file file_input_raw was rewound 1 times
diehard_count_1s_byt| 0| 256000| 100|0.08190277| PASSED
# The file file_input_raw was rewound 1 times
diehard_parking_lot| 0| 12000| 100|0.83699181| PASSED
# The file file_input_raw was rewound 1 times
diehard_2dsphere| 2| 8000| 100|0.95686960| PASSED
# The file file_input_raw was rewound 1 times
diehard_3dsphere| 3| 4000| 100|0.99233785| PASSED
# The file file_input_raw was rewound 1 times
diehard_squeeze| 0| 100000| 100|0.96072528| PASSED
# The file file_input_raw was rewound 1 times
diehard_sums| 0| 100| 100|0.57914487| PASSED
# The file file_input_raw was rewound 1 times
diehard_runs| 0| 100000| 100|0.90135448| PASSED
diehard_runs| 0| 100000| 100|0.16385325| PASSED
# The file file_input_raw was rewound 1 times
diehard_craps| 0| 200000| 100|0.08028264| PASSED
diehard_craps| 0| 200000| 100|0.56360433| PASSED
# The file file_input_raw was rewound 4 times
marsaglia_tsang_gcd| 0| 10000000| 100|0.33259823| PASSED
marsaglia_tsang_gcd| 0| 10000000| 100|0.03050897| PASSED
# The file file_input_raw was rewound 4 times
sts_monobit| 1| 100000| 100|0.97148091| PASSED
# The file file_input_raw was rewound 4 times
sts_runs| 2| 100000| 100|0.28219246| PASSED
# The file file_input_raw was rewound 4 times
sts_serial| 1| 100000| 100|0.35579413| PASSED
sts_serial| 2| 100000| 100|0.36324454| PASSED
sts_serial| 3| 100000| 100|0.96272178| PASSED
sts_serial| 3| 100000| 100|0.01257694| PASSED
sts_serial| 4| 100000| 100|0.95116462| PASSED
sts_serial| 4| 100000| 100|0.77351319| PASSED
sts_serial| 5| 100000| 100|0.11167134| PASSED
sts_serial| 5| 100000| 100|0.43545752| PASSED
sts_serial| 6| 100000| 100|0.03955459| PASSED
sts_serial| 6| 100000| 100|0.49509367| PASSED
sts_serial| 7| 100000| 100|0.54105723| PASSED
sts_serial| 7| 100000| 100|0.34156956| PASSED
sts_serial| 8| 100000| 100|0.10471962| PASSED
sts_serial| 8| 100000| 100|0.16366930| PASSED
sts_serial| 9| 100000| 100|0.19866319| PASSED
sts_serial| 9| 100000| 100|0.83008866| PASSED
sts_serial| 10| 100000| 100|0.52377778| PASSED
sts_serial| 10| 100000| 100|0.88503393| PASSED
sts_serial| 11| 100000| 100|0.80646749| PASSED
sts_serial| 11| 100000| 100|0.88100940| PASSED
sts_serial| 12| 100000| 100|0.98819546| PASSED
sts_serial| 12| 100000| 100|0.36571213| PASSED
sts_serial| 13| 100000| 100|0.34246257| PASSED
sts_serial| 13| 100000| 100|0.20584354| PASSED
sts_serial| 14| 100000| 100|0.28470331| PASSED
sts_serial| 14| 100000| 100|0.96681292| PASSED
sts_serial| 15| 100000| 100|0.17196949| PASSED
sts_serial| 15| 100000| 100|0.56678498| PASSED
sts_serial| 16| 100000| 100|0.10885051| PASSED
sts_serial| 16| 100000| 100|0.21679957| PASSED
# The file file_input_raw was rewound 4 times
rgb_bitdist| 1| 100000| 100|0.75755337| PASSED
# The file file_input_raw was rewound 4 times
rgb_bitdist| 2| 100000| 100|0.24559046| PASSED
# The file file_input_raw was rewound 4 times
rgb_bitdist| 3| 100000| 100|0.31840340| PASSED
# The file file_input_raw was rewound 4 times
rgb_bitdist| 4| 100000| 100|0.70484062| PASSED
# The file file_input_raw was rewound 4 times
rgb_bitdist| 5| 100000| 100|0.22229752| PASSED
# The file file_input_raw was rewound 4 times
rgb_bitdist| 6| 100000| 100|0.95721531| PASSED
# The file file_input_raw was rewound 4 times
rgb_bitdist| 7| 100000| 100|0.95461611| PASSED
# The file file_input_raw was rewound 4 times
rgb_bitdist| 8| 100000| 100|0.10947090| PASSED
# The file file_input_raw was rewound 5 times
rgb_bitdist| 9| 100000| 100|0.81983779| PASSED
# The file file_input_raw was rewound 5 times
rgb_bitdist| 10| 100000| 100|0.72972825| PASSED
# The file file_input_raw was rewound 5 times
rgb_bitdist| 11| 100000| 100|0.86017380| PASSED
# The file file_input_raw was rewound 6 times
rgb_bitdist| 12| 100000| 100|0.49492861| PASSED
# The file file_input_raw was rewound 6 times
rgb_minimum_distance| 2| 10000| 1000|0.45879860| PASSED
# The file file_input_raw was rewound 6 times
rgb_minimum_distance| 3| 10000| 1000|0.79775642| PASSED
# The file file_input_raw was rewound 6 times
rgb_minimum_distance| 4| 10000| 1000|0.65490744| PASSED
# The file file_input_raw was rewound 6 times
rgb_minimum_distance| 5| 10000| 1000|0.99998384| WEAK
# The file file_input_raw was rewound 6 times
rgb_permutations| 2| 100000| 100|0.06886222| PASSED
# The file file_input_raw was rewound 6 times
rgb_permutations| 3| 100000| 100|0.12798774| PASSED
# The file file_input_raw was rewound 6 times
rgb_permutations| 4| 100000| 100|0.05407457| PASSED
# The file file_input_raw was rewound 6 times
rgb_permutations| 5| 100000| 100|0.71970459| PASSED
# The file file_input_raw was rewound 6 times
rgb_lagged_sum| 0| 1000000| 100|0.89914248| PASSED
# The file file_input_raw was rewound 6 times
rgb_lagged_sum| 1| 1000000| 100|0.91807981| PASSED
# The file file_input_raw was rewound 7 times
rgb_lagged_sum| 2| 1000000| 100|0.88250769| PASSED
# The file file_input_raw was rewound 7 times
rgb_lagged_sum| 3| 1000000| 100|0.84357829| PASSED
# The file file_input_raw was rewound 8 times
rgb_lagged_sum| 4| 1000000| 100|0.66286812| PASSED
# The file file_input_raw was rewound 8 times
rgb_lagged_sum| 5| 1000000| 100|0.65416539| PASSED
# The file file_input_raw was rewound 9 times
rgb_lagged_sum| 6| 1000000| 100|0.65814328| PASSED
# The file file_input_raw was rewound 10 times
rgb_lagged_sum| 7| 1000000| 100|0.88325203| PASSED
# The file file_input_raw was rewound 11 times
rgb_lagged_sum| 8| 1000000| 100|0.73038115| PASSED
# The file file_input_raw was rewound 13 times
rgb_lagged_sum| 9| 1000000| 100|0.72493156| PASSED
# The file file_input_raw was rewound 14 times
rgb_lagged_sum| 10| 1000000| 100|0.42001270| PASSED
# The file file_input_raw was rewound 16 times
rgb_lagged_sum| 11| 1000000| 100|0.76646166| PASSED
# The file file_input_raw was rewound 17 times
rgb_lagged_sum| 12| 1000000| 100|0.07396891| PASSED
# The file file_input_raw was rewound 19 times
rgb_lagged_sum| 13| 1000000| 100|0.65308069| PASSED
# The file file_input_raw was rewound 21 times
rgb_lagged_sum| 14| 1000000| 100|0.82744548| PASSED
# The file file_input_raw was rewound 23 times
rgb_lagged_sum| 15| 1000000| 100|0.29800685| PASSED
# The file file_input_raw was rewound 25 times
rgb_lagged_sum| 16| 1000000| 100|0.47557053| PASSED
# The file file_input_raw was rewound 27 times
rgb_lagged_sum| 17| 1000000| 100|0.11015354| PASSED
# The file file_input_raw was rewound 29 times
rgb_lagged_sum| 18| 1000000| 100|0.90713869| PASSED
# The file file_input_raw was rewound 32 times
rgb_lagged_sum| 19| 1000000| 100|0.95001124| PASSED
# The file file_input_raw was rewound 35 times
rgb_lagged_sum| 20| 1000000| 100|0.98604591| PASSED
# The file file_input_raw was rewound 37 times
rgb_lagged_sum| 21| 1000000| 100|0.52015903| PASSED
# The file file_input_raw was rewound 40 times
rgb_lagged_sum| 22| 1000000| 100|0.95892483| PASSED
# The file file_input_raw was rewound 43 times
rgb_lagged_sum| 23| 1000000| 100|0.00003096| WEAK
# The file file_input_raw was rewound 46 times
rgb_lagged_sum| 24| 1000000| 100|0.06317806| PASSED
# The file file_input_raw was rewound 49 times
rgb_lagged_sum| 25| 1000000| 100|0.86579617| PASSED
# The file file_input_raw was rewound 53 times
rgb_lagged_sum| 26| 1000000| 100|0.58423834| PASSED
# The file file_input_raw was rewound 56 times
rgb_lagged_sum| 27| 1000000| 100|0.86529381| PASSED
# The file file_input_raw was rewound 60 times
rgb_lagged_sum| 28| 1000000| 100|0.41184149| PASSED
# The file file_input_raw was rewound 64 times
rgb_lagged_sum| 29| 1000000| 100|0.81736046| PASSED
# The file file_input_raw was rewound 67 times
rgb_lagged_sum| 30| 1000000| 100|0.28461423| PASSED
# The file file_input_raw was rewound 71 times
rgb_lagged_sum| 31| 1000000| 100|0.36322747| PASSED
# The file file_input_raw was rewound 76 times
rgb_lagged_sum| 32| 1000000| 100|0.05140941| PASSED
# The file file_input_raw was rewound 76 times
rgb_kstest_test| 0| 10000| 1000|0.02448435| PASSED
# The file file_input_raw was rewound 76 times
dab_bytedistrib| 0| 51200000| 1|0.84928041| PASSED
# The file file_input_raw was rewound 76 times
dab_dct| 256| 50000| 1|0.37264438| PASSED
Preparing to run test 207. ntuple = 0
# The file file_input_raw was rewound 76 times
dab_filltree| 32| 15000000| 1|0.64814623| PASSED
dab_filltree| 32| 15000000| 1|0.36970382| PASSED
Preparing to run test 208. ntuple = 0
# The file file_input_raw was rewound 76 times
dab_filltree2| 0| 5000000| 1|0.52176096| PASSED
dab_filltree2| 1| 5000000| 1|0.37962593| PASSED
Preparing to run test 209. ntuple = 0
# The file file_input_raw was rewound 76 times
dab_monobit2| 12| 65000000| 1|0.30470240| PASSED




rngtest: bits received from input: 21858240032
rngtest: FIPS 140-2 successes: 1092038
rngtest: FIPS 140-2 failures: 874
rngtest: FIPS 140-2(2001-10-10) Monobit: 116
rngtest: FIPS 140-2(2001-10-10) Poker: 122
rngtest: FIPS 140-2(2001-10-10) Runs: 358
rngtest: FIPS 140-2(2001-10-10) Long run: 285
rngtest: FIPS 140-2(2001-10-10) Continuous run: 0
rngtest: input channel speed: (min=178.257; avg=8314.235; max=19073.486)Mibits/s
rngtest: FIPS tests speed: (min=11.659; avg=38.808; max=40.155)Mibits/s
rngtest: Program run time: 540006024 microseconds

[skypi]

in conclusion, /dev/urandom on pi without rng-tools is about same rngtest error rate as /dev/random or /dev/hwrng on pi with rng-tools installed.

except the possible area close to boot time.

anyway, to regen your ssh keys (without a passphrase)

#rsa and dsa keys
sudo ssh-keygen -N "" -h -t rsa -b 4096 -f /etc/ssh/ssh_host_rsa_key
sudo ssh-keygen -N "" -h -t dsa -f /etc/ssh/ssh_host_dsa_key

#ecdsa key
sudo ssh-keygen -N "" -h -t ecdsa -f /etc/ssh/ssh_host_ecdsa_key

#for archlinux etc
sudo ssh-keygen -N "" -h -t ed25519 -f /etc/ssh/ssh_host_ed25519_key

[Heater]

skypi,

If you are going to be a (pseudo)random number nerd you will want to look at testing with PractRand and TestU01.

testu01 is available as a Debian/Raspbian package.

PractRand will have to be built from source https://github.com/MartyMacGyver/PractRand

Some advice for running bractrand here: http://www.pcg-random.org/posts/how-to- ... trand.html

Note: that some PRNG failures won't show up until you have fed PractRand hundreds of giga bytes!

Warning: Getting obsessive about making random number generators and testing them is an addiction that is hard to shake off

[skypi]

I was testing using arch, and that was where I first discovered needed to set up the hwrng, but came to the conclusion using arch was a lot of work where raspbian just works, which when trying to concentrate on a few levels higher scada/node stuff makes a big difference not having to deal with the intricacies of what is installed or not and how to configure the various packages, and even select which ones to use from a large selection for the same task like cron even. When i get to distro stage i may look at arch again to reduce bloat further than stretch lite.

just happened to notice an article that said raspberry was generating weak ssh keys so thought I would look into it a bit.

I reckon using multiple sources would probably be best, but then need to know more detail on the ssh-keygen function. I am satisfied dev/urandom is not bad, and that can use the hwrng if required.

thanks for all the useful references everyone!

[scruss]

One should always decide on an acceptable level of failures in your random number tests. Any long enough truly random stream could contain short runs of consecutive values. Also, if your randomness tests never fail, the only thing it proves is that you haven't tested enough criteria.

Heater's circuit, one I've used many times before, is not without problems:

• as components age, the output of the avalanche noise circuit tends to fade. This is very evident if you use an ADC to read the output: what starts out as a nice full-range output will fade over weeks or months to fewer and fewer bits. So it's no guarantee that what was random enough yesterday will be random enough today
• the circuit looks like it relies on two 4.7 kΩ resistors to act as a voltage divider. If these aren't exactly the same resistance, the output will have bias one way or another
• the circuit doesn't have any way to detect external interference or tampering. A small RF source could easily placed nearby to influence the output (my microwave oven was quite good at this). Now, if you're just experimenting, you'll have control over the device and you won't need to worry about malicious users. I have heard of very secure HWRNG devices that effectively self-destruct, never to work again, if they detect tampering or external interference. This kind of paranoia is way above my pay grade, though.

ISTR Intel published an article on a few things you should care about when producing an HWRNG round about when RdRand came out. Can't find it right now as I'm stuck on the bad network at YUL waiting for the runway at YHX defrost so I can get to the assisting tech buildathon I'm co-running later in the week at Acadia University.

[jahboater]

One should always decide on an acceptable level of failures in your random number tests. Any long enough truly random stream could contain short runs of consecutive values.

Thats the good thing about pseudo random number generators, they cannot ever do that.
Even the simplest LCG:

state = 2862933555777941757 * state + 3037000493

has a period of 2^64 before repeating itself and obviously will never produce consecutive numbers.
Monsters like the mersenne twister have a period of 2^19937.

[Heater]

scruss,

...as components age, the output of the avalanche noise circuit tends to fade

That is interesting. Why is that? How long before it is significant?

...the circuit looks like it relies on two 4.7 kΩ resistors to act as a voltage divider. If these aren't exactly the same resistance, the output will have bias one way or another

Yes but all such hardware will have a bias. That is why we clean it up it with the Von Neumann "whitening" algorithm before using the output. https://en.wikipedia.org/wiki/Hardware_ ... _generator

...the circuit doesn't have any way to detect external interference or tampering....

Very true.

But what are we supposed to do? Even if we use some quantum mechanical source of randomness there is a long chain of electronics between it and our computer that can be interfered and tampered with.

Should we hide in a bunker, surrounded by lead, kilometers under the ground, to escape enemy interference?

[Heater]

jahboater,

...has a period of 2^64 before repeating itself and obviously will never produce consecutive numbers

Not quite.

If you only deliver 32 bits of output on each iteration, as is quite common, there is is a lot of opportunity to to produce consecutive numbers.

The Mersenne Twister has a long period. But is broken in other ways.

[jahboater]

jahboater,

...has a period of 2^64 before repeating itself and obviously will never produce consecutive numbers

Not quite.

No, the period is exactly 2^64 (it returns 64-bits each iteration). It was discovered by L'Ecuyer. The addend is prime.
See: https://nuclear.llnl.gov/CNP/rng/rngman/node4.html.
None of these things are perfect, but this one performs well as lcg's go, and executes with a single "madd" instruction on the Pi3, so its very fast.
Knuth gave the criteria for maximal period for lcg's which I have forgotten, but it is fairly simple. He found another good 64-bit lcg with proven full period: state = 6364136223846793005 * state + 1442695040888963407

You are right, MT19937 isn't as good as people think (and I thought). According to Melissa's presentation you linked to, a simple 88bit+ LCG can beat it for some tests like bigcrush. Which is a shame because it is now the default PRNG for many many things.

[Heater]

jahboater,

Perhaps we are talking at cross-purposes.

If the period is 2^64 and each random number returned is 64 bits then it follows that for any given batch of output, less than 2^64, any given number can only occur once.

This is not what we expect from a truly random stream of 64 bit values. As scruss says: "Any long enough truly random stream could contain short runs of consecutive values."

For this reason that simple LCG does a very poor job of simulating randomness. If you use all 64 bits of output on every iteration.

Even worse is that the sequence is trivially predictable if you have even just one of those 64 bit outputs.

[jahboater]

If the period is 2^64 and each random number returned is 64 bits then it follows that for any given batch of output, less than 2^64, any given number can only occur once.

Correct

This is not what we expect from a truly random stream of 64 bit values. As scruss says: "Any long enough truly random stream could contain short runs of consecutive values."

Correct again; but I said:-

Thats the good thing about pseudo random number generators, they cannot ever do that.

- yes we are talking cross purposes!

Even worse is that the sequence is trivially predictable if you have even just one of those 64 bit outputs.

Indeed, thats the nature of pseudo rng's if you know the algorithm, otherwise you need more than one.

[Heater]

jahboater,

Thats the good thing about pseudo random number generators, they cannot ever do that.

That is the part I don't understand. Why is that a good thing?

My idea of a PRNG is that it strives to simulates a random stream. That it's output is not distinquishable from random, at least not without non-trivial effort by an observer. As such a statistically appropriate number of repetitions, consecutive values, etc in it's output is a good thing.

Indeed, that's the nature of pseudo rng's if you know the algorithm, otherwise you need more than one.

Not entirely. Cryptographically secure PRNGs are designed to not be predictable. Even if you do know the algorithm.

https://en.wikipedia.org/wiki/Cryptogra ... _generator

As usual it all comes down to one's requirements. There are many PRNG algorithms out there offering an assortment of features.

[jahboater]

Thats the good thing about pseudo random number generators, they cannot ever do that.

That is the part I don't understand. Why is that a good thing?

Thats a good question. I probably should not have used the word "good".
I guess a TRNG could in theory produce a long sequence of identical numbers, in the same way that in theory I could win the lottery every single week! For many uses of random numbers such as simulation, and lotteries for that matter, that would be highly undesirable. Could you imagine the outcry if the lottery produced the same numbers several weeks in a row! People would shout "Its fixed", "the machine is broken", "its not random at all"
I think we are digressing into philosophy ....

The premium bonds lottery in the UK, at least at first, used a metre square metal plate on the roof of a building. The coordinates where cosmic ray particles hit the plate were considered random.

My idea of a PRNG is that it strives to simulates a random stream. That it's output is not distinquishable from random, at least not without non-trivial effort by an observer. As such a statistically appropriate number of repetitions, consecutive values, etc in it's output is a good thing.

Yes I agree.
It follows though, that there is no expectation that a TRNG will produce what are considered statistically good (as in test passing) numbers - it could produce anything. Which may explain the popularity of CSPRNG's, and why Intel hardware goes to such lengths to test the randomness from the thermal noise before presenting it to the user.

[Heater]

Yes. It's amazing how a seemingly simple, technical, little thing like a PRNG can lead so much debate and ultimately philosophy.

I was once playing with a simple PRNG algorithm that had multiple 32 bit words as it state. Sorry I forget which one now. Anyway if you seeded it's state with a single 1 bit and all the rest zero it would take billions of iterations to start producing anything that looked random. It would produce a lot of zeros in it's output during that time.

Initially I thought that was pretty crappy. But on reflection it seemed to me that as it's cycle length was 2^1024 or something ridiculously huge then such a long non-random looking sequence is what we would expect to see eventually if it were really random.

[jahboater]

I was once playing with a simple PRNG algorithm that had multiple 32 bit words as it state. Sorry I forget which one now. Anyway if you seeded it's state with a single 1 bit and all the rest zero it would take billions of iterations to start producing anything that looked random. It would produce a lot of zeros in it's output during that time.

I think thats called the "0-excess state" and it is a well known problem with the mersenne twister. They since improved the seeding to use an LCG to fill in all the state with large numbers.

Never seed an xorshift generator with zero, it will produce only zeros for eternity ...

I added a random number function to a calculator, it didn't have to be cryptographically secure or fancy in any way, but it should give that warm feeling to the user (that you didn't get) when it immediately produces "random looking" numbers. Something like this from Knuth was ideal: "state = 6364136223846793005 * state + 1442695040888963407" it recovers from a zero seed instantly and it will never repeat in practice.

As you say, lots of different requirements.

[davidcoton]

"state = 6364136223846793005 * state + 1442695040888963407"

state = 6364136223846793005 * state + 1442695040888963407

That quote repeated after a fairly short interval. Obviously not random.