Liberation through Computer Literacy

[John_Spikowski]

Even if some of the extensions leak

It seems GMP is the only extension that has a memory use/free issue. All the other extension modules work fine.

AIR wrote the GMP2 extension module and has indicated via his testing memory is release when scriba exits. For the real world tasks that might require big int support it works just fine.

Absurd shouldn't be the standard of excellence.

[ejolson]

Unfortunately, Fido is currently coding a sluggish boot loader.

I had a conversation with the developer of Fido Basic last night. As usual the canine coder sounded barking mad.

Apparently it's possible to add a bunch of 128MB fat-formatted DOS partitions to the end of the SD card using an extended partition. These extra partitions can then be used as boot partitions by executing

# systemctl reboot 5

for example, to reboot the Pi using partition 5 as the boot partition.

It now becomes simple to create a sluggish USB bootloader for the turbo encabulator by using rsync to update one of the extra boot partitions with a copy of boot from the USB drive and then rebooting the updated partition. It may even be possible to scan iSCSI targets over a local WireGuard-encrypted VPN and do the same.

As a result, the slug bootloader is almost finished. This liberation, that comes from coding, may allow the person who owns the 4B to enter the data center with the root filesystem securely mounted from a NAS appliance.

[scruss]

…. As a result, the slug bootloader is almost finished.

Pfft. Call that sluggish? 'Cos it's a positive racing snail — _@/ — compared to what I'm building: a computer with a clock that can be slowed down to 3 Hz. Yup, I didn't miss a prefix out: that's three clock pulses per second. You can slow it to zero, but that makes the blinkenlights rather boring.

It's an 1802-based Retro Elf. So much slowness.

[John_Spikowski]

For grins, here is AIR's GMP2 1milfibo running on my new laptop and Ubuntu 19.04.

Code: Select all

IMPORT gmp2.bas

count = 4784969

a = 1
b = 0
p = 0
q = 1

WHILE count > 0
    IF (count % 2) = 0 THEN
        psq   = gmp2::mul(p, p)
        qsq   = gmp2::mul(q, q)
        twopq = gmp2::mul(p, q)
        twopq = gmp2::mul_si(twopq, 2)
        p     = gmp2::add(psq, qsq)
        q     = gmp2::add(twopq, qsq)
        count = count / 2
    ELSE
        bq    = gmp2::mul(b, q)
        aq    = gmp2::mul(a, q)
        ap    = gmp2::mul(a, p)
        bp    = gmp2::mul(b, p)
        a     = gmp2::add(bq, aq)
        a     = gmp2::add(a, ap)
        b     = gmp2::add(bp, aq)
        count = count - 1
    END IF
WEND
PRINT LEN(b),"\n"

Output

Code: Select all

ubuntu@ubuntu:~/sbrt/examples$ time scriba 1milfibo.sb
1000000

real	0m2.590s
user	0m2.578s
sys	0m0.012s
ubuntu@ubuntu:~/sbrt/examples$ 

Hopefully I will have some time to get the hash extension module working with the anagram challenge and return some respectable results.

[John_Spikowski]

Here is the time for 1milfibo on the RPI 4B 4GB.

Code: Select all

pi@RPi4B:~/sbrt/examples $ time scriba 1milfibo.sb
1000000

real	0m19.103s
user	0m18.801s
sys	0m0.281s
pi@RPi4B:~/sbrt/examples $ 

Is there there a RPi 4B Fibo chart?

[ejolson]

Here is the time for 1milfibo on the RPI 4B 4GB.

Code: Select all

pi@RPi4B:~/sbrt/examples $ time scriba 1milfibo.sb
1000000

real	0m19.103s
user	0m18.801s
sys	0m0.281s
pi@RPi4B:~/sbrt/examples $ 

Is there there a RPi 4B Fibo chart?

The Fibonacci results I reported earlier were made using the Pi Zero to calculate thousands of different Fibonacci numbers up to a million digits. I find the timings you just reported interesting. It seems your new notebook is about 7.3 times faster than the Pi 4B.

How do they compare from a price-performance point of view? What about energy efficiency?

[John_Spikowski]

I picked up the Lenovo laptop at Costco on sale for $800. Normally sells for a grand.

With the Sandisk SSD and persistant LIVE, I get that performance under Win10 and Ubuntu. I'm happy at the moment how it turn out including the splitters for HDMI and USB for my Raspberry Pi farm.

The Lenovo laptop has no DVD drive or Ethernet port. If I screw up the Windows 10 install it might be difficult to recover. I'm keeping Linux's gruby hands off my main SSD drive.

[gkreidl]

Here is the time for 1milfibo on the RPI 4B 4GB.

Code: Select all

pi@RPi4B:~/sbrt/examples $ time scriba 1milfibo.sb
1000000

real	0m19.103s
user	0m18.801s
sys	0m0.281s
pi@RPi4B:~/sbrt/examples $ 

Is there there a RPi 4B Fibo chart?

The Fibonacci results I reported earlier were made using the Pi Zero to calculate thousands of different Fibonacci numbers up to a million digits. I find the timings you just reported interesting. It seems your new notebook is about 7.3 times faster than the Pi 4B.

How do they compare from a price-performance point of view? What about energy efficiency?

No. He used gmp2 library. If I do the same using Python3 on a RPi 4, it's more than twice as fast as his Desktop and 20 times faster than his RPi 4 script.

Code: Select all

 ./fibo_fdi.py -t
Fibonacci(4784969) calculation took 0.2052764892578125 seconds
String conversion took 0.7193107604980469 seconds
Fibonacci number has 1000000 digits

[John_Spikowski]

Please remember that when ScriptBasic entered this challenge starting native using the QB code it was taking 4 hours or so to complete. Adding GMP BIGINT support was a huge step forward.

I'm happy and for me that's all that counts.

[ejolson]

Here is the time for 1milfibo on the RPI 4B 4GB.

Code: Select all

pi@RPi4B:~/sbrt/examples $ time scriba 1milfibo.sb
1000000

real	0m19.103s
user	0m18.801s
sys	0m0.281s
pi@RPi4B:~/sbrt/examples $ 

Is there there a RPi 4B Fibo chart?

The Fibonacci results I reported earlier were made using the Pi Zero to calculate thousands of different Fibonacci numbers up to a million digits. I find the timings you just reported interesting. It seems your new notebook is about 7.3 times faster than the Pi 4B.

How do they compare from a price-performance point of view? What about energy efficiency?

No. He used gmp2 library. If I do the same using Python3 on a RPi 4, it's more than twice as fast as his Desktop and 20 times faster than his RPi 4 script.

Code: Select all

 ./fibo_fdi.py -t
Fibonacci(4784969) calculation took 0.2052764892578125 seconds
String conversion took 0.7193107604980469 seconds
Fibonacci number has 1000000 digits

Aha! But now you have to run fibo_fdi.py on that new Lenovo laptop to make a comparison relative to that program.

On one hand 64-bit will slow the Python down; on the other, the 32-bit Raspbian version is also slowed by being reverse compatible with the ARMv6 architecture.

Do you think the Python 3 code will run more than 7.3 times faster on that new notebook or less?

[John_Spikowski]

Code: Select all

#!/usr/bin/python3

## Fibonacci script using an improved version of fast doubling

import time, sys, math

use_gmp = True
use_gmp_fibo = False
timing_only = False
string_conversion = True
index = 4784969
compare = False

usage = '''Usage: python fibo_fdi.py [options] [index]
or
       python3 fibo_fdi.py [options] [index]
or
       pypy fibo_fdi.py [options] [index]
       Note: pypy cannot use gmpy2

index: Fibonacci number to be calculated, default = 4784969

Options:
-t don't print result, only timing for Fibonacci calculation and string conversion
-n don't convert result to string in timing or compare mode 
-i use Python internal BIGINTS, even if gmpy2 is installed
-c cheat = use GMP's internal Fibonacci function if possible
--tt compare internal and GMP BIGINT timing for a fixed list of values
-h, --help print this text
'''
def fibo(n):
    if n <= 0:
        return 0
    if use_gmp:
        a = mpz(0)
        b = mpz(1)
    else:
        a = 0
        b = 1
    numbits = int(math.log(n,2))
    for i in range(numbits,-1,-1):
        if i > 0:
            d = a*(b*2-a)
            e = a*a+b*b
            if (n >> i) & 1:
                a = e
                b = d + e 
            else:
                a = d
                b = e
        else:
            if n & 1:
                d = a*a+b*b
            else:
               d = a*(b*2-a)
    return d

if len(sys.argv) > 1:
    for arg in sys.argv[1:]:
        if arg in ['-h','--help']:
            print(usage)
            sys.exit(0)
        if arg in ['-t', '-i', '-c', '-n', '--tt']:
            if arg == '-t':
                timing_only = True
            elif arg == '-i':
                use_gmp = False
            elif arg == '-c':
                use_gmp_fibo = True
            elif arg == '-n':
                string_conversion = False
            elif arg == '--tt':
                compare = True
        else:
            try:
                index = int(arg)
            except:
                pass

if use_gmp:
    try:
        import gmpy2
        from gmpy2 import mpz
    except:
        use_gmp = False
        use_gmp_fibo = False

if compare and use_gmp:
    ctable = [100,1000,10000,100000,1000000,4784969,10000000] 
    for v in ctable:
        use_gmp = False
        t = time.time()
        res = fibo(v)
        if string_conversion:
            restr = str(res)
        fibt = time.time()-t
        use_gmp = True
        t = time.time()
        res = fibo(v)
        if string_conversion:
            restr = str(res)
        fibtgmp = time.time()-t
        print((v,fibt,fibtgmp,fibt/fibtgmp))
    sys.exit(0)   

if timing_only:
    if use_gmp and use_gmp_fibo:
        t = time.time()
        res = gmpy2.fib(index)
        fibt = time.time()-t
    else:
        t = time.time()
        res = fibo(index)
        fibt = time.time()-t
    if string_conversion:
        t = time.time()
        restr = str(res)
        strcvt = time.time()-t
    print('Fibonacci('+str(index)+') calculation took '+str(fibt)+' seconds')
    if string_conversion:
        print('String conversion took '+str(strcvt)+' seconds')
        print('Fibonacci number has '+str(len(restr))+' digits')
else:
    if use_gmp and use_gmp_fibo:
        print(gmpy2.fib(index))
    else:
        print(fibo(index))

Lenovo laptop

Code: Select all

ubuntu@ubuntu:~/sbrt/examples$ time ./fibo_fdi.py 4784969 > fibo_py.out

real	0m11.388s
user	0m11.374s
sys	0m0.012s
ubuntu@ubuntu:~/sbrt/examples$ ls -l fibo_py.out
-rw-r--r-- 1 ubuntu ubuntu 1000001 Oct  8 01:10 fibo_py.out
ubuntu@ubuntu:~/sbrt/examples$ 

Raspberry Pi 4B 4GB

Code: Select all

pi@RPi4B:~/sbrt/examples $ time ./fibo_fdi.py 4784969 > fibo_py.out

real	2m50.590s
user	2m50.486s
sys	0m0.060s
pi@RPi4B:~/sbrt/examples $ 
pi@RPi4B:~/sbrt/examples $ ls -l fibo_py.out
-rw-r--r-- 1 pi pi 1000001 Oct  7 18:23 fibo_py.out
pi@RPi4B:~/sbrt/examples $ 

[ejolson]

…. As a result, the slug bootloader is almost finished.

Pfft. Call that sluggish? 'Cos it's a positive racing snail — _@/ — compared to what I'm building: a computer with a clock that can be slowed down to 3 Hz. Yup, I didn't miss a prefix out: that's three clock pulses per second. You can slow it to zero, but that makes the blinkenlights rather boring.

It's an 1802-based Retro Elf. So much slowness.

How is the project going? When do you think it will be finished? Is there an 1802 emulator that runs on the Pi?

[gkreidl]

Code: Select all

#!/usr/bin/python3

## Fibonacci script using an improved version of fast doubling

import time, sys, math

use_gmp = True
use_gmp_fibo = False
timing_only = False
string_conversion = True
index = 4784969
compare = False

usage = '''Usage: python fibo_fdi.py [options] [index]
or
       python3 fibo_fdi.py [options] [index]
or
       pypy fibo_fdi.py [options] [index]
       Note: pypy cannot use gmpy2

index: Fibonacci number to be calculated, default = 4784969

Options:
-t don't print result, only timing for Fibonacci calculation and string conversion
-n don't convert result to string in timing or compare mode 
-i use Python internal BIGINTS, even if gmpy2 is installed
-c cheat = use GMP's internal Fibonacci function if possible
--tt compare internal and GMP BIGINT timing for a fixed list of values
-h, --help print this text
'''
def fibo(n):
    if n <= 0:
        return 0
    if use_gmp:
        a = mpz(0)
        b = mpz(1)
    else:
        a = 0
        b = 1
    numbits = int(math.log(n,2))
    for i in range(numbits,-1,-1):
        if i > 0:
            d = a*(b*2-a)
            e = a*a+b*b
            if (n >> i) & 1:
                a = e
                b = d + e 
            else:
                a = d
                b = e
        else:
            if n & 1:
                d = a*a+b*b
            else:
               d = a*(b*2-a)
    return d

if len(sys.argv) > 1:
    for arg in sys.argv[1:]:
        if arg in ['-h','--help']:
            print(usage)
            sys.exit(0)
        if arg in ['-t', '-i', '-c', '-n', '--tt']:
            if arg == '-t':
                timing_only = True
            elif arg == '-i':
                use_gmp = False
            elif arg == '-c':
                use_gmp_fibo = True
            elif arg == '-n':
                string_conversion = False
            elif arg == '--tt':
                compare = True
        else:
            try:
                index = int(arg)
            except:
                pass

if use_gmp:
    try:
        import gmpy2
        from gmpy2 import mpz
    except:
        use_gmp = False
        use_gmp_fibo = False

if compare and use_gmp:
    ctable = [100,1000,10000,100000,1000000,4784969,10000000] 
    for v in ctable:
        use_gmp = False
        t = time.time()
        res = fibo(v)
        if string_conversion:
            restr = str(res)
        fibt = time.time()-t
        use_gmp = True
        t = time.time()
        res = fibo(v)
        if string_conversion:
            restr = str(res)
        fibtgmp = time.time()-t
        print((v,fibt,fibtgmp,fibt/fibtgmp))
    sys.exit(0)   

if timing_only:
    if use_gmp and use_gmp_fibo:
        t = time.time()
        res = gmpy2.fib(index)
        fibt = time.time()-t
    else:
        t = time.time()
        res = fibo(index)
        fibt = time.time()-t
    if string_conversion:
        t = time.time()
        restr = str(res)
        strcvt = time.time()-t
    print('Fibonacci('+str(index)+') calculation took '+str(fibt)+' seconds')
    if string_conversion:
        print('String conversion took '+str(strcvt)+' seconds')
        print('Fibonacci number has '+str(len(restr))+' digits')
else:
    if use_gmp and use_gmp_fibo:
        print(gmpy2.fib(index))
    else:
        print(fibo(index))

Lenovo laptop

Code: Select all

ubuntu@ubuntu:~/sbrt/examples$ time ./fibo_fdi.py 4784969 > fibo_py.out

real	0m11.388s
user	0m11.374s
sys	0m0.012s
ubuntu@ubuntu:~/sbrt/examples$ ls -l fibo_py.out
-rw-r--r-- 1 ubuntu ubuntu 1000001 Oct  8 01:10 fibo_py.out
ubuntu@ubuntu:~/sbrt/examples$ 

Raspberry Pi 4B 4GB

Code: Select all

pi@RPi4B:~/sbrt/examples $ time ./fibo_fdi.py 4784969 > fibo_py.out

real	2m50.590s
user	2m50.486s
sys	0m0.060s
pi@RPi4B:~/sbrt/examples $ 
pi@RPi4B:~/sbrt/examples $ ls -l fibo_py.out
-rw-r--r-- 1 pi pi 1000001 Oct  7 18:23 fibo_py.out
pi@RPi4B:~/sbrt/examples $ 

Obviously the the python3 gmp2 module is not installed. It falls back to Python's internal BIGINT.

BTW, on my Desktop PC it runs about 3 times faster than on my RPi 4B.

[John_Spikowski]

You would think that Python would throw an error calling functions that don't exist. Auto fallback is news to me.

[gkreidl]

You would think that Python would throw an error calling functions that don't exist. Auto fallback is news to me.

All my fibo scripts offer different options. If the gmp2 module is found, it will be imported and used by default. If not (or the the -i options is set), it will fall back to using Python's built-in BIGINTs.

[John_Spikowski]

So what I'm hearing is Python's internal BIGINT is crap and GMP should be installed by default to make the language usable?

[ejolson]

So what I'm hearing is Python's internal BIGINT is crap and GMP should be installed by default to make the language usable?

So was Python's GMP support installed on the notebook but not the Pi, or was it not installed on either?

[Paeryn]

So what I'm hearing is Python's internal BIGINT is crap and GMP should be installed by default to make the language usable?

No, Python's multi-precision integers are fine for most cases, you probably won't notice much difference in performance to GMP's until you get to really big numbers. The biggest problem with Python's implementation is when converting them into strings as the base conversion scales badly. If you need performance out of your huge numbers then by all means use GMP to get it, but by not using GMP doesn't make Python unusable by any stretch of the imagination.

[gkreidl]

So what I'm hearing is Python's internal BIGINT is crap and GMP should be installed by default to make the language usable?

No, Python's multi-precision integers are fine for most cases, you probably won't notice much difference in performance to GMP's until you get to really big numbers. The biggest problem with Python's implementation is when converting them into strings as the base conversion scales badly. If you need performance out of your huge numbers then by all means use GMP to get it, but by not using GMP doesn't make Python unusable by any stretch of the imagination.

Yes, and the built-in BIGINTs may be even faster than using GMP2, if the numbers are not too big. Example:

Code: Select all

./fibo_rd.py -n --tt
(100, 3.504753112792969e-05, 6.985664367675781e-05, 0.5017064846416383)
(1000, 5.745887756347656e-05, 6.0558319091796875e-05, 0.9488188976377953)
(10000, 0.0002396106719970703, 0.0001621246337890625, 1.4779411764705883)
(100000, 0.00709986686706543, 0.0013675689697265625, 5.191596931659693)
(1000000, 0.2637336254119873, 0.025539636611938477, 10.326443927894625)
(4784969, 3.8424465656280518, 0.22975802421569824, 16.723884089553014)
(10000000, 10.25012469291687, 0.4478626251220703, 22.886760622463363)

The first number is the Fibo index, the second the time needed using BIGINTs, the third is the time using GMP2 and the last one is the quotient of 3 and 2 (= speed improvement). For fibonacci numbers up to fibo(10000) BIGINTs are faster or about the same.

[John_Spikowski]

So what I'm hearing is Python's internal BIGINT is crap and GMP should be installed by default to make the language usable?

So was Python's GMP support installed on the notebook but not the Pi, or was it not installed on either?

I ran Python out the box on the laptop and RPi4B.

Why is Python so slow on the RPi4B?

[jalih]

Lenovo laptop

Code: Select all

ubuntu@ubuntu:~/sbrt/examples$ time ./fibo_fdi.py 4784969 > fibo_py.out

real	0m11.388s
user	0m11.374s
sys	0m0.012s
ubuntu@ubuntu:~/sbrt/examples$ ls -l fibo_py.out
-rw-r--r-- 1 ubuntu ubuntu 1000001 Oct  8 01:10 fibo_py.out
ubuntu@ubuntu:~/sbrt/examples$ 

Raspberry Pi 4B 4GB

Code: Select all

pi@RPi4B:~/sbrt/examples $ time ./fibo_fdi.py 4784969 > fibo_py.out

real	2m50.590s
user	2m50.486s
sys	0m0.060s
pi@RPi4B:~/sbrt/examples $ 
pi@RPi4B:~/sbrt/examples $ ls -l fibo_py.out
-rw-r--r-- 1 pi pi 1000001 Oct  7 18:23 fibo_py.out
pi@RPi4B:~/sbrt/examples $ 

Obviously the the python3 gmp2 module is not installed. It falls back to Python's internal BIGINT.

BTW, on my Desktop PC it runs about 3 times faster than on my RPi 4B.

I think 11 secs on desktop PC is a long time...

My desktop PC runs the following 8th code much faster:

Code: Select all

\
\ Fibonacci with million digits
\
: bits?  \ n -- nbits-1
  n:ln 2 n:ln n:/ n:int ;

: fibo-loop
  >r                             \ Put loop counter on r-stack
  \ a b
  2dup 2 n:*                     \ a b a 2b
  over n:-                       \ a b a (2b-a)
  n:* -rot                       \ d=(2b-a)*a a b
  n:sqr swap n:sqr n:+           \ d=(2b-a)*a e=b*b+a*a
  r> r@ swap n:shr 1 n:band if
    dup  \ a b b
    rot  \ b b a
    n:+  \ b c=b+a
  then ;

: fibo  \  n -- fibo(n)
  >r    \ Put n on r-stack
  F0 F1 \ a b
  ' fibo-loop 1 r@ bits? loop-
  \ a b
  r> 1 n:band if
    n:sqr swap n:sqr n:+  \ b*b+a*a
  else
    2 n:* over n:- n:*    \ (2b-a)*a
  then ;

: app:main
  4784969 fibo
  "%.f\n" strfmt .
  bye ;

Result is:

Code: Select all

PS C:\temp\fibo> Measure-Command { 8th fibo3.8th >result.txt }


Days              : 0
Hours             : 0
Minutes           : 0
Seconds           : 0
Milliseconds      : 386
Ticks             : 3864625
TotalDays         : 4,47294560185185E-06
TotalHours        : 0,000107350694444444
TotalMinutes      : 0,00644104166666667
TotalSeconds      : 0,3864625
TotalMilliseconds : 386,4625

[ejolson]

So what I'm hearing is Python's internal BIGINT is crap and GMP should be installed by default to make the language usable?

So was Python's GMP support installed on the notebook but not the Pi, or was it not installed on either?

I ran Python out the box on the laptop and RPi4B.

Why is Python so slow on the RPi4B?

I don't know. It is amusing that the difference between the 4B and the new notebook computer is only 7.3 fold when running Script Basic but closer to 15 fold for Python 3.

Can you check which versions of Python 3 are being used on each computer?

[John_Spikowski]

Both the laptop and RPi are running 3.7.3.

I'm curious what other scripting engines / interpreters use the GMP library for their BIGINT support?

What slows down Python more, 32 bit or ARM?

[John_Spikowski]

Is Heater on vacation?

I would have thought he would be investigating and offered reasons for the wide gap by now.

 but by not using GMP doesn't make Python unusable by any stretch of the imagination.

Can we agree on bloated and partially crippled?

@ejolson was correct by saying the purpose of the 1 mil fibo was to flush out language deficiencies.

[bensimmo]

I thought this article goes quite well with the current topic title

Mainly the last part, "The hope is that this paper gets us to talk a bit more about how we treat and view software that we exchange back and forth," said Williams. "We somehow naively assume this stuff will work, being experimentalists who don't have a lot of background in computer science."

Chemists bitten by Python scripts: How different OSes produced different results during test number-crunching
https://www.theregister.co.uk/AMP/2019/ ... n_scripts/