User avatar
sakaki
Posts: 382
Joined: Sun Jul 16, 2017 1:11 pm

Re: Liberation through Computer Literacy

Fri Aug 30, 2019 11:10 am

Hello,

apologies for the slight OT, but the Python 64-bit performance discrepancy mentioned above caught my attention (and I observed it also, trying out selfgrams.py), so I decided to try some more detailed benchmarking, the results of which are reported below.

For the tests, I set up a Raspbian Buster system with a 64-bit kernel, but otherwise stock, on a 4GiB Pi4, 1.5GHz, performance CPU governor, Pimoroni fan shim (so no thermal throttling). I then ran the pyperformance benchmark suite in a clean virtualenv. Python v3.7.3 was used. This was the baseline.

I then ran the same test suite in:
  • a 32-bit armhf Debian Buster chroot (same 64-bit kernel, Raspbian host OS, physical machine), Python v3.7.3;
  • a 64-bit arm64 Debian Buster chroot (ditto);
  • a Gentoo64 v1.5.0 system booted under the same kernel, Python v3.7.3 (built under the stock (-O2 no-pgo) settings);
  • ditto (but with Python v3.7.3 built using -O3 and profile guided optimization in use (since Debian appear to use this now);
  • ditto (but with Python v3.7.4 built using -O3 and profile guided optimization).
I then normalized the reported runtime statistics for each sub-benchmark in the suite, so that 1.00 = the time taken by the 32-bit baseline [1], and then took the median [2] of the full suite's relative performance for each platform as an overall performance measure (lower is better).

The results are tabulated below. Very rough, and with the caveats that apply to any benchmarks, but in summary:
  • The Python interpreter (at least, v3.7.3) seems to run programs faster on average in 32-bit than 64-bit (whether Gentoo or Debian), by a significant margin.
  • Debian armhf (32-bit) is marginally faster than Raspbian 32-bit, on a median basis.
  • The stock (-O2, no pgo) Gentoo 64-bit v3.7.3 is significantly slower than Debian's 64-bit arm64 version... however
  • Once I turned on -O3 and pgo (which appears to be Debian's default build settings, and are also now mine for Python from the forthcoming v1.5.1 release onwards ^-^) Gentoo 64 marginally outperformed Debian 64 at v3.7.3 (although was still slower than both 32-bit variants tested).
  • The v3.7.4 Gentoo 64-bit Python loses some ground against v3.7.3, but still keeps up with Debian64 v3.7.3 (there are some apparent performance regressions in there, such as unpickle_list, which account for most of this).
Results:

Code: Select all

pyperformance benchmark, Pi4, common 64-bit kernel, fan shim, 1.5GHz performance governor

                         Raspbian     Debian     Debian     Gentoo     Gentoo     Gentoo
                           32-bit     32-bit     64-bit     64-bit     64-bit     64-bit
                            stock      armhf      stock      stock    -O3 pgo    -O3 pgo
             Benchmark     v3.7.3     v3.7.3     v3.7.3     v3.7.3     v3.7.3     v3.7.4
-----------------------------------------------------------------------------------------
 Median (lower=faster)       1.00       0.98       1.19       1.37       1.16       1.19
-----------------------------------------------------------------------------------------
                  2to3       1.00       1.00       1.23       1.34       1.18       1.20
             chameleon       1.00       0.93       1.12       1.32       1.04       1.05
                 chaos       1.00       0.94       1.26       1.45       1.21       1.22
          crypto_pyaes       1.00       0.97       1.23       1.41       1.13       1.20
             deltablue       1.00       1.03       1.26       1.45       1.25       1.26
       django_template       1.00       0.94       1.28       1.45       1.21       1.24
           dulwich_log       1.00       0.93       1.15       1.33       1.13       1.13
              fannkuch       1.00       1.08       1.17       1.29       1.06       1.07
                 float       1.00       1.02       1.16       1.45       1.20       1.26
           genshi_text       1.00       1.05       1.27       1.39       1.16       1.20
            genshi_xml       1.00       0.99       1.23       1.34       1.11       1.16
                    go       1.00       1.00       1.19       1.35       1.17       1.18
                hexiom       1.00       1.07       1.23       1.47       1.20       1.25
              html5lib       1.00       0.98       1.22       1.36       1.19       1.19
            json_dumps       1.00       0.89       1.14       1.26       1.02       1.04
            json_loads       1.00       0.94       1.09       1.24       0.92       0.93
        logging_format       1.00       0.92       1.19       1.35       1.14       1.14
        logging_silent       1.00       1.10       1.38       1.47       1.16       1.26
        logging_simple       1.00       0.93       1.19       1.35       1.14       1.12
                  mako       1.00       1.03       1.26       1.46       1.15       1.19
        meteor_contest       1.00       1.02       1.18       1.28       1.09       1.13
                 nbody       1.00       1.03       1.03       1.23       1.04       1.07
               nqueens       1.00       1.00       1.22       1.53       1.22       1.24
               pathlib       1.00       0.93       1.23       1.44       1.21       1.19
                pickle       1.00       0.86       1.08       1.22       0.95       0.97
           pickle_dict       1.00       1.03       1.09       1.46       1.09       1.10
           pickle_list       1.00       1.03       1.20       1.59       1.10       1.07
    pickle_pure_python       1.00       1.05       1.27       1.50       1.19       1.22
              pidigits       1.00       0.96       0.57       0.60       0.56       0.56
        python_startup       1.00       0.89       1.13       1.27       1.18       1.16
python_startup_no_site       1.00       0.88       1.09       1.25       1.18       1.14
              raytrace       1.00       0.98       1.21       1.47       1.24       1.26
         regex_compile       1.00       0.99       1.19       1.40       1.16       1.19
             regex_dna       1.00       1.12       1.00       0.99       0.92       0.84
          regex_effbot       1.00       1.22       1.03       1.04       0.94       0.96
              regex_v8       1.00       1.36       1.18       1.28       1.07       1.06
              richards       1.00       1.06       1.21       1.38       1.22       1.21
           scimark_fft       1.00       0.96       1.07       1.22       1.08       1.14
            scimark_lu       1.00       1.03       1.31       1.58       1.28       1.34
   scimark_monte_carlo       1.00       1.01       1.27       1.51       1.25       1.41
           scimark_sor       1.00       1.06       1.12       1.37       1.22       1.24
scimark_sparse_mat_mul       1.00       0.98       0.91       1.18       1.02       1.04
         spectral_norm       1.00       0.98       1.11       1.27       1.10       1.09
sqlalchemy_declarative       1.00       0.94       1.20       1.34       1.23       1.33
 sqlalchemy_imperative       1.00       0.90       1.20       1.37       1.24       1.54
          sqlite_synth       1.00       0.84       1.12       1.29       1.08       1.12
          sympy_expand       1.00       0.99       1.18       1.41       1.19       1.53
       sympy_integrate       1.00       0.96       1.23       1.45       1.21       1.36
             sympy_str       1.00       0.96       1.21       1.45       1.20       1.44
             sympy_sum       1.00       0.93       1.24       1.46       1.23       1.39
                 telco       1.00       0.85       1.28       1.40       0.98       1.06
          tornado_http       1.00       0.92       1.17       1.30       1.16       1.22
       unpack_sequence       1.00       1.05       0.96       0.99       0.98       0.95
              unpickle       1.00       0.97       1.12       1.32       1.04       1.52
         unpickle_list       1.00       1.15       1.28       1.38       1.17       2.25
  unpickle_pure_python       1.00       1.10       1.38       1.57       1.28       1.33
    xml_etree_generate       1.00       0.96       1.40       1.69       1.31       1.29
   xml_etree_iterparse       1.00       1.04       1.16       1.36       1.13       1.07
       xml_etree_parse       1.00       0.88       1.00       1.16       0.97       0.99
     xml_etree_process       1.00       0.96       1.39       1.58       1.29       1.28

Hope that is of some interest! On the basis of the above, I'd expect the performance gap for the Python programs in the chart below (which I copy again for ease of reference) to narrow significantly under the forthcoming v1.5.1 Gentoo 64 release, but still underperform the 32-bit Raspbian tests (in contrast to the Rust, most of the C/C++ tests, which do better under 64-bit). NB for avoidance of doubt this chart has not been updated using the -O3/pgo v3.7.3 or v3.7.4 64-bit Python builds yet.

Image

Best,

sakaki

[1] Oops, just noticed this is reversed to the "divide by 64-bit runtime" metric used in the chart. Apologies for any confusion!
[2] I guess taking logs might have been an idea first >< ... but don't think this will affect things too much.
Edit: just confirmed this: working with log relative performance gives the same comparative ranking:
  • Debian 32-bit armhf v3.7.3 (fastest)
  • Raspbian 32-bit v3.7.3
  • Gentoo 64-bit v3.7.3 -O3 pgo
  • Debian 64-bit v3.7.3 / Gentoo 64-bit v3.7.4 -O3 pgo (dead heat)
  • Gentoo 64-bit v3.7.3 -02 no-pgo (slowest)

ejolson
Posts: 3732
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Sat Aug 31, 2019 6:16 pm

sakaki wrote:
Fri Aug 30, 2019 11:10 am
[1] Oops, just noticed this is reversed to the "divide by 64-bit runtime" metric used in the chart. Apologies for any confusion!
[2] I guess taking logs might have been an idea first >< ... but don't think this will affect things too much.
Thanks for the comprehensive tests. I'll rerun the Python anagram programs after updating my Gentoo image.

The way you have normalised the 64-bit runtimes with respect to the baseline 32-bit Raspbian runtimes makes good sense. The reason to flip the ratios is to compare speeds rather than runtimes. Note that logarithmic coordinates make the increase in the length of the bar for a program that runs twice as fast compared to the baseline exactly the same as the decrease in the length of the bar for a program that runs half as fast compared to the baseline.

In terms of computer literacy, creating bar charts with logarithmic coordinates was done mostly because of the insane nature of the anagram challenge.

jdonald
Posts: 417
Joined: Fri Nov 03, 2017 4:36 pm

Re: Liberation through Computer Literacy

Sat Aug 31, 2019 10:38 pm

That's indeed a large difference that deserves investigation. There are further ways to ensure that the 64-bit native modules are properly optimized, but Debian armhf is already running with a handicap here in that it limits itself to Cortex-A9 compatible code.

I had a few hypotheses but this is the only one that's promising so far: Take a look at for example this CPython implementation code: https://github.com/python/cpython/blob/ ... .c#L16-L41

It appears the CPython codebase incorrectly uses long and unsigned long throughout. Everything including shorts and bytes may get converted to 64-bit long ints leading to wasted computation cycles, extra instructions, doubled register usage, and more cache capacity misses.

To investigate this theory, you could run the same pyperformance experiment comparing i386 to x86_64 on Linux and see if there's a correlation. Note it wouldn't match exactly because some benchmarks would be offset with the advantage of more registers in x86_64. At that point you could then run the same experiment comparing x86 to x64 on Windows and expect no correlation, because in Visual Studio long and unsigned long remain 32 bits regardless of Win32 vs x64.

User avatar
John_Spikowski
Posts: 1608
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sun Sep 01, 2019 11:03 am

because in Visual Studio long and unsigned long remain 32 bits regardless of Win32 vs x64.
Max string length doesn't improve going from 32 bit to 64 bit on Windows.

User avatar
sakaki
Posts: 382
Joined: Sun Jul 16, 2017 1:11 pm

Re: Liberation through Computer Literacy

Sun Sep 01, 2019 4:22 pm

Hello,

the v1.5.1 (bugfix) release of gentoo-on-rpi-64bit is now available for download; please see my main post here.

NB Python v3.7.4 is shipped with the image, but is (following upstream) not used as the default. To set it as such, once you have the image running issue:

Code: Select all

demouser@pi64 ~ sudo eselect python set python3.7
after which running "python" or "python3" should start up this version of the interpreter.

Best, sakaki

ejolson
Posts: 3732
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Sat Sep 07, 2019 6:44 am

sakaki wrote:
Sun Sep 01, 2019 4:22 pm
To set it as such, once you have the image running issue:

Code: Select all

demouser@pi64 ~ sudo eselect python set python3.7
I've downloaded the latest Gentoo 64-bit image and performed the eselect command to make the new Python interpreter active. After rerunning the insane Python anagram programs, the 64-bit performance ratios look like

Image

Although the 64-bit version of Python is still slower than the 32-bit version, it is not as slow as before. For the sake of computer literacy, it feels quite liberating to have a faster Python in the 64-bit Gentoo image!

User avatar
John_Spikowski
Posts: 1608
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sun Sep 08, 2019 11:18 am

What is the goal? Speed or reach?

ejolson
Posts: 3732
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Sun Sep 08, 2019 2:54 pm

John_Spikowski wrote:
Sun Sep 08, 2019 11:18 am
What is the goal? Speed or reach?
The goal is liberation through computer literacy.

Have you compared the speed of Script Basic running in 64-bit mode to 32-bit mode? Does it run faster or slower under the 64-bit Gentoo image?

User avatar
John_Spikowski
Posts: 1608
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Sun Sep 08, 2019 8:40 pm

Currently I'm wrapped up in a project using ODBC and ScriptBasic to integrate a custom manufacturing package with QuickBooks Enterprise. Win10 seems more refined than previous versions of Windows.

I hope to get back to getting the hash extension module working for the insane dictionary anagram challenge soon.
Have you compared the speed of Script Basic running in 64-bit mode to 32-bit mode? Does it run faster or slower under the 64-bit Gentoo image?
I only run 32 bit ScriptBasic on Windows and Raspberry Pi.

If I was after speed I would write my code in C. ScriptBasic is fast enough for most projects I do to earn a living.

Heater
Posts: 13704
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 6:21 am

My hat is off to you John. You are making a living using what is, by a wide margin, the slowest, most limited, programming language I have ever seen.

I don't mean that in a derogatory way. You are obviously doing something right. Whilst I struggle along trying to earn a crust of bread using modern tools with orders of magnitude better performance.

There must be a lesson to learn there.
Memory in C++ is a leaky abstraction .

User avatar
John_Spikowski
Posts: 1608
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 12:59 pm

The only thing slow about ScriptBasic is when you abuse its array feature. I don't waste away my life building frameworks with bloated scripting engines meant for a web browers.

Most of my clients needed the project done yesterday by the time I get the call. One aspect you forget is these clients don't have the budget for your over complicate solutions that cost as much to support as it did to initially write it.
Last edited by John_Spikowski on Mon Sep 09, 2019 1:13 pm, edited 1 time in total.

Heater
Posts: 13704
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 1:12 pm

Neither do I. I use node.js for that. I hate "frame works" of any kind. Although now I'm turning towards Rust where more speed is required.

I will never understand your attitude that using arrays as, well, arrays, is some how abusing them.
Memory in C++ is a leaky abstraction .

User avatar
John_Spikowski
Posts: 1608
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 1:18 pm

I'm using associative arrays to cache SQL SELECTs that reduces the time to process journal entry creation by 5X.

Your argument is nothing but noise.
Last edited by John_Spikowski on Mon Sep 09, 2019 1:31 pm, edited 1 time in total.

Heater
Posts: 13704
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 1:26 pm

Noise? No need to be rude about it John.

What you have there is linked lists or hash tables or whatever. Not arrays.

I'm not sure what caching SQL queries has got to do with anything. Quite likely using a hash table or whatever there is helpful though.
Memory in C++ is a leaky abstraction .

User avatar
John_Spikowski
Posts: 1608
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 1:34 pm

Rude is calling a language you never used crap.

I can actually make a profit with ScriptBasic.

I'll take any challenge you throw my way and finish it before you settle on a design.

Heater
Posts: 13704
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 2:12 pm

John,
Rude is calling a language you never used crap.
It would be. If I had done so.

As it happens, and as you know, I have used ScriptBasic. Not to mention other BASICs going back a long way.
I can actually make a profit with ScriptBasic.
Indeed. And I was commenting on how brilliant that is.
I'll take any challenge you throw my way and finish it before you settle on a design.
I don't doubt your programming skills.

We are however still waiting for reasonable solutions in ScriptBasic to the two simple challenges that we have discussed a lot here this year.

No hurry.
Memory in C++ is a leaky abstraction .

ejolson
Posts: 3732
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 2:29 pm

John_Spikowski wrote:
Mon Sep 09, 2019 1:34 pm
Rude is calling a language you never used crap.

I can actually make a profit with ScriptBasic.

I'll take any challenge you throw my way and finish it before you settle on a design.
It's pretty clear that Script Basic is useful. I think this is because Script Basic is a light-weight interpreter that has a relatively easy way to call C libraries through what are called extensions. Even if some of the extensions leak, the core interpreter appears solid and bug free.

Gone are the days where one general-purpose programming language is sufficient. Instead we use special-purpose tools for each different task to make it easier. While there are still some engineers who use C for everything (including text processing), the next generation uses special-purpose integrated development editors, special-purpose build systems and library managers which go with a different domain-specific programming language for every project.

Since the goal of this thread is liberation through computer literacy, exploring a wide variety of programming languages is essential. Since programming languages are people too, it is important to remember that having equal abilities is not a prerequisite for equal respect.
Last edited by ejolson on Mon Sep 09, 2019 2:43 pm, edited 6 times in total.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23981
Joined: Sat Jul 30, 2011 7:41 pm

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 2:32 pm

Message to John_Spikowski and heater....fancy some time in the cooler? You'll get it if things deteriorate into childish mudslinging.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Heater
Posts: 13704
Joined: Tue Jul 17, 2012 3:02 pm

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 3:02 pm

ejolson,

When were those days when one general-purpose programming language is sufficient?

I must have missed it, what with having to use so many different languages over the decades.

As our grand fathers used to say, and I often repeat "Horses for courses".
Memory in C++ is a leaky abstraction .

User avatar
Zebedeee
Posts: 5
Joined: Tue Jul 16, 2019 12:19 pm

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 3:33 pm

jamesh wrote:
Mon Sep 09, 2019 2:32 pm
Message to John_Spikowski and heater....fancy some time in the cooler? ...
You don't want a heater in your cooler... :roll: :lol:

User avatar
John_Spikowski
Posts: 1608
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 5:39 pm

Or at your party.
🦸‍

User avatar
John_Spikowski
Posts: 1608
Joined: Wed Apr 03, 2019 5:53 pm
Location: Anacortes, WA USA
Contact: Website Twitter

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 5:45 pm

It's pretty clear that Script Basic is useful. I think this is because Script Basic is a light-weight interpreter that has a relatively easy way to call C libraries through what are called extensions. Even if some of the extensions leak, the core interpreter appears solid and bug free.
It's a solid embeddable interpreter that hasn't exposed any limitations to me.

jalih
Posts: 94
Joined: Mon Apr 15, 2019 3:54 pm

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 6:38 pm

ejolson wrote:
Mon Sep 09, 2019 2:29 pm
Since the goal of this thread is liberation through computer literacy, exploring a wide variety of programming languages is essential. Since programming languages are people too, it is important to remember that having equal abilities is not a prerequisite for equal respect.
Yeah, why fight when you can power up Russian PDB-11 compatible UKNC and start calculating fibonacci numbers... :lol:
Attachments
uknc.jpg
uknc.jpg (143.85 KiB) Viewed 1335 times

ejolson
Posts: 3732
Joined: Tue Mar 18, 2014 11:47 am

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 7:38 pm

jalih wrote:
Mon Sep 09, 2019 6:38 pm
ejolson wrote:
Mon Sep 09, 2019 2:29 pm
Since the goal of this thread is liberation through computer literacy, exploring a wide variety of programming languages is essential. Since programming languages are people too, it is important to remember that having equal abilities is not a prerequisite for equal respect.
Yeah, why fight when you can power up Russian PDB-11 compatible UKNC and start calculating fibonacci numbers... :lol:
If you can boot Research Unix Release 7 on that computer, then it should be possible to calculate those Fibonacci numbers using the program at the link

https://www.raspberrypi.org/forums/view ... 0#p1458709

in the Why Avoid Basic thread.

As I write this I find it interesting to see how that previous thread has gathered 109367 views to date. It would appear there are many people as well as web-crawling robots with artificial intelligence that are interested to know what first programming languages provide a suitable context for a beginner to learn computer literacy.

From what I understand, the language question will be definitively answered when the lead developer of FidoBasic finally releases a working compiler. Unfortunately, Fido is currently coding a sluggish boot loader. Therefore, it seems best to continue writing programs using whatever tools are available with at least one eye watching for which programming languages are truly liberating.

User avatar
davidcoton
Posts: 4212
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK

Re: Liberation through Computer Literacy

Mon Sep 09, 2019 11:07 pm

John_Spikowski wrote:
Mon Sep 09, 2019 5:39 pm
Or at your party.
🦸‍
Heater is welcome at my party.
Signature retired

Return to “General programming discussion”