polas
Posts: 33
Joined: Tue Jan 15, 2013 9:52 am

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 11:03 am

Markodius wrote:Returned a value for every second elapsed. To what 'proper benchmarks' do you refer?
The problem is that it is all too easy to write some trivial code, pop some timing in and come up with some numbers which are entirely meaningless. Whilst this might not sound obvious I motivate it via an example; you could (very easily) write a loop, grab the time on each iteration and then every second write out the number of iterations completed. However making claims based upon this would problematic for two reasons, firstly because such a simple code bears no relevance to more complex ones (bottlenecks are often mathematical operations, larger memory creation, memory accesses, control flow transitions across functions/procedures which are not being tested sufficiently here) and secondly because it is very easy to perform loop optimisation over simple loops which, if it is being performed in BASIC, as Keith mentioned entirely nullifies the point about the OS having any impact.

There are lots of existing, peer reviewed, benchmarks (just google for them) which would help to make a fairer comparison. I quite like those at http://benchmarksgame.alioth.debian.org/ which you could take, implement a mathematically relatively simple one such as Mandlebrot in BBC BASIC (but base it on their existing code for it to be a fair test) and then compare against their Python version they supply running on the RPi. Of course to get a full picture you would need to implement a number of them in BASIC, but at least one would give us a general idea.

pygmy_giant
Posts: 1562
Joined: Sun Mar 04, 2012 12:49 am

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 5:59 pm

I disagree - some crude benchmarking is better than none - try converting this to Python:

Code: Select all

PRINT "The Ackermann test. Expect this to run in about"
PRINT "0.5 seconds on a 2GHz Intel CPU."
PRINT
PRINT "Start"
rec=0
start = TIME
FOR y = 0 TO 3
  FOR x = 0 TO 7
    z = FNackermann(y,x)
    PRINT z;  " ";
  NEXT
  PRINT
NEXT
PRINT "Done in ";  (TIME - start) / 100;  " seconds.":REM RISC OS TIME function counts in centiseconds
END

DEFFNackermann(m,n)
IF m = 0 THEN:= n+1
IF n = 0 THEN:= FNackermann(m-1,1)
 = FNackermann(m-1,FNackermann(m,n-1))

Please use floating point variables to make the comparison fair - it should display:

Code: Select all

The Ackermann test. Expect this to run in about
0.5 seconds on a 2GHz Intel CPU.

Start
         1          2          3          4          5          6          7          8
         2          3          4          5          6          7          8          9
         3          5          7          9         11         13         15         17
         5         13         29         61        125        253        509       1021
Done in 2.63 seconds.
*
Don't get too excited when it prints the first 3 lines in a second - it's the last line that'll take the time.

2.63 seconds is the time to beat using BBC BASIC under RISC OS on the Pi - no sneaky overclocking now....

NB. I nicked this code from Gordon / Burngate off of this thread : http://www.raspberrypi.org/phpBB3/viewt ... &start=575

;)

NigelJK
Posts: 65
Joined: Wed Sep 05, 2012 1:44 pm

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 6:10 pm

Oh that brought back some memories. We 'bench tested' Zenith XT clones vs BBC B's using BASICA vs BBCBasic. No guesses which won, but we were astounded that it was some 20X quicker.

DaveS - I love the sound of your multi-core experiment. There was something doing the rounds years ago which 'pre-empted' multi-core processors. It was a very small set of instructions which would allow the passing of transactions between processors, I think it was called TAOS or something.

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 6:25 pm

Interesting conversation

As to comparing the effect of the OS (and GUI) on the two platforms, would not it make the most since to use as close to identical as possible hand optimized assembly code on both platforms to benchmark this. First something that is not affected by IO such as 100000 iterations of Dhrystone benchmark to determine how much is lost just to task switching (making fure that it is a registered WIMP task, and calls Wimp_Poll once at a decent frequency [once every 20 iterations or so]), then simple console IO benchmarks, then bench the GUI.

Seems that this would better compare the effect of the OSes on performance.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

pygmy_giant
Posts: 1562
Joined: Sun Mar 04, 2012 12:49 am

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 6:41 pm

Perhaps from a technical point of view, but how would you interprate that in terms of end-user experience?

Not sure how you could otherwise do a quantafiable test that relates to the qualative experience of the end-user though. Maybe run multiple programs compiled with GCC on both problems simultaneously in several windows and time them?

From a testing languages perspective, one could run a number of the same Python programs under the two OSeses as
dboddie pointed out that Python does seem to be available for RISC OS:

https://www.riscosopen.org/forum/forums/11/topics/1341
http://www.cgransden.co.uk/python27.zip

although I have not tried it myself.
Last edited by pygmy_giant on Tue Jan 29, 2013 7:10 pm, edited 1 time in total.

NigelJK
Posts: 65
Joined: Wed Sep 05, 2012 1:44 pm

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 7:05 pm

I've been playing this game for years. I decided years ago that the only really effective 'test' is how much useful work can you get done. I'll leave to one side the constant 'fighting' with the OS to get anything done that some 'other' OS's seem to think is standard. Only today using Word 2010 I not only lost an important document after working for an hour on it, but it wiped out it's own recovery file also. Apparently there's a bug in pasting some text from a command prompt ...

pygmy_giant
Posts: 1562
Joined: Sun Mar 04, 2012 12:49 am

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 7:13 pm

I feel your pain - hasn't happened to me yet but thanks for the heads-up. I use Vista and had to system restore before I could even get on this web site!

nr.
Posts: 144
Joined: Wed Oct 03, 2012 8:51 am
Location: The Fens
Contact: Website

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 8:01 pm

pygmy_giant wrote: NB. I nicked this code from Gordon / Burngate off of this thread : http://www.raspberrypi.org/phpBB3/viewt ... &start=575
I was hanging around in IRC this afternoon when Gordon pointed me at that code and asked me to convert it to BBC BASIC and run it on RISC OS - perhaps we just all did the same thing at the same time :) For reference though, 2.74 seconds on my unclocked rev. 1 - and 2.3something when I converted it to integer maths.

But, to agree with NigelJK, the best OS for the job is the one that gets the job done the most effectively. It's like asking for the best thing to put on toast. Sometimes it's sardines, sometimes it's mushrooms, sometimes it's Marmite. That's just the way it is.
--
nr.

neilf
Posts: 72
Joined: Sun Nov 11, 2012 8:14 am

Re: RISC OS; Pros and Cons

Tue Jan 29, 2013 11:54 pm

pygmy_giant wrote:I disagree - some crude benchmarking is better than none - try converting this to Python:

Code: Select all

PRINT "The Ackermann test. Expect this to run in about"
PRINT "0.5 seconds on a 2GHz Intel CPU."
PRINT
PRINT "Start"
rec=0
start = TIME
FOR y = 0 TO 3
  FOR x = 0 TO 7
    z = FNackermann(y,x)
    PRINT z;  " ";
  NEXT
  PRINT
NEXT
PRINT "Done in ";  (TIME - start) / 100;  " seconds.":REM RISC OS TIME function counts in centiseconds
END

DEFFNackermann(m,n)
IF m = 0 THEN:= n+1
IF n = 0 THEN:= FNackermann(m-1,1)
 = FNackermann(m-1,FNackermann(m,n-1))

Please use floating point variables to make the comparison fair - it should display:

Code: Select all

The Ackermann test. Expect this to run in about
0.5 seconds on a 2GHz Intel CPU.

Start
         1          2          3          4          5          6          7          8
         2          3          4          5          6          7          8          9
         3          5          7          9         11         13         15         17
         5         13         29         61        125        253        509       1021
Done in 2.63 seconds.
*
Don't get too excited when it prints the first 3 lines in a second - it's the last line that'll take the time.

2.63 seconds is the time to beat using BBC BASIC under RISC OS on the Pi - no sneaky overclocking now....

NB. I nicked this code from Gordon / Burngate off of this thread : http://www.raspberrypi.org/phpBB3/viewt ... &start=575

;)
This doesn't compare apples with apples though. It compares print routines with print routines - which are normally nothing to do with the programming language itself. E.g. BBC BASIC V uses the OS print routine, over which it has no control. I daresay Python does the same. A more useful comparison would be to avoid printing to the screen until the respective language had finished its processing.

User avatar
rurwin
Forum Moderator
Forum Moderator
Posts: 4258
Joined: Mon Jan 09, 2012 3:16 pm
Contact: Website

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 12:06 am

If RiscOS uses significant resources to print four lines in two seconds, then it's lost the race. At those times, printing is insignificant. But if you must, take out the PRINT lines.

AMcS
Posts: 184
Joined: Sun Jan 06, 2013 11:23 am
Location: Dublin, Ireland

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 2:12 am

rurwin wrote:If RiscOS uses significant resources to print four lines in two seconds, then it's lost the race. At those times, printing is insignificant. But if you must, take out the PRINT lines.
Agreed, RISC OS doesn't use significant resources for printing in fact the bulk of the time in the Ackermann function is computational printing a few characters is a negligible added load. From experience BBC BASIC printing is pretty darned fast so leaving the PRINTs in or out is neither here nor there.

If - on the other hand - you wanted a bit more speed (and remembering this is an interpreter eliminating unnecessary spaces would probably help more (at the BASIC prompt ">" just type CRUNCH %11111). Ok it means the program is more difficult to read but should up the performance a tad ;-)

nr.
Posts: 144
Joined: Wed Oct 03, 2012 8:51 am
Location: The Fens
Contact: Website

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 8:29 am

rurwin wrote:If RiscOS uses significant resources to print four lines in two seconds, then it's lost the race. At those times, printing is insignificant. But if you must, take out the PRINT lines.
I did - it made no difference at all to the timing. Well, I think it dropped from 2.74 to 2.73 seconds or something. Certainly nothing that could be considered significant.
--
nr.

Steve Drain
Posts: 105
Joined: Tue Oct 30, 2012 2:08 pm
Location: Exeter UK

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 12:05 pm

AMcS wrote:If - on the other hand - you wanted a bit more speed (and remembering this is an interpreter eliminating unnecessary spaces would probably help more (at the BASIC prompt ">" just type CRUNCH %11111). Ok it means the program is more difficult to read but should up the performance a tad ;-)
BASIC V will do that crunch at runtime if the system variable BASIC$Crunch is set. Any value will do, so in a TaskWindow (Ctrl-F12) or the Command Shell (F12):

Code: Select all

Set BASIC$Crunch 1
BASIC VI, the 64-bit floating point version, always crunches. The anomalous behaviour of BASIC V dates back to the very earliest versions, which had a bug in the assember that could be induced by crunching. It is 25 years since then, but we still have to set that damned variable to get the automatic crunching.

If you were to want the best speed, there are crunching applications that will reduce names, join lines etc, a whole bunch of tricks. Unfortunately, I think the most comprehensive one, StrongBS, does not run on the current machines.

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 12:58 pm

Problem I have run the ackermann test in BBC BASIC and I am getting 1.82 seconds consistently.

I entered it as:

Code: Select all

PRINT"ACKERMANN TEST"
PRINT"START"
rec=0
S=TIME
FORY%=0TO3
 FORX%=0TO7
  Z%=FNA(Y%,X%)
  PRINTZ%;" ";
  NEXT
  PRINT
NEXT
PRINT"DONE IN ";(TIME-S)/100;" SECONDS"
DEFNA(M%,N%)
 IFM%=0THEN:=N%+1
 IFN%=0THEN:=FNA(M%-1,1)
 =FNA(M%-1,FNA(M%,N%-1)
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 1:13 pm

Or 1.88 seconds if I run it in a task window.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

User avatar
Burngate
Posts: 6091
Joined: Thu Sep 29, 2011 4:34 pm
Location: Berkshire UK Tralfamadore
Contact: Website

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 3:08 pm

DavidS wrote:

Code: Select all

 =FNA(M%-1,FNA(M%,N%-1)
Mismatched brackets

I don't know whether you need END after the main section and before DEFFN

Also, (sorry) rec=0 was added by me as a bit of a mod, but not removed when I took out the rest of the mod

neilf
Posts: 72
Joined: Sun Nov 11, 2012 8:14 am

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 3:12 pm

[quote="DavidS"]Problem I have run the ackermann test in BBC BASIC and I am getting 1.82 seconds consistently.

Probably because you're using integers instead of floating point.

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 3:47 pm

I am currently composing a benchmark series for RISC OS BBC BASIC V, and will be translating it into Charm, Squeak, C, and hand optimized Assembly. It is not a standard CPU benchmark, instead concentrating on those things that effect the user experience. Benchmark series will include:
Ackermann
For Loop Speed
While Loop Speed
Task Switch Speed
Until Loop Speed
Print to Console Speed
Draw Rectangles Speed
Draw Filled Rectangles Speed
Draw Ellipse Speed
Draw Filled Ellipse Speed
Draw Circle Speed
Draw Filled Ellipse Speed
Draw Sprite Speed
Window Open Speed
Window Close Speed
Window Stack Order Change Speed
Menu Open and Close Speed
Draw Sprite Icon To Window Speed
Draw Text Icon To Window Speed
Draw Rectangle To Window Speed
Draw Filled Rectangle To Window Speed
Draw Ellipse to Window Speed
Draw Filled Ellipse to Window Speed
Draw Circle to Window Speed
Draw Filled Circle To Window Speed
Memory Copy Speed
Load File Speed

If you can think of anything else that should be added PLEASE DO SAY.

@Burngate:
Sorry, I had miss typed in the post. I forgot the END, and the parentheses are matched in the version I run. I am using one computer for RISC OS, and another for forums (because a few forums require JS (thankfully not this one).

@neilf:
Yes I am using integers because there is no use for floating point with this. I have always done so, if you are doing integer math use integers it is faster, and automatic for me.
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

polas
Posts: 33
Joined: Tue Jan 15, 2013 9:52 am

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 4:45 pm

DavidS wrote:If you can think of anything else that should be added please do say.
If you are looking at benchmarking the underlying OS rather than the languages, which most of your suggestions seem to be aimed towards. Then it might be interesting to compare the scalability of, for example, the cost of opening windows (i.e. when lots and lots are already open) against Linux/other OS. I think in such a case then ROS has a good chance of coming out favourably on the RPi.

AMcS
Posts: 184
Joined: Sun Jan 06, 2013 11:23 am
Location: Dublin, Ireland

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 8:49 pm

Steve Drain wrote:BASIC V will do that crunch at runtime if the system variable BASIC$Crunch is set. Any value will do, so in a TaskWindow (Ctrl-F12) or the Command Shell (F12):

Code: Select all

Set BASIC$Crunch 1
Wasn't aware of that one Steve, thanks !

AMcS
Posts: 184
Joined: Sun Jan 06, 2013 11:23 am
Location: Dublin, Ireland

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 9:05 pm

@DavidS: In terms of testing I'd tentatively suggest adding a test for Saving data to secondary storage - and also something to test random seeks in a file open for read/write. It might also be informative when testing to test saving/writing files of different sizes (some filing systems handling lots of small files being written less efficiently than a few large file writes).

Markodius
Posts: 134
Joined: Fri Jan 04, 2013 11:14 pm

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 9:22 pm

Yeah, pygmy_giant! Something is better than nothing 8)

polas, yes - my benchmark is certainly a trivial computing task. In many ways, however, it's more revealing than an overly complicated one. The reason for that is that it deals only with integers, and a minimum of flow-control. Screen updates only occur once a second. That should be fast on just about anything!

Some interesting stuff. Are you overclocked DavidS? I got 2.01 seconds at stock speed with your code (although I had some stealth EVAL statements appearing as a result of the copy/paste).

I look forward to the benchmarks DavidS.
“In the modern age, to call a man unelectable means he cannot be bought”

User avatar
DavidS
Posts: 4334
Joined: Thu Dec 15, 2011 6:39 am
Location: USA
Contact: Website

Re: RISC OS; Pros and Cons

Wed Jan 30, 2013 11:20 pm

No not overclocked, actually under-clocked at 600MHz ARM. (Though RAM is overclocked to 500MHz I do not think this should effect this particular task).
RPi = The best ARM based RISC OS computer around
More than 95% of posts made from RISC OS on RPi 1B/1B+ computers. Most of the rest from RISC OS on RPi 2B/3B/3B+ computers

timrowledge
Posts: 1289
Joined: Mon Oct 29, 2012 8:12 pm
Location: Vancouver Island
Contact: Website

Re: RISC OS; Pros and Cons

Thu Jan 31, 2013 1:23 am

A quickly hacked out version of this takes 1.7sec on Squeak on my pi.
Making Smalltalk on ARM since 1986; making your Scratch better since 2012

Markodius
Posts: 134
Joined: Fri Jan 04, 2013 11:14 pm

Re: RISC OS; Pros and Cons

Thu Jan 31, 2013 1:38 am

Ahh.. curiouser and curiouser!

To be honest my experiments with overclocking have been periodically highly negative but if I go at stock speed she's rock steady under RiscOS or Raspbian. It also doesn't help that I'm writing and debugging whilst RiscOS is running.. more bugs than code really.

As to the 'feel' of RiscOS..

Image

It really does feel fast.
“In the modern age, to call a man unelectable means he cannot be bought”

Return to “RISCOS”