obarthelemy
Posts: 1399
Joined: Tue Aug 09, 2011 10:53 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Fri Oct 21, 2011 11:28 am

The issues with clusters are
1- either you parallelize your workload when writing the software, or you count on the OS to do it. In one case, you've got to write everything anew, quite possibly in specialized languages to make it easier. In the other case, you need a cluster-aware OS, and to trust it to do a good job parallelizing your stuff.
2- The different processes on different CPUs need to communicate and synchronize. In the case of a multicore CPU, this happens in-chip, very fast, both in terms of transfer speed and latency. On a Pi, going through USB, speed will be quite low and latency quite high.
3- cluster does not imply fault-tolerance. If you want that too, you have to devote significant resources to it, and avoid some architectures (for example, a hierarchical one with a dispatch controller farming out tasks to other CPUs wouldn't be resilient to that dispatcher failing... but if you go peer-to-peer, synchronization messages grow exponentially with the number of nodes).

Obviously, those are oversimplifications. What I'm trying to convey is that a cluster of Pis is probably not worth it for any real-world workload (neither in terms of performance, effort involved, reliability....), except maybe if you manage to tap the GPU.

On the other hand, Pis make an excellent platform to play around with clusters, not least because they exacerbate all issues.

Scribe
Posts: 98
Joined: Wed Sep 28, 2011 12:00 pm
Contact: Website

Re: Cluster (Bramble...) Design Discussion (Advanced)

Fri Oct 21, 2011 11:51 am

All cluster issues are easy to workaround provided you're willing to adopt an appropriate programming framework and methodology.

OFC seeing a reasonable performance from the Pi as obarthelemy says is unlikely unless you tap the gpu and even then, it would probably be more effective to attempt to interface an arm core with an NVIDIA graphics card (hasn't thought this through but theoretically high performance). Doesn't that new ARM Cortex-A7 look nice and impressive and affordable? :P

nvoid82
Posts: 18
Joined: Tue Oct 25, 2011 3:07 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Oct 25, 2011 4:18 pm

Would a Bramble be a good idea to use for a blender render farm?

tufty
Posts: 1456
Joined: Sun Sep 11, 2011 2:32 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Oct 25, 2011 7:04 pm

All my blender needs is a 240v socket. It renders everything down to its component parts quite rapidly without computer assistance. My guess, though, is that a Bramble would blend less rapidly than a single Pi.

Bakul Shah
Posts: 321
Joined: Sun Sep 25, 2011 1:25 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Oct 25, 2011 7:27 pm

Check out Ron Minnich's cluster. 196 gumstix overos. 7 each attached to a gumstix stagecoach. the stagecoach has an 8 port 100MbE switch. 7 to connect the overos. 7 such assemblies in a 'shelf'. Each shelf has an 8 port switch. 7 go to the stagecoaches. In the end only one external connection. Requires 80A @ 5V per shelf and as per Ron wiring and assembly was rather painful. Runs plan9. No idea about performance.

A RasPi bramble would be even more painful as it has connections on all four sides and I get the feeling each Raspi board is bigger than an overo. Power will be somewhat less of a concern.

Other than as an academic exercise can't think of any application (well... may be a wirespeed router but for that you really need two ports per board. 256MB may be enough buffering....). But I am sure my failure of imagination would be cured if I had access to one :-)

Svartalf
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Oct 25, 2011 9:25 pm


Other than as an academic exercise can't think of any application (well... may be a wirespeed router but for that you really need two ports per board. 256MB may be enough buffering....). But I am sure my failure of imagination would be cured if I had access to one :-)

Heh... It's more of an academic exercise to see what it would take to DO one with them and documenting all the effort for the purposes of providing a base curriculum for CompSci on the subject...

....Well...that and doing it because we can... ;)

Svartalf
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Oct 25, 2011 9:27 pm

Quote from nvoid82 on October 25, 2011, 17:18
Would a Bramble be a good idea to use for a blender render farm?

Depends. From the perspective of the FP performance of the ARM11...probably wouldn't be except from a learning how things get done perspective. Now, if you could come up with something GPGPU-ish that would work with Blender, perhaps...

Svartalf
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Oct 25, 2011 9:29 pm

Quote from Scribe on October 21, 2011, 12:51
OFC seeing a reasonable performance from the Pi as obarthelemy says is unlikely unless you tap the gpu and even then, it would probably be more effective to attempt to interface an arm core with an NVIDIA graphics card (hasn't thought this through but theoretically high performance).

Heh... You're describing the story that's presented by a Tegra 2/3 right now. ;)

Bakul Shah
Posts: 321
Joined: Sun Sep 25, 2011 1:25 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Fri Oct 28, 2011 9:00 am

Here is a potential real application of a bramble: A giant billboard made from multiple displays. Of course, the displays will cost ridiculously more than the RasPis so may be you can dedicate N RasPis per display. Put tiny cameras at corners for interactivity. Create an edge to edge crisscross pattern of infrared beams & sensors and may you can do something like a giant `touch' sensitive display (when you break some beams by touching).

slacer
Posts: 32
Joined: Mon Dec 26, 2011 9:13 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Mon Dec 26, 2011 12:12 pm

Just wondering what has happened to this thread...

I am interested in this project, too. Just can't wait to get at least 4 of these devices and start to build my own cluster.

Dioxin
Posts: 5
Joined: Fri Dec 30, 2011 12:43 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Fri Dec 30, 2011 1:55 pm

Extending the Idea of the bramble a little further I have a couple of questions.

1. Is it possible to network boot a R-pi?

2. If the above is feasible, how feasible would it be to produce a Bramble on a board?

I've had the idea for a while of some sort of configurable 4-node cluster, configurable dependent on the network boot image used. I've most of the hardware, but not the time or expertise to get much further.

Details [spoiler] 4 x 2500k i5's on Mini-Itx boards with 8Gbs RAM, hooked up via a 5 port gigabit switch, all sharing the same PSU, only having the 5th port exposed outside the unit.

Intention would be to have some Master connected via the LAN to provide the boot images and the work units, easy to switch between different protocols just by changing the boot img.

Could also add and remove any number of my 4-node unit easily[/spoiler]

Is it possible to do something like this at the board level? 4 cut-down R-Pi's attached to a network switch chip all on the same board. Power in/network out?

There would be no requirement (in my own head) for any other outputs which should cut down on the board real estate back to credit card size (with a little magic ofc).

Pi in the sky idea perhaps

KuroSaru
Posts: 3
Joined: Mon Dec 19, 2011 7:15 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sat Dec 31, 2011 1:25 am

I love name bramble for the R-Pi cluster.

Im not sure if this is of any help to anyone but my plan is to create a small R-Pi cluster that will run Hadoop, Hive, HBase. So when doing work I can avoid a noisy server room.

This is a basic idea what I'm thinking.

Custom cases.

Power/Ethernet Box:

- 8 port switch

- 7 port powered USB hub

power from each USB port on the hub wired into the 4,5,7,8 wires/pins of each Ethernet port, apart from the last two Ethernet ports which will be outside link as such, and to allow for daisy chaining them.

R-Pi Box:

- mounting for R-Pi

- ethernet plug with micro-usb & ethernet wire coming off (PoE Adaptar)

- 2x LEDs (hope to connect via GPIO, power on led and network traffic)

Hopefully this is possible, as it would make wiring lots together quite a snap.

---------

7 Port - Powered USB Hub (type 1):

http://www.ebuyer.com/259358-x.....ed-n-uh702

Price: £5.99

7 Port - Powered USB Hub  - with switch (type 2):

http://www.ebuyer.com/279680-x.....ed-n-uh720

Price:£4.98

--

8 Port - 10/100 Ethernet Switch:

http://www.ebuyer.com/38043-dy.....-sw80010-m

Price: £10.00

----------

It be nice to know if i can network boot them, as that would be the most ideal method, have one R-Pi as master with netboot image, DHCP installed, Hadoop NameNode.

Then have a slave image that contains hadoop datanode, hive etc.. installed.

This would make adding a new node to the cluster as simple as use PoE adapter and connect to the network via one of the Power/Ethernet boxes.

74andy74
Posts: 1
Joined: Sun Jan 01, 2012 10:24 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 11:19 am

Hi all. New to the board.

The pi seems really interesting, especially if you can cluster them. I have no real experience and have only recently gotten a few servers at home and am learning daily.
When i read about clusters, i cant help but imagine 4 pi boards with a small 5 port switch that all fits in a regular 5 1/4 bay in a normal pc or server chassis. Possibly drawing power internally from the normal sized pc in the chassis. This could then be expanded by adding a second 5 1/4 sized cluster and so on until you end up with one ordinary pc and as many clusters as you have drivebays until you are satisfied. The normal pc would be the master. Just a thought but i cant wait to learn more and give it a try.

Cheers

pishboy
Posts: 19
Joined: Sun Jan 01, 2012 7:11 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 12:27 pm

Once i get a Raspi board i'll probably try to join in one of the contests (mentioned by Liz in another forum thread) and if i ever win a cash prize (once again, mentioned by Liz) i'd buy more raspis
And try to make my own bramble

Hiro
Posts: 78
Joined: Fri Dec 30, 2011 11:09 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 12:31 pm

Svartalf said:



Other than as an academic exercise can"t think of any application (well... may be a wirespeed router but for that you really need two ports per board. 256MB may be enough buffering....). But I am sure my failure of imagination would be cured if I had access to one


Heh... It"s more of an academic exercise to see what it would take to DO one with them and documenting all the effort for the purposes of providing a base curriculum for CompSci on the subject...

....Well...that and doing it because we can...


I couldn't agree more "Because we can" is why most of the things we will do on the Pi will be done. I have no purpose for a bramble but still fancy the challenge of making one work. The learning curve may be steep but we are up to the challenge, i think.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23870
Joined: Sat Jul 30, 2011 7:41 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 12:45 pm

Lets say you build a machine using multiple Raspi, get all the comms worked out and can perform distributed computting tasks. You have learn a A LOT.

But its not that powerful considering the cash outlay even if you can use the GPU.

Then someone comes out with a Raspi like board with 4 times the Arm power, you can upgrade, and suddenly all that work put in starts to pay off - you now have a much more powerful cluster.

Then it happens again.

And again.

Within a few SoC iterations you have a very powerful desktop cluster.

And all your software works brilliantly, because it was developed on a cheap as chips Raspi system.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

pishboy
Posts: 19
Joined: Sun Jan 01, 2012 7:11 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 1:07 pm

Just realized what a raspi cluster can be used for

A few years ago i have tried compiling WRF but the models ran too slow on a 1GHz pc of mine. Maybe a mini raspi cluster with 5 units could do the trick

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23870
Joined: Sat Jul 30, 2011 7:41 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 1:12 pm

Distributed compilation would a good thing to try to start with. Would teach you quite a bit of scripting, ethernet etc.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

pishboy
Posts: 19
Joined: Sun Jan 01, 2012 7:11 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 1:51 pm

JamesH said:


Distributed compilation would a good thing to try to start with. Would teach you quite a bit of scripting, ethernet etc.


Exactly what i intend doing with bramble

Just a tip: Stop thinking of having raspi for kids who don't have access to computers, it also has alot of uses for kids who do. Like me. I'm not planning to buy raspi for hobby use (well, it's my secondary reason) but i'm planning to buy it because i want to have experience working with embedded computers.
FYI i'm 14 y/o

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23870
Joined: Sat Jul 30, 2011 7:41 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 1:56 pm

pishboy said:


JamesH said:


Distributed compilation would a good thing to try to start with. Would teach you quite a bit of scripting, ethernet etc.


Exactly what i intend doing with bramble

Just a tip: Stop thinking of having raspi for kids who don't have access to computers, it also has alot of uses for kids who do. Like me. I'm not planning to buy raspi for hobby use (well, it's my secondary reason) but i'm planning to buy it because i want to have experience working with embedded computers.
FYI i'm 14 y/o


Don't worry, the primary purpose of the Raspi is to get more people programming, whether they already have existing access or not!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

david13lt
Posts: 20
Joined: Thu Sep 08, 2011 4:55 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 2:23 pm

Well, it would be nice to run N number of virtual machines based on Linux on R-P. Someone did that with Gumstixs to create a virtual network of 1 million [virtual machines] machines for network analysis or something.

pishboy
Posts: 19
Joined: Sun Jan 01, 2012 7:11 am

Re: Cluster (Bramble...) Design Discussion (Advanced)

Sun Jan 01, 2012 2:46 pm

I have an idea: The bramble can be mounted on a rack-style case because all you need to have access to is the front and rear, the sides can be used for troubleshooting and testing

and 40mm fans attached to the side can be used to cool it if you are planning to run it for weeks on end

EDIT: just read archived blog posts, turns out RASPI runs only warm to the touch. scrap the fan idea

Did some calculations on how much a extremely basic setup would be with most of the products coming locally here in the philippines (except the raspi, of course) and 5 raspis with ethernet cables, ethernet switch, case (rough estimate), 4 2gb SD cards (nodes) and 1 16gb SD (main) would set me back Php10500, or around USD250. Dunno if it sounds cheap but php10k isn't cheap here. Will definitely try it with less raspis for experience

zenith
Posts: 1
Joined: Tue Jan 03, 2012 12:29 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Jan 03, 2012 1:24 pm

Hello Just throwing this into the mix.

Would it be possible to use the SD card interface to create a very fast bus between two Raspberry PI's?

Thus, two RPI's could communicate and share their workload via the high speed SD card interface. These could then be connected in a star topology to other dual-RPI pairs via USB 2?

I'm not sure whether you could connect the small clusters together with a pair of RPIs that would be bridged using the SD card connection method.

Isn't 100mbps ethernet the slowest (albeit most "out-of-the-box user friendly") interface to use for building a multi RPI multiprocessing cluster?

Like I said, just an idea

Dioxin
Posts: 5
Joined: Fri Dec 30, 2011 12:43 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Jan 03, 2012 1:47 pm

unless I've very much mistaken the issue here is that you cant PUSH from one side.

A request has to be made to the SD interface to illicit a response, so the bandwidth might be severely reduced when you add the requirement for additional message passing.

(plus R-Pi's aren't high performance so the effort might be in vain)

ethernet works well enough not to hack around.

DoctorRad
Posts: 1
Joined: Tue Jan 03, 2012 2:05 pm

Re: Cluster (Bramble...) Design Discussion (Advanced)

Tue Jan 03, 2012 2:26 pm

Interesting discussion. I crunched some numbers over the weekend, comparing the RPi Model B specs with a dual-core Atom Mini-ITX motherboard:




 

RPi Model B



Jetway JNC96-525





Volume (cc)



92.41376



1156





Power (W)



3.5



30





DMIPS



965



8964





Coremarks



1605



9076.68





RAM (MB)



256



1024





Cost (GB£)



22



112.5





DMIPS / W



275.7143



298.8





DMIPS / cc



10.44217



7.75432526





DMIPS / £



43.86364



79.68






The RPi is one-third better than the Jetway on DMIPS per unit volume, but this comes at a price: almost twice as expensive per DMIP. Plus, the Jetway has gigabit Ethernet.

The newer generation of "Thin Mini-ITX" motherboards - 20mm height rather than the 40mm assumed here - would effectively double the DMIPS per unit volume of Mini-ITX boards. Not sure they would be fanless, though.

Return to “Other projects”