File and/or media server

[Bloodred]

I think a RasPi B model would make a perfect cheap, low-power home file server. I want to use one and a couple HDDs to set up a low-cost network storage solution. HDDs would obviously be USB or use external USB enclosures, I guess it depends on which ends up being cheaper.

I plan on installing SAMBA to share files with my Windows PCs and probably a FTP server as well. I'd also need some sort of efficient torrent client (rtorrent maybe?) that would be capable of downloading at high speeds on the RasPi. I wonder though if it would be capable of downloading torrents at about 10MB/s, since both the HDD and NIC would be using the same USB port on the CPU. That should still be less than half of the theoretical USB 2.0 bandwidth, so I hope it'll manage that without too much trouble. Any ideas if this should pose a problem or not?
Along all these things I'd also have to have some way to remotely access it, either by SSH (but then again I'm not very familiar to Linux) or by some flavor of VNC.
Something that could wake up other machines on my network remotely would also be pretty useful, but I don't know if anything like this has been made or not. If not, I think I'd be up to the task of writing it myself.

Other than that I was also thinking of running something like PS3 Media Server on it, but even if Java runs properly (afaik there's no official JVM, right?) it probably wouldn't have the hardware resources to handle 1080p live transcoding, worth a try though.

Anyway, I'm really excited to see the RasPi release and I'm sure I'll have lots of fun toying with it.

[obarthelemy]

My plan exactly, but with a Model B, so the HD and the 'net hang off different I/Os. At least I hope they do, and that the ethernet+2xUSB chip has enough internal banwidth to not clog things down too much.

[Bloodred]

I'll use a Model B too, but to my understanding the model B just has a chip connected via USB 2.0 to the ARM. This chip provides 2 more USB 2.0 ports and 1 internal USB 2.0 connection to the NIC, so the maximum bandwidth would still be the 400Mbps between the ARM and chip. I don't know any technical details beside this, so I don't know if active CPU intervention will be needed to transfer the actual data from the NIC to the HD through this chip. Maybe someone else can shed some light on the subject.

[jeicrash]

Maybe FreeNAS or like project will release an ARM distro. Or someone with enough time and kindness from the community may op to port it ?
I think a low powered nas server is the perfect use for this, I almost purchased a gumstix system years back to do this, but the price was just to much at that time. R-pi looks to be just what the geek-doc ordered for this though. The other thing would be maybe a stripped down FOGproject server?

[obarthelemy]

Oh, OK, too bad, I was thinking the Ethernet+2xUSB chip was using something better than USB as a bus to the CPU.

[Cafe]

Quote from Bloodred on August 20, 2011, 03:59
I'll use a Model B too, but to my understanding the model B just has a chip connected via USB 2.0 to the ARM. This chip provides 2 more USB 2.0 ports and 1 internal USB 2.0 connection to the NIC, so the maximum bandwidth would still be the 400Mbps between the ARM and chip. I don't know any technical details beside this, so I don't know if active CPU intervention will be needed to transfer the actual data from the NIC to the HD through this chip. Maybe someone else can shed some light on the subject.

This is a very good point. I have shared the same thought in the abstract. While I think it will be fun to use the Raspi for file/storage services, in its current form, it isn't the right choice for a speedy service. A plug computer such as Guruplug with eSATA and Gbit Ethernet maybe a better fit.

[jamesh]

As said above, 480Mbit/s is the limit for USB 2, but you won't get that in the real world. Let's say 250. And let's say that's shared between the network and a USB harddrive, 125 each? I use a wireless G network at home and can stream HD OK, that 52MBits maximum but I rarely get that. Although H264 1080P30 requires at most 10-15MBits for high quality.

So still quite a bit of spare bandwidth there.

Of course, wireless N or 1000T ethernet is a different kettle of fish, but the Raspi is still pretty capable.

[abishur]

I want to do this exact thing as well! One of the big things I'm looking for is a USB that can handle RAID 5. I know that Linux can emulate a RAID, but if my Linux install craps out on me, I don't think I could remake the RAID of a new install of Linux. But if I had an enclosure doing the work, all I'd need to do it buy the same enclosure again. I'd even be open to an enclosure that let you daisy chain drive to create a raid rather then one large enclosure that holds all the drives in it. If I find something that matches the bill I'll let you know!

[Lob0426]

The RasPi has enough power to do this just fine for a NAS. As an Internet file server you need even less. I have used USB attached drives, including a 4GB USB stick to serve files before. The Internet connections available in rural (max upload is only 2Mbps here in northern California) USA are the bottleneck, not the USB. With the Ethernet and USB using the same theoretical 480Mbps his estimate of throughput is pretty fair. Use shorter cords for USB to maximize performance. I recommend you buy some 18" (500mm) or shorter cords.

http://www.cyberguys.com/produ.....ctid=10027

They have them in Several different connector combos.

[Bloodred]

A Guruplug would definitely be much faster (since I do have a Gbit network) but in this application the speed wouldn't benefit me much. I only plan to use it for downloads and storage, so if it can handle 80-90Mbps downloads I'll be happy as can be. I have a 100Mbps internet connection so I wouldn't be getting more than that with a Gbit capable device anyway. These things considered I don't think a much more expensive Guruplug would be worth it, not to mention it's in lots of ways more limited than the RasPi (no multimedia, no GPIO, no video output of any kind).
Anyway the bandwidth should theoretically be enough for 10MB/s, that's 80Mbps coming in through the NIC and 80Mbps going to the HDD for a total of 160Mbps, which is way below the theoretical (and practical) maximum bandwidth USB can carry. This is still all just theory, can't wait to get my hands on a RasPi and actually try it out.

@jamesh Regarding the media server part, I wasn't really worried that network bandwidth wouldn't suffice, I was thinking the CPU may not have enough in it to do live 1080p transcoding and that even 256MB RAM (minus what the OS and other services use) wouldn't be enough for a large enough transcoding buffer. Of course if the GPU could be used in the transcoding process it may work. This would be really great if it were possible.

@abishur A RAID 5 enclosure would be nice, even though I haven't checked out prices I have a bad feeling that it may prove to be pretty expensive. If it costs too much you may just as well make an older desktop PC with a RAID-capable motherboard into a server.

@Lob0426 My internet connection provides 100/100Mbps though, though actual speeds are closer to 90/65 Mbps. Even so when I'm not home I probably won't be able to use a connection to match those numbers, so I'm not really worried about the FTP part being in any way bottlenecked.

[obarthelemy]

@abishur why raid 5 ? raid gives higher performance, higher capacity, and higher availability. For a home NAS, you don't really need performance (you need performance not to suck, but not to be up there), and you don't need high availability. As far as capacity is concerned, I find I'm fine with a single 3TB HD, and another one for backups, which are vastly more important than anything RAID offers, and RAID sure doesn't replace backups, and takes up a lot of space, power, and money.

[jamesh]

Quote from Bloodred on August 20, 2011, 17:16
@jamesh Regarding the media server part, I wasn't really worried that network bandwidth wouldn't suffice, I was thinking the CPU may not have enough in it to do live 1080p transcoding and that even 256MB RAM (minus what the OS and other services use) wouldn't be enough for a large enough transcoding buffer. Of course if the GPU could be used in the transcoding process it may work. This would be really great if it were possible.


The Arm11 doesn't have any part in the transcoding - that's all done by the GPU. It usually does either encode OR decode, not sure how well it performs trying to do both at the same time in real time - not a standard use case for mobile use! It can certainly do one OR the other without affecting the CPU overmuch.

[Bloodred]

I guess GPU transcoding (let's say for the sake of this discussion it can handle transcoding well) would require specialized software, wouldn't it? For instance PS3 Media Server, which I currently use on a Windows PC (though I'd be happy with anything that worked on a RasPi, doesn't have to be PS3MS) uses mencoder to do the actual trascoding but it runs on the CPU, not the GPU. I don't even think mencoder has GPU acceleration support for regular desktop GPUs. So does this mean the only hope for hardware transcoding is software made to work with the RasPi's GPU?

[ShiftPlusOne]

Quote from Bloodred on August 20, 2011, 21:23
I guess GPU transcoding (let's say for the sake of this discussion it can handle transcoding well) would require specialized software, wouldn't it? For instance PS3 Media Server, which I currently use on a Windows PC (though I'd be happy with anything that worked on a RasPi, doesn't have to be PS3MS) uses mencoder to do the actual trascoding but it runs on the CPU, not the GPU. I don't even think mencoder has GPU acceleration support for regular desktop GPUs. So does this mean the only hope for hardware transcoding is software made to work with the RasPi's GPU?

Great question

[abishur]

Quote from obarthelemy on August 20, 2011, 18:29
@abishur why raid 5 ? raid gives higher performance, higher capacity, and higher availability. For a home NAS, you don't really need performance (you need performance not to suck, but not to be up there), and you don't need high availability. As far as capacity is concerned, I find I'm fine with a single 3TB HD, and another one for backups, which are vastly more important than anything RAID offers, and RAID sure doesn't replace backups, and takes up a lot of space, power, and money.

Well I like RAID 5 for several reasons, but it really boils down to I'm planning on using it for an internet accessible archive for family photos/videos/misc. As such I need it to both have a large capacity, have data redundancy, and have high availability . Yes RAID 1 will provide for redundancy, but that means every time I want to expand my RAID I need to purchase 2 new hard drives, which as you stated is fine with you, but is not fine for me it's a tremendous waste of money for me, especially has the whole thing grows. With RAID 5 however, all I need to do is slap a new hard drive in there and it takes care of the rest (ah the beauty of RAID 5).

Also RAID 5 does replace backups, that's the whole point of RAID 5. When one hard drive goes down, you swap it out and then it rebuilds the hard drive based off the information in the other drives. I can start off doing this with three drives. This is only a single additional drive than what you're suggesting and a single drive doesn't take up a lot more space, power, or money. Especially when it's coupled with the r-pi. A good multidrive RAID 5 enclosure with the r-pi equals savings as I no longer need a power hog PC running all the time. Plus with RAID 5 I get the added benefit of getting more storage capacity than you do with just RAID 1.

[Lob0426]

abishur that was the beauty of the original WHS. It used drive extender technology to mount new drives into the DATA drive. It did not care if the drive was SATA, USB or IDE. It used them all in conjunction to create the DATA drive. But wouldn't you know microsoft, had something that worked, threw it out of the newer version of Windows Home Server! Vail only supports RAID. I would like to see Linux develop something similar to drive extender. Pop in a new drive answer one question, whether you want to extend the data drive or not, and its there. WHS had folder replication to protect your data. It knew the drives as seperate devices. Unlike RAID it does not care about the individual drive size. Too bad microsoft is stupid.

Can you software RAID through a USB hub? two RasPi's one to handle the array and one to handle the web interface? Though relatively expensive 2.5" drives could be powered from a powered USB hub, possibly the RasPi also, and all of it put into a project box.

[jamesgarner82]

I would use this device to record live video from my IP Camera network I would then be able to play the stored and recorded footage locally through the HDMI/Video Out.

....Who knows, some coding, a few AT Commands and a GSM/3G Modem later and it could SMS/MMS my mobile with any detected movement etc...
....as well as being a NAS, a networked media player and maybe even an actual IP Camera itself! It would be nice if it could do all of that but I suppose we'll just have to wait and see how hardy the CPU/GPU really is.

The scope for this device is pretty much endless and so buying a couple of each of the board versions is my intent so that I can start learning, development and deployment! I wonder how popular these little things will become in the future, I cant wait to see.

[abishur]

I've been able to software RAID any drive (be it internally connected or externally connected) in Windows, so I can't imaging Linux not being able to support it! One of the things I'm looking into is finding out if there exists a USB hardware RAID card. You connect the USB card to the USB port and the drives to the card. I've never heard of something like this... but maybe it's out there? At this rate, as much as I want RAID 5, cost limitations are going to force me into purchasing two large drives and mirroring it :/ It will keep my data redundant, but it's more costly in the long run....

[jamesh]

Quote from Bloodred on August 20, 2011, 17:16
A Guruplug would definitely be much faster (since I do have a Gbit network) but in this application the speed wouldn't benefit me much. I only plan to use it for downloads and storage, so if it can handle 80-90Mbps downloads I'll be happy as can be. I have a 100Mbps internet connection so I wouldn't be getting more than that with a Gbit capable device anyway. These things considered I don't think a much more expensive Guruplug would be worth it, not to mention it's in lots of ways more limited than the RasPi (no multimedia, no GPIO, no video output of any kind).
Anyway the bandwidth should theoretically be enough for 10MB/s, that's 80Mbps coming in through the NIC and 80Mbps going to the HDD for a total of 160Mbps, which is way below the theoretical (and practical) maximum bandwidth USB can carry. This is still all just theory, can't wait to get my hands on a RasPi and actually try it out.

@jamesh Regarding the media server part, I wasn't really worried that network bandwidth wouldn't suffice, I was thinking the CPU may not have enough in it to do live 1080p transcoding and that even 256MB RAM (minus what the OS and other services use) wouldn't be enough for a large enough transcoding buffer. Of course if the GPU could be used in the transcoding process it may work. This would be really great if it were possible.

@abishur A RAID 5 enclosure would be nice, even though I haven't checked out prices I have a bad feeling that it may prove to be pretty expensive. If it costs too much you may just as well make an older desktop PC with a RAID-capable motherboard into a server.

@Lob0426 My internet connection provides 100/100Mbps though, though actual speeds are closer to 90/65 Mbps. Even so when I'm not home I probably won't be able to use a connection to match those numbers, so I'm not really worried about the FTP part being in any way bottlenecked.

The GPU can do encode/decode 1080p at 30fps, but not at the same time (so you cannot take an incoming h264 stream and transcode it to a different format stream in real time at 1080p. You could at lower resolutions). The Arm wouldn't be used at all in the process.

[Lob0426]

You may have seen these articles already.
http://news.softpedia.com/news.....9976.shtml
http://bigbruin.com/reviews05/.....humbraid_1
http://cs.joensuu.fi/~mmeri/usbraid/
I have not been able to find a hub or card that supports RAID 5. The above links are software RAID with USB hubs. They are all in linux but only appear to be RAID 0/1 through software.

http://www.google.com/products.....GAQ8gIwAA#
Ran across this and it states you can build your own RAID. I guess you probably could using a router or switch and a spare RasPi. Not cheap though at $35 apiece.

No luck at finding any affordable 4 drive enclosure that supports RAID 5. All of them are in the $350 plus area.

[Bloodred]

Quote from jamesh on August 25, 2011, 10:35
The GPU can do encode/decode 1080p at 30fps, but not at the same time (so you cannot take an incoming h264 stream and transcode it to a different format stream in real time at 1080p. You could at lower resolutions). The Arm wouldn't be used at all in the process.


So, basically what I want to do is decode incoming h264, encode it to MPEG2 and send it on its way to my PS3. Do you think it's feasible to have the GPU handle one of the processes and the ARM the other? I kind of doubt the ARM would have the juice to do it live and without stutters or noticeable quality loss.

Then again, since the RasPi can play the videos itself I may not even need to stream to my console anymore, thus completely bypassing the issue.

[swaite]

Quote from abishur on August 20, 2011, 22:08

Also RAID 5 does replace backups, that's the whole point of RAID 5. When one hard drive goes down, you swap it out and then it rebuilds the hard drive based off the information in the other drives. I can start off doing this with three drives. This is only a single additional drive than what you're suggesting and a single drive doesn't take up a lot more space, power, or money. Especially when it's coupled with the r-pi. A good multidrive RAID 5 enclosure with the r-pi equals savings as I no longer need a power hog PC running all the time. Plus with RAID 5 I get the added benefit of getting more storage capacity than you do with just RAID 1.

No RAID replaces backups. RAID is meant to provide high availability, and the higher RAIDs like RAID5 just increase that availability.

Consider this: In a hardware RAID configuration (not applicable here, but the point should be made) the RAID controller could go haywire and take out the metadata on the disks, or could just plain die, and you've lost all of your data. In either soft or hard RAID, a power surge could knock out more drives than the array is capable of surviving, and you've lost all of your data. You could lose one disk, and then a second disk could die while you're still rebuilding the array, and you've lost all of your data. Different brands and manufacturing lots of drives can help mitigate the "OH CRAP MY DRIVES ARE DROPPING LIKE FLIES" factor, but risk is risk. Drives are inherently fallible.

It's often said in the IT industry - you want your data in 3 places: live data, hot backup, cold backup, one of which should be offsite. A RAID array only counts as 1 place.

These do look like they'll be great little devices for personal file servers, even with some USB drives slung off of them in a RAID array, or similar. I'll likely be picking up at least one for that purpose.

[Svartalf]

Quote from abishur on August 20, 2011, 17:03
I want to do this exact thing as well! One of the big things I'm looking for is a USB that can handle RAID 5. I know that Linux can emulate a RAID, but if my Linux install craps out on me, I don't think I could remake the RAID of a new install of Linux. But if I had an enclosure doing the work, all I'd need to do it buy the same enclosure again. I'd even be open to an enclosure that let you daisy chain drive to create a raid rather then one large enclosure that holds all the drives in it. If I find something that matches the bill I'll let you know!

You won't find one that does daisy-chain. You WILL find one that will allow you to do SATA drives within it's enclosure off to USB/eSATA:

http://www.newegg.com/Product/....._-16216004
http://www.enhance-tech.com/pr.....-raid.html

Honestly...unless you're tinkering with RAID as a classroom or personal learning exercise, or using it as a store for a Bramble, thinking RAID for an R-Pi is more than a bit of overkill...

[obarthelemy]

To chime in, yet again, on why raid is NOT a backup:
RAID does NOT protect your data in case of:
- data corruption/deletion. You make a mistake, an ex sabotages your files, a nephew hits delete a few times too many, you mess up after an all-nighter...
- viruses or grave bugs. Same as above
- equipment destruction: a fire, big power surge that fries everything, water, earthquake...
- equipment theft
RAID is a (sometimes) high performance, high availability, high capacity tool. It is NOT a backup, because backups are:
- offline
- offsite
- multiple
- tested
Guess what ? RAID is none of that.
I used to sell networks and servers. I've had clients actually go under because their whole records burned down with their office.. RAID but no bakcups, their choice. Also, RAID introduces the dreaded Single Failure Point: if you're doing hardware raid with a controller card, you'll pretty much need the exact same card if your first one goes south. Maybe not a biggie right now, but in a few years...

[Svartalf]

Let me also throw in a bit in the "RAID is NOT a Backup" ring here...

Major company providing telecom traffic monitor systems. Sophisticated distributed cluster machines running in ATCA blade server cages. They opted to use, at least initially, software RAID 1 on them to increase uptime. Heh... Opposite thinking. Between the issues with RAID1 and the issues brought about with the software itself, the faint differing geometries they had with disks, etc. it was...a disaster. One I ended up undoing for them as one of my first tasks working for them.

RAID0 is insane unless you're using it for a scratch volume or using SSDs for the backing store. In fact, the only good use for RAID0 is stacking SSDs together to increase capacity and speed from the disks.

RAID1 is worthless because you can't tell me which copy is valid, even for uptime purposes.

RAID5 is sort of useful- because you can assure some semblance of uptime with it. You can reasonably determine what is the right data through at least one soft/hard read/write failure of the disks. It'll buy you some insurance for uptime. It is NOT a backup though- lose a disk in the array, and you're at risk of losing EVERYTHING until you restore the entire integrity of the array. Where you have one failure, you're liable, statistically speaking, to have more than one do it- lose two disks and it's likely to be gone.