pauljlamp
Posts: 5
Joined: Tue Jun 26, 2018 6:42 pm

Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Tue Jun 26, 2018 7:20 pm

Hello all,

I am a relatively new Raspberry Pi user, following this tutorial, using a Raspberry Pi Zero W and Raspbian Stretch.
https://www.raspberrypi.org/magpi/pi-ze ... ash-drive/

It seems to be working fine. I didn't complete steps 13 and 14 because they don't apply to my project, but I do understand them. An important difference with my project is that the device that the Raspberry Pi is plugged into, is making changes to the files, instead of being read only. For this example, I have it plugged into my phone and then copy pictures on my phone to the "flash drive".

However, the Samba network share does not update with new changes unless I mount and and un-mount the "/mnt/usb_share" directory (using mount and umount). Rebooting also works, probably because everything is unmounted and mounted during the process.

Nothing else I have tried succeeds in getting the new changes to refresh in the network share. Originally I spent a lot of time thinking the problem was Samba, so I added options in the smb.conf file (such as oplocks=no, and csc policy=disable) to troubleshoot, as well as rebooting services related to Samba (smbd.service, nmbd.service, winbind.service) after making a change to the "flash drive" from my phone.

Code: Select all

sudo modprobe g_mass_storage file=/piusb.bin stall=0 ro=0 removable=1
sudo modprobe -r g_mass_storage
These lines, used to activate and deactivate the usb mode of the Raspberry, did not work either, the network share did not update with new changes. The program written for the tutorial in step 14, http://rpf.io/usbzw, primarily uses these commands, but I wasn't having any luck getting them to refresh the network share.

As I stated before, the only action that succeeded was mounting and un-mounting the directory in question, with "mount" and "umount". I looked into the documentation for these commands to figure out what they were doing that successfully made the changes appear, but all I could really determine was that some level of syncing was occurring before the un-mount took place. However, the actual "sync" command also failed to refresh the network share.

I don't really want to be constantly dismounting and mounting the directory in use, I would rather just figure out how exactly those two commands are succeeding in syncing the files I have changed. And that is the core of my question, why does "sync" not work in this scenario, and why do "umount" and "mount" work?

Thank you for your help. I've tried to be detailed in my post, but please let me know if you need any more information.

User avatar
thagrol
Posts: 1851
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Tue Jun 26, 2018 9:59 pm

It's a limitation of the mass_storage gadget.

There is no notification from one side to the other when a change takes place so the only way to update the side that didn't make the change is to unmount and remount the drive.

It also means that it's a very bad idea to have write access from both sides at the same time. And potentially problematic to have it from one side only.

Both sides (Pi and USB host) expect low level access to the storage and both use caching so you can run into all sorts of issues when changes are made.

That tutorial is unsafe if changes are being made from both sides and insecure, though that may not matter to you. It also has the potential for being extremely annoying (someone updates the storage when you're watching a movie, or worse deletes the file you're watching).

It's insecure as unseen changes can result it deleted data being visible to the other side of the link, it's dangerous as you could get random data passed to your output device with the possibility of damaging it.

That turorial has some basic protection as it exports the storage passed to the gadget as read only so as to prevent changes from that end. To duplicate that you'll ned to mount the storage on the Pi as readonly. Modifying that tutorial gives an fstab entry like this:

Code: Select all

/piusb.bin /mnt/usb_share vfat ro,users,umask=222 0 2
Unfortunately, unless the USB host can monitor for changes to the filesystem and then run scripts on the Pi you can't get automatic updates(via umount/mount) on that end.
This space unintentionally left blank.

User avatar
thagrol
Posts: 1851
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Tue Jun 26, 2018 10:03 pm

pauljlamp wrote:
Tue Jun 26, 2018 7:20 pm
And that is the core of my question, why does "sync" not work in this scenario
Because the linux filesystem drives don't know about the changes made by the USB host so don't have anything in their buffers to sync.
and why do "umount" and "mount" work?
Because umount then mount forces the drive/file/partition to be reread.
This space unintentionally left blank.

pauljlamp
Posts: 5
Joined: Tue Jun 26, 2018 6:42 pm

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Tue Jun 26, 2018 11:58 pm

Okay then... thank you. That was very helpful and I appreciate the response.

My current goal is to allow changes from the USB end only. However, I would like to see and likely copy the files from the network share, at a semi-regular interval. So the USB side (the device that is using the Raspberry Pi like flash drive) is the side that needs write access.

Can I make it only read only from the Pi side, and will that put me in a better position as far as security/safety? Can I copy a file from a read only data source? Is that the purpose of the code you posted?:

Code: Select all

/piusb.bin /mnt/usb_share vfat ro,users,umask=222 0 2
Do you have any other suggestion or resources I might be able to look at it in order to improve the security of the process? Unfortunately the USB host is going to be somewhat dumb, and won't be able to run scripts on the Pi. I was considering just running umount/mount at an interval, and I think the intended host for the device will not try to write unless it detects a connected USB.

Thanks again, you've already been very informative.

User avatar
thagrol
Posts: 1851
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Wed Jun 27, 2018 12:53 pm

Yep, that code snippet is to make the Pi treat the USB gadget storage as read only. "ro" in the mount options tells the Pi to mount it read only. "umask=222" sets permission so that write access on files and directories is not present.

You can copy files from read only mounts etc, you just can't write to them.

Unmounting and re-mounting the storage on the linux (pi) side won't affect the USB host, nor will copying files from it. But fiddling with the g_mass_storage module will.

The danger is that when copying files you may get incomplete or wrong data if files are being written at the same time as the copy. Or if the SD card/USB host is doing any housekeeping.

The safest way is to only have the storage mounted on one side at a time. However with a dumb host there is no way to do this automatically, or at least no way without risking corruption. The host may be writing to it when you remove it. Think of it as yanking a USB stick midway through a write.

If you've looked at their script you'll have seen that it loads and unloads the g_mass_storage module to force updates. You don't need to do this as you won't be writing to it from the linux side. But if you want to remove it when copying files, there is a better way:

Load g_mass_storage like this:

Code: Select all

modprobe g_mass_storage file=/piusb.bin stall=0 ro=1 removable=1
This makes it appear to the host as a device with removable storage (think CD-ROM).
To remove storage from the host:

Code: Select all

sudo echo "" > /sys/devices/platform/soc/20980000.usb/gadget/lun0/file
To give it storage:

Code: Select all

sudo echo "/piusb.bin" > /sys/devices/platform/soc/20980000.usb/gadget/lun0/file
Check those paths though. They may be different on a Zero or ZeroW and I can't remember which I wrotre that on.
This space unintentionally left blank.

pauljlamp
Posts: 5
Joined: Tue Jun 26, 2018 6:42 pm

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Wed Jun 27, 2018 2:01 pm

Okay, I'm learning a lot. Your reply answered my questions pretty succinctly.

However, I'm not sure what the purpose of unloading and loading using these commands:

Code: Select all

sudo echo "" > /sys/devices/platform/soc/20980000.usb/gadget/lun0/file

Code: Select all

sudo echo "/piusb.bin" > /sys/devices/platform/soc/20980000.usb/gadget/lun0/file
I checked the paths, and they appear to correct... as in, the files do exist, I'm not sure if they are right ones. When I attempt to run the command, I get a "permission denied". Do you know why I'm getting this error, and can you further explain what is the purpose/benefit of using these commands?

Thank you.

User avatar
thagrol
Posts: 1851
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Wed Jun 27, 2018 3:08 pm

The purpose of those two commands is to prevent the USB host writing to the storage whie you're copying from it.

The first one tells g_mass_storage that it no longer has somewhere to read from/write to so the USB host sees an empty drive. Think of it as pressing eject on a USB DVD drive and removing the disk.

The second one tells it where its storage is. Think of it as inserting a DVD and closing the drive tray.

Not sure why you're getting a permission denied error but:
  • Did you run those commands as root or with sudo?
  • Did you load g_mass_storage with removable=1?
  • If you had previously loaded g_mass_storage did you remove it first?

    Code: Select all

    sudo rmmod g_mass_storage
This space unintentionally left blank.

pauljlamp
Posts: 5
Joined: Tue Jun 26, 2018 6:42 pm

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Wed Jun 27, 2018 3:43 pm

Did you run those commands as root or with sudo?
Did you load g_mass_storage with removable=1?
If you had previously loaded g_mass_storage did you remove it first?
1. I ran them with sudo
2. Yes
3. When you say "previously loaded g_mass_storage", do you mean with:

Code: Select all

modprobe g_mass_storage file=/piusb.bin stall=0 ro=1 removable=1
If so, yes it is loaded. If I remove g_mass_storage with "rmmod" or "modprobe", then instead of getting permission denied, I get "file does not exist." Should g_mass_storage be loaded? I thought yes, because my understanding is that:

Code: Select all

sudo echo "" > /sys/devices/platform/soc/20980000.usb/gadget/lun0/file
is a better way of preventing the USB host from writing while I'm copying files, versus unloading "g_mass_storage".

From very basic research it looks like "permission denied" is a common troubleshooting problem, so maybe I will have to figure this one out.
Thanks.

User avatar
thagrol
Posts: 1851
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Wed Jun 27, 2018 4:29 pm

Sorry wasn't clear.

What I mean was that was g_mas_storage had already been loaded when you ran that modprobe command? Trying to modprobe an already loaded module doesn't do anything. It just silently exits.

Try this:

Code: Select all

sudo rmmod g_mass_storage
sudo modprobe g_mass_storage file=/piusb.bin stall=0 ro=0 removable=1
Then those echo commands

Note: you need ro=0 not ro=1 otherwise the USB host won't be able to write to it.

Edit: the g_mass_storage module does need to be loaded for those files/folders to be present.
This space unintentionally left blank.

pauljlamp
Posts: 5
Joined: Tue Jun 26, 2018 6:42 pm

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Wed Jun 27, 2018 4:39 pm

Alright, I think we are on the same page then, I was understanding you correctly.

Unfortunately, I'm still getting a "permission denied" after following those steps correctly.

I will update this post if I am able to find a solution. It isn't critical to making progress on this project, but I sure would like to be able to use those commands.

eckirchn
Posts: 3
Joined: Tue Apr 18, 2017 2:42 am

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Sat Oct 12, 2019 7:09 am

Have you had any luck with this? I would like to copy files off a DVR as they are written, but the only way I can think of is to disable SAMBA, umount, remount, enable SAMBA periodically, which though it wold work it's far from ideal.

User avatar
thagrol
Posts: 1851
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Sat Oct 12, 2019 10:36 am

eckirchn wrote:
Sat Oct 12, 2019 7:09 am
Have you had any luck with this? I would like to copy files off a DVR as they are written, but the only way I can think of is to disable SAMBA, umount, remount, enable SAMBA periodically, which though it wold work it's far from ideal.
That's about the only way. The trick is in knowing when the DVR has finished writing each file...
This space unintentionally left blank.

hippy
Posts: 5976
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Sat Oct 12, 2019 12:58 pm

pauljlamp wrote:
Wed Jun 27, 2018 4:39 pm
Unfortunately, I'm still getting a "permission denied" after following those steps correctly.
A bit late in the day, but if you mean these -

Code: Select all

sudo echo "" > /sys/devices/platform/soc/20980000.usb/gadget/lun0/file
sudo echo "/piusb.bin" > /sys/devices/platform/soc/20980000.usb/gadget/lun0/file
They need to be refactored -

Code: Select all

echo "" | sudo tee /sys/devices/platform/soc/20980000.usb/gadget/lun0/file
echo "/piusb.bin" | sudo tee /sys/devices/platform/soc/20980000.usb/gadget/lun0/file
But I haven't seen any advantage in doing it that way over "sudo modprobe g_mass_storage ..." and "sudo modprobe -r g_mass_storage". They seem to be equivalents.
eckirchn wrote:
Sat Oct 12, 2019 7:09 am
Have you had any luck with this?
Not me. I have a very simple set-up; Pi Zero generates an image of a clock which it updates every minute to share out to a digital picture frame. Despite the Pi side being write-only, the DPF read-only, having no caching and re-reading the entire directory when it looks for an image to display, I am still having issues when the Pi writes as the DPF is reading.

I have tried many things to avoid unsharing the gadget backing file store, double-buffering files, saving images as files the DPF doesn't understand then renaming them so the directory remains correct while the files change, the directory list only changing for the briefest of time, so it should display something without error no matter what it's doing and when things change but that doesn't work; I still see the DPF get confused, corrupt images, hang or stall.

The only thing which probably would help is to change when the files and directory are updated to just after the DPF has read the directory and file, but I have not found a way to do that, and think I may have to revert to actually monitoring the USB D+/D- lines on separate GPIO pins, rewriting the g_mass_storage gadget driver, using sector access to the backing store rather than file access.

And, it's even worse when there is read-write access from one or both sides, especially with caching going on which is in what's connected, not in the Pi itself.

It's on par with standing outside a library while an invisible and silent ninja sneaks in the back, adds and removes books, adds and removes pages from books. It's impossible to tell that's happened without going in to check what's changed, and impossible to know if they've finished adding or removing books, adding or removing pages of any particular books without a level of caching and analysis which doesn't exist.

Even worse when both sides have arbitrary read-write access.

Bottom line for me, as others have discovered themselves, is that I don't believe it can be done 'perfectly' with g_mass_storage gadget. One needs to unmount/mount, unshare/share, to see or let the other side know something has changed, make changes in a way which doesn't bork the other side.

This is why Android and other systems have moved away from MSD to MTP and other protocols.

User avatar
thagrol
Posts: 1851
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Followed this MagPi tutorial to make a smart flash drive: Having some trouble with mount/umount

Sat Oct 12, 2019 10:07 pm

Ultimately, we're all trying to do something the USB mass storage device class and block devices in general were never intended to do:concurrent access from multiple masters.

Now, a cluster aware filesystem can probably solve this but that's very niche and most of the posts around this topic want FAT32 which definitely isn't cluster aware.

About the only safe way I've found to pass writes from one side to the other is to manually unmount from the writing side forcing all buffers to be written to disc before unmounting and remounting only on the non-writing side. Once you've read/copied the new data unmount on the reading side and remount on the writing side.

If you're doing it by hand, unless there is no physical access to the devices you might as well be using sneakernet.

Sure, you can force unmounting the USB gadget from the USB host (rmmod or sysfs) but you have absolutely no idea when it's safe to do so. Even with write caching disabled on the USB host, it cannot be assumed that the last block written was the final one.

g_mass_storage has it's uses (e.g infrequent content updates, composite devices with drivers provided onboard to name two) but the risks and complications are too great for it to be used for live data transfer in anyway other than read only on all devices.

That said, if you've got the full OS source code for both the USB host and raspbian along with the neccessary skills...
This space unintentionally left blank.

Return to “Troubleshooting”