User avatar
seeingwithsound
Posts: 165
Joined: Sun Aug 28, 2011 6:07 am
Contact: Website

Re: Sight for the Blind for <100$

Thu Dec 24, 2015 10:27 pm

Found UVC compliant USB camera glasses on eBay for $15 including shipping!

November 29 I reported here about gambling on cheap UVC compliant camera glasses, and yes, they exist! Today I received the $15.31 glasses from http://www.ebay.com/itm/131330061361

I charged them for one or two hours, and first tested them under Windows 10 to figure out how to get them into webcam mode. I did not have a memory card inserted in the glasses. As usual, after connecting them to the USB port they are by default not in webcam mode but in drive mode. One short press on the small button on the inner side of the right leg then switches them into webcam mode (a longer press of a few seconds toggles webcam mode off again). In webcam mode Device Manager lists them simply as "USB Video Device" when the glasses are in webcam mode.

Under Windows the glasses don't seem to like the 176 x 144 video format at all, even while this format is listed as supported: The vOICe for Windows then shows a garbled view and becomes exceedingly sluggish as if it is about to crash. However, for instance the 320 x 240 format works fine, as do the 160 x 120 and 640 x 480 formats.

Next I tried them with the Raspberry Pi prototype device for The vOICe, and connected the glasses to one of its USB ports. After switching on the Raspberry Pi I waited a few seconds to make sure that the USB port was powered (a small LED in the glasses turns on) and pressed the small button. That is still before raspivoice tries to detect a camera, and lo and behold, a few seconds later I heard noises that were responsive to movements in front of the camera. So my hunch was right and these camera glasses are UVC compliant. So we now have an option for $15 UVC compliant camera glasses. The most serious downside that I noticed thus far is that the camera in these glasses works really poorly under low light conditions. It is insensitive to light to the point that indoor use under normal artificial light at night is nearly useless. However, I think the camera is OK for broad daylight. There may be other quirks but this is what I encountered thus far.

Merry Christmas!

Peter


Seeing with Sound - The vOICe
http://www.seeingwithsound.com


Specifications:
Model: V11
Product size 150 x 140 x 40mm (folded)
Physical weight Approx 35g
Read/write speed 1.5MBps-60MBps
USB 2.0 (High Speed)
Power supply Embedded 280mAh Li-polymer Battery
Power duration 1-2 hours
Power adaptor 5V DC/ 500mAh
Power consumption <0.8W
Memory Support Max 32GB
Resolution 720*480
Video format AVI
Audio Stereo
Lens specs 5 mega pixels CMOS camera
Recording speed Max 30fps
Working temp 0 - 60
Storage temp -100
Player Windows Media Player, Real player,Storm codec
Menu Language English/Chinese
Operating System WinXP, Win2000,Windows Vista, Win7
Last edited by seeingwithsound on Thu Dec 24, 2015 10:40 pm, edited 1 time in total.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Thu Dec 24, 2015 10:39 pm

so, some good news. I retried the jetpac/deepbelief install following their guide on github. I ran into the same issue I had the last time I tried (missing files). It turns out they werent missing, but just werent in the right place.

I had to manually copy a few files to the right places, and now it executes as expected.


My two copy commands are at the end of the following string:

Code: Select all

mkdir ~/projects
cd ~/projects
git clone https://github.com/jetpacapp/DeepBeliefSDK
sudo apt-get install -y mercurial
hg clone https://bitbucket.org/eigen/eigen
ln -s ~/projects/eigen ~/projects/DeepBeliefSDK/eigen
cd ~/projects/DeepBeliefSDK/source
make clean
sudo apt-get install gcc-4.8 g++-4.8
sudo rm -rf /usr/bin/gcc
sudo rm -rf /usr/bin/g++
sudo ln -s /usr/bin/gcc-4.8 /usr/bin/gcc
sudo ln -s /usr/bin/g++-4.8 /usr/bin/g++
make GEMM=eigen TARGET=pi2
./jpcnn -i data/dog.jpg -n ../networks/jetpac.ntwk -t -m s -d #Didnt work missing files
sudo cp libjpcnn.so /usr/lib/
sudo cp ~/projects/DeepBeliefSDK/source/src/include/libjpcnn.h /usr/include/
./jpcnn -i data/dog.jpg -n ../networks/jetpac.ntwk -t -m s -d  #Works this time
With it running as expected I just need to make a parser that looks for greater than 10-15% certainty and repeats it back over the audio using the camera as a source. I should be able to cook that up before I go back to work.

Maybe it will be my late christmas gift to the project.

User avatar
seeingwithsound
Posts: 165
Joined: Sun Aug 28, 2011 6:07 am
Contact: Website

Re: Sight for the Blind for <100$

Thu Dec 24, 2015 11:35 pm

mikey11 wrote:I retried the jetpac/deepbelief install [...], and now it executes as expected.
That is exciting progress, Mike! Concerning the choice of image recognition engine, a lot will depend on what things get recognized in (blind) daily living situations using the default databases, because taking millions of photos to train a neural network on seems best left to others. I do not know enough about the status of deep belief versus teradeep in this respect, and it is hard to predict what Google and other big data companies will offer in due course. I do believe in a combination of raw vision via sensory substitution with semantic vision through image recognition and speech.

Thanks,

Peter


Seeing with Sound - The vOICe
http://www.seeingwithsound.com

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Fri Dec 25, 2015 1:25 am

The good news is that because the jetpac routines are paralelized and raspivoice is not, I had both running at the same time, and it only increased the jetpac routine by about 100ms. (from 3.8 to 3.9 seconds)

I did not however have audio present, so I don't know the impact on raspivoice yet, and I didn't have jetpac looping or getting camera input yet, just analyzing a picture file.

As I develop the parser of jetpacs output I will discover how to run them at the same time.

As it relates to jetpac, they ship with a few neural nets. The default is fast, but has less objects and less accuracy. There is another that is slower but more accurate.

The thing I get excited about is having a few networks to choose from and let the user decide.

We will see what the next few days bring, but when I have something ready I will make a filesystem image and post it here.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Fri Dec 25, 2015 10:21 pm

so I of course ran into the issue of needing two video streams for the separate processes (I am not an accomplished enough programmer to merge raspivoice and jetpac).

I'm currently trying to get v4l2loopback to work so I can take the first webcam device, and then make two named video streams that raspivoice and jetpac can access independently.

I had hoped that I could use openCV to access the camera at the same time at a lower level, but had no success with that.
The plan is to use ffmpeg to create two new virtual cameras from v4l2loopback.

I'm currently at the point where v4l2loopback does not install because of missing kernel headers.

I'm trying to install them using rpi-source.

If anyone has been through this and has a better alternative, let me know.

PranavLal
Posts: 124
Joined: Fri Jun 28, 2013 4:49 pm

Re: Sight for the Blind for <100$

Sat Dec 26, 2015 2:01 am

Hi Mikey11,

I do not have any experience with this but the following links may help.
post giving v4l building instructions
http://www.ics.com/blog/raspberry-pi-camera-module
another link with updated instructions
http://www.linux-projects.org/modules/s ... e&artid=16
a link with generic build instructions
http://www.linuxtv.org/wiki/index.php/H ... ce_Drivers
url giving examples of using the loop back device
http://www.lavrsen.dk/foswiki/bin/view/ ... backDevice
Pranav

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sat Dec 26, 2015 7:31 am

well, I got it to build, but it won't autostart with the kernel. I've also used avconv to create clone webcam devices. This was more fruitful in terms of actually seeing that new video streams were cloned from the video glasses.

However none of those are able to be assigned to raspivoice. I get the distinct impression that this is a rathole at the moment. I have thought of some short term solutions which include writing an image from raspivoice, but the resolution used there is way too low for jetpac.

So I need to start with a high camera resolution, and then down convert for raspivoice after and pass a high res still image to jetpac.

I think the best bet is a named memory space in a separate process that raspivoice and jetpac can both access. However, this is beyond my capability for the current time.

As a compromise, I am going to continue to set things up for separate processes rather than simultaneous just to get a feel for how they both work, and as a half measure until I can get a higher level of commitment by real programmers.

I have found that in the past if I can lay down the meta of the program, real programmers can fix my problems up without too much trouble.

In any case, I'm glad to have some work to do for a few more days.

johnf
Posts: 30
Joined: Fri Dec 23, 2011 10:35 pm

Re: Sight for the Blind for <100$

Sat Dec 26, 2015 10:15 pm

I read a while ago that caution is required when switching off The RaspiVoice , the power needs to be maintained during a period of closing down, otherwise the SD card risks being corrupted.
a) How serious an operational risk is this?
b) Would it be feasible to have a tiny onboard battery reserved just for this purpose?
I am not nicking pits , I hope, I am just aware that there are a lot of unaware clients out there.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Sat Dec 26, 2015 11:32 pm

That problem has more or less been solved.

There is a battery used to power the device when it is worn, and there is a combination of a software and hardware solution that keep power applied until after the filesystem has been parked, thus avoiding the potential of corrupting the SD card.

The hardware side involves a resistor capacitor timer circuit, and the power switch also send a signal to the pi which causes a software shutdown to occur beforethe timer circuit counts down and totally cuts power.

The only thing that throws a wrench in this is if the software has crashed, the timer circuit won't be able to tell, but I suspect that doesn't really matter, as a crashed software on a wearable device will probably get a hard shutdown even on the more common consumer electronics such as phones.

johnf
Posts: 30
Joined: Fri Dec 23, 2011 10:35 pm

Re: Sight for the Blind for <100$

Sun Dec 27, 2015 1:23 pm

Good precautionary thinking. Another idea might be to have a duplicated operating system on a separate partition on SD card. Feasible?
If things can go wrong, they will go wrong. People forget about connecting leads in the heat of the moment .
It is so easy for impatient users to get a new product a bad name , because they can`t be bothered to read Page One of the manual. Been there myself, time after time.
Maybe tutors should keep a few spare SD cards , just in case of brickages. john

User avatar
seeingwithsound
Posts: 165
Joined: Sun Aug 28, 2011 6:07 am
Contact: Website

Re: Sight for the Blind for <100$

Tue Dec 29, 2015 4:14 pm

Yet another potential future alternative for the Raspberry Pi? "PINE A64, First $15 64-Bit Single Board Super Computer" https://www.kickstarter.com/projects/pi ... per-comput

Peter

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Wed Dec 30, 2015 3:57 am

So my magnum opus for 2015 is almost finished:

A new filesystem image currently syncing as soon as it's done I will link here and on the after-sight website

The features:

mandatory disclaimer (boo, but required and only forces on the first startup)
persistent device configuration between boot. (yay)

as a result of the changes, I implemented a nested menu system which is simple and intuitive.

You can now change settings for raspivoice that are not available in the running process:
you can cycle between three soundscape speeds - slow (2 seconds), normal (1.05 seconds) and fast (0.5 seconds)

Jetpac has been implemented, and currently only reads the top three possibilities regardless of certainty
You can select from three neural networks: two are LibCCV open source networks from 2010 and 2012, with a network from Jetpac available as well.

The Jetpac network is the fastest with ok accuracy, the 2010 is slow with ok accuracy, and the 2012 is slow with high accuracy. At least thats what I can gather from reading I haven't actually tested out all three networks in the wild much yet

raspivoice or jetpac can be set for autostarting on device boot with the saved settings. This should let end users find how they want to use the device primarily, and always go back to it.

When programs that use the audio channel operate, the menu system is suspended. You can kill all processes and return to the main menu by pressing the rotary button in for 4 seconds. This allows users to switch between raspivoice and jetpac relatively quickly.

The rangefinder/vibration motor are toggled from the main menu with their own menu item.

volume control is found under settings and ranges from 70-100% (although I put fake values read over audio to make people feel better. You really can't hear anything below 50%, and then using the menu becomes almost impossible)

Tinkerers can set raspivoice up to use the camera module from the settings. Jetpac defaults to the first USB camera.

So technically, you probably could run both programs at the same time, but I will leave that for another day. There is a well commented part early in execution on menu.py that excludes both from running at the same time. Delete a line or two, and you can have both autostarting.

I've cleaned up a lot of my terribly bad code, put python processes into separate processes with their own names for better management.

It's probably the best step towards a more use friendly implementation of everything out there for the moment, and gives people a chance to try out jetpac.

Obviously there will be errors, and far better ways of doing things. If you want to get your feet wet, by all means take a whack at it. Let me know what to fix, and we will all fail a little better tomorrow.

I will put the separate source files on the github repository and link that here as well.

Keep in mind there are new dependencies (and I honestly did so much in the past week I can hardly remember what is required and what isn't) so getting the filesystem image first will be very helpful.

Final Note:

The filesystem image will be on a 16GB card. I simply didn't have any smaller ones around when I was working, and christmas holidays made purchasing one a problem while I was feeling most inspired. I hope it doesn't put anyone out too much.

Anyhow, Happy New Year to all, and may 2016 be the best year yet!

User avatar
seeingwithsound
Posts: 165
Joined: Sun Aug 28, 2011 6:07 am
Contact: Website

Re: Sight for the Blind for <100$

Wed Dec 30, 2015 9:54 pm

Hi Mike,

This reads like impressive progress! I especially like the work on including support for neural network based object recognition along with raspivoice. I ran into something similar for Android the other day, where a new app named BlindTool wraps the MXNet deep learning framework. It speaks the names of objects recognized in the live camera view. BlindTool, available from Google Play, can be launched from within The vOICe for Android by adding its package name "the.blindtool" in The vOICe's "External apps" menu. Reliability of object recognition is not really ready for practical use yet, but that may come in due course through the worldwide effort in improving deep learning algorithms.

Will your new image work with the "old" prototype that I have? I do have a 16GB SD Card for it.

Thanks, and soon Happy New Year!

Peter


The vOICe for Android
http://www.seeingwithsound.com/android.htm

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Wed Dec 30, 2015 10:06 pm

It makes the device much more user friendly for sure.

I would like to send you a new version of the hardware, but you should be able to execute it on the old hardware without too much trouble

On the new hardware, the power switch governs total shutdown including hardware and software.

On the old version of hardware you will need to hold in the rotary knob for the full seven seconds to cause a software shutdown of the device and then follow that successful shutdown with flipping the power switch off.

I'm having a heap of troubling syncing the file on google drive. I left it syncing with my cellphone providing the uplink (wired internet where I am is abysmal). I came back to over 16Gb uploaded (the zipped image is only 3.2 Gb). I am limited to 5Gb per month, and that unforseen activity has just cost about $200 (Canadian cell phone companies are an oligopoly that quite simply exert an information tax on the citizens. It's pathetic). So now I am limited to my slow connection.

Does anyone know another easy way to share large files in a persistent fashion that is free?

User avatar
seeingwithsound
Posts: 165
Joined: Sun Aug 28, 2011 6:07 am
Contact: Website

Re: Sight for the Blind for <100$

Wed Dec 30, 2015 10:24 pm

Hi Mike,
mikey11 wrote:Does anyone know another easy way to share large files in a persistent fashion that is free?
I do not know your compilation tool chain, but I suppose that your (zipped) sources add up to much less than 3.2 GB? If you have access (e.g. via Windows' Remote Desktop Connection) to some remote PC that in turn does have a fast and cheap (wired) Internet connection, you could upload and unzip the sources and run the compilation to 3.2 GB on that PC and then let it sync the image with Google Drive?

Peter

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Wed Dec 30, 2015 11:32 pm

Well,

I've tried something new: MS OneDrive

https://onedrive.live.com/redir?resid=B ... file%2czip

Hopefully that link will work, please let me know if it doesn't and I will work somethings else out.

As before, the image file needs to be unzipped first, then a tool like win32diskimager can be used to put the image file on a 16 GB micro SD card.

If you have any other troubles, let me know.

PranavLal
Posts: 124
Joined: Fri Jun 28, 2013 4:49 pm

Re: Sight for the Blind for <100$

Thu Dec 31, 2015 1:46 am

Hi Mikey11,

The link did not work. Could you resend it?
The link was broken when I copied it. I got
https://onedrive.live.com/redir?resid=B ... file%2czip
Pranav


PranavLal
Posts: 124
Joined: Fri Jun 28, 2013 4:49 pm

Re: Sight for the Blind for <100$

Thu Dec 31, 2015 11:30 am

Hi Mikey11,

I am downloading it as I write this message. The link that worked for me is:
https://onedrive.live.com/redir?resid=B ... file%2czip

User avatar
seeingwithsound
Posts: 165
Joined: Sun Aug 28, 2011 6:07 am
Contact: Website

Re: Sight for the Blind for <100$

Thu Dec 31, 2015 7:44 pm

I successfully downloaded the image from the same link that Pranav used. After unzipping I flashed it using Win32 Disk Imager. A first test indicates that it works without major issues on my prototype system. Raspivoice appears to work fine as before. I tried the default Jetpac, but thus far found its recognition hopeless. Not once did it give a meaningful recognition, even with a clean view on the computer screen of a fork on a black background displayed on the computer screen, or an image of a house. It spoke many object names with the camera pointed at things but never even close in appearance to the common object that it was supposed to recognize. Anyone who got half-decent results?

Peter

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Thu Dec 31, 2015 7:52 pm

Using the libccv2012 network gave some positive results.

I identified a chair, lightswitch, desktop computer amongst some others.

I had to stand quite still and wait for two result sets usually.

It's not always great yet for sure. Might need to adjust the picture resolution and do some transforms first to be more successful.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Thu Dec 31, 2015 7:54 pm

Using the libccv2012 network gave some positive results.

I identified a chair, lightswitch, desktop computer amongst some others.

I had to stand quite still and wait for two result sets usually.

It's not always great yet for sure. Might need to adjust the picture resolution and do some transforms first to be more successful.

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Thu Dec 31, 2015 11:11 pm

with the somewhat underwhelming performance of jetpac I am looking at teradeep now.

I found that I was unable to install torch7 in the past. The reason for this was missing dependencies which I could not solve in the past.

I noticed today that one of the dependency installer scripts is screening for which linux install you have and if you dont match it excludes you and exits.

I have tried to rework that script for the rpi v 2 and am attempting again.

the ./install-deps.sh script had a section for ubuntu which is what I stripped it down to and tried. I did see many dependencies install correctly, and then openBLAS installed correctly afterwards (I never even got that to install properly before)

I'm currently rebuilding torch again and fingers crossed it will go well.

My understanding is this:

jetpac on the pi V 2 uses the cpu in a parallel fashion, it works with a framerate of once/4 seconds give or take a few hundred milliseconds.

teradeep through openBLAS uses the GPU.

My money is on teradeep having higher performance at lower power consumption and having a higher framerate.

User avatar
seeingwithsound
Posts: 165
Joined: Sun Aug 28, 2011 6:07 am
Contact: Website

Re: Sight for the Blind for <100$

Fri Jan 01, 2016 12:02 pm

Hi Mike,

Yes, libccv2012 worked a little better for me. Last night I was hampered by the battery being low, causing jetpac and all speech to stop, making the device unresponsive to a shutdown request, which in turn led to a corrupted filesystem when forcibly turning off power. Anyway, I reflashed the image and charged the device overnight, which fixed these basic issues. Still, I think there is an issue with frame buffering: using libccv2012 and pointing the camera to a screen image of a house, it returns reasonable matching names like barn and church. However, if I then switch to an image of a fork on a black background it continues to speak barn and church for about a minute or so. I therefore suspect that images are captured at a higher rate than jetpac digests and speaks, and that those images are put on a stack, leading to huge and perhaps every-increasing latencies. This buffering issue would likely affect any alternative neural network that you try to use, because it happens somewhere in the interface between frame grabbing and neural network.

Peter


Seeing with Sound - The vOICe
http://www.seeingwithsound.com

mikey11
Posts: 355
Joined: Tue Jun 25, 2013 6:18 am
Location: canada
Contact: Website

Re: Sight for the Blind for <100$

Fri Jan 01, 2016 3:08 pm

That 'sounds' like a relatively small and easy to correct problem.

I will take a stab at it tonight after work.

The relevant file is in /home/pi/jetpac.py

It loads some info from the config file, then goes into a loop taking a pic, writing that to a file, and then that is passed to jetpac in a command line argument.

Jetpac returns a two dimensional list of certainty and identity.

A very small amount of parsing the results takes place and the top three are spoken with espeak

Then the image file 'should' be deleted to make room for the next capture.

The loop then repeats.

I used a three line opencv image capture example to get started, so I'm sure more can be done. I suspect that this will be fixed tonight.

Return to “Assistive technology and accessibility”