Sight for the Blind for <100$

[PranavLal]

Hi Mikey11,

I am using opencv for 3 key things.
1. Capture images from the camera.
2. Adjust the image resolution.
3. Convert the image to the array of xy coordinates required by the vOICe.

I suspect you can handle tasks 1 and 2 from any application but task 3 is going to be tricky. However, the image magic library has an exportPixels method that may help.

Pranav

[mikey11]

So...

I didn't understand it very well before, but I think I'm starting to.

openCV plus the rpi camera still uses the provided closed source capture programs to get its functionality, but still uses the gpu.

ie you source your image using raspistill, then use openCV to do further manipulation.

sourcing images with raspistill incurs a startup time penalty, but using the timelapse function seems to get around this by keeping the camera going.

So, I am looking at using the timelapse function to the ramdrive, and having part of my routine go in and delete images. after they have been processed.

If possible, the timelapse function will be busy in the background, constantly grabbing images.

Then using openCV, I can probably relate the hificode in CPP that Peter has worked on. I am trying to use Berak's simple.cpp first just to do proof of concept because quite frankly I don't fully understand the sound generation stuff, and I also don't program in C at all (but with the internet and code examples, I'm sure that will change).

What I think will be useful is to try to leverage all of the transformation flags from raspistill to achieve as many of the effects as desired.

I could easily see how if hit '1', it will add the invert flag, hit it again, and the invert flag goes away. Hit '2' to engage the blinders, and the flags for width and height change... etc.

the major feature I know I don't know how to do is the foveal transform, but I saw in openCV there is a polar coordinates function that might achieve this. At some point, I will not be able to do certain things, but hopefully once a core program is functioning, it could generate the interest in the people who have the skills.

So the looping program for raspistill would have to terminate the current timelapse function, incur the time penalty of about 0.8s, and then the new mode would engage.

Pranav How annoying would it be to have a slow frame every 5-10 seconds?

It seems to me the best way to handle raspistill is to let it run in timelapse mode for a given length of time, search for transform input, and then restart the timelapse with the new feature enabled.

[johnf]

Hi... It has occurred to me that ,if you consider the normal use of The vOICe, it is as a moving scan of a vertical stack of pixels. According to height and brightness of each pixel, a sound waveform is generated, and additively mixed with the other outputs of the stack.
For this demonstration of sight substitution, it is not necessary to capture and process the whole picture.
With the Pi camera so closely coupled to the GPU, maybe there are opportunities to investigate and to produce a really minimal but street-functional vOICe system, possibly based on a high-performance embedded microprocessor ?
The R-Pi is evolving so as to replace the PC-in -a -backpack , the need for USB may disappear, maybe the Pi itself will be a bridge towards an even more miniature and portable device.
What do you think? johnf

[seeingwithsound]

Hi Mikey11,

> the major feature I know I don't know how to do is the foveal transform, but
> I saw in openCV there is a polar coordinates function that might achieve this.

The vOICe foveal mapping algorithm is not public, and unlike a polar mapping leaves horizontal and vertical straight lines perfectly straight. However, I would not worry about adding a foveal mapping for the initial versions. It is a refinement that one can live without. You will probably need to focus more on performance optimizations in order to get about one soundscape per second on the Raspberry Pi's CPU.

Best regards,

Peter Meijer


USB Camera Glasses for the Blind
http://www.seeingwithsound.com/camera_glasses.htm

[PranavLal]

Hi Mikey11,
<snip Pranav How annoying would it be to have a slow frame every 5-10 seconds?
PL] Very. The soundscapes have to stay at one frame per second. I am not sure if the current slowdown is due to the opencv camera capture. The problem appears to be more with the actual processing of the image. Let us keep things simple for now.

Have you been able to run the existing code on your raspberry pi? If so, load the code into gdb and step through it line by line. You will get an idea of where the slowdown occurs. I know, this is not the best way to do things but it works for me. <chuckle

Pranav

[PranavLal]

Hi Johnf,

I am all for high-performance embedded microprocessors as long as they are mainstream hardware and can be acquired by the ordinary user without any difficulty. We do need the entire picture. I have tried limiting the field of view and it is uncomfortable to say the least. That could be an individual preference though.

Pranav

Hi... It has occurred to me that ,if you consider the normal use of The vOICe, it is as a moving scan of a vertical stack of pixels. According to height and brightness of each pixel, a sound waveform is generated, and additively mixed with the other outputs of the stack.
For this demonstration of sight substitution, it is not necessary to capture and process the whole picture.
With the Pi camera so closely coupled to the GPU, maybe there are opportunities to investigate and to produce a really minimal but street-functional vOICe system, possibly based on a high-performance embedded microprocessor ?
The R-Pi is evolving so as to replace the PC-in -a -backpack , the need for USB may disappear, maybe the Pi itself will be a bridge towards an even more miniature and portable device.
What do you think? johnf

[mikey11]

Hi everyone,

I realize I have been slow to work on this, and that is unlikely to change in the relatively near future as I am busy with other seasonally related obligations (building a garage).

I can confirm that I have successfully used the ramdrive:

/dev/shm

to capture raspistill to timelapse images to. I can also confirm that this method incurs almost no penalty to the CPU. I monitored it during the capture and saw a few spikes to about 3% usage. I was using a 200ms delay between frames at 320x240.

This suggests that we will definitely be able to use the rpi camera for the capture process, leaving the CPU open to the other tasks.

What is the best resolution to capture at? I saw some values of 176x144 or something like that. The lower the capture resolution, the less likely it is that this process will incur CPU penalties.

My thoughts with this is to use raspistill to capture timelapse images for a 10 minute period. If the user provides input to change a visual mode, the raspistill process would be terminated, and restarted with the new mode in place.

This will incur a 0.8 second penalty for mode changes.

A separate process will be running that looks in /dev/shm, loads an image for audio processing, and deletes the rest. The audio is output, and the loop repeats.

The timelapse capture function will keep filling the ramdrive until it's full, and I don't think there is any convenient way to have it overwrite old files to keep within the ramdrives size allowance. I haven't extensively tested this yet, but by looping, and deleting files, as the audio plays, the timelapse function will drop new frames into the ramdrive until the audio is sent. With a capture rate of under 500ms, there should be at least one new frame available no matter what.

At the current time, I have been unable to get the openCV functions to work, and keep getting linker errors regardless of what I do. I am going to start with a fresh install of everything again, as I seem to have fallen into some sort of dependency hell.

When I get more time, I will pursue the hificode of Peters, and the simple.cpp that berak created.

I am frustrated by my lack of availability for this, but I will keep coming back when I can.

[mikey11]

I honestly think this is a project that might almost fly on kickstarter.

The advantages of that would be that crowdsourced funding could:

1. Pay a devoted programmer to implement the code. ie.the software
2. Enable someone to explore the best options for batteries/cases/cabling/mounting the camera. ie, the hardware (I'm thinking 3D printed camera/glasses mount, 0.75-1M long cable for the camera so the computer can be located away from the head, etc.)
3. use any extra funding not consumed by these processes for preparing Raspberry PI vOICe bundles for whatever the cost ends up being.

Then those who fund could be acknowledged as contributors.

I did look into this briefly, and it looked like kickstarter is not really interested in helping people raise funds for charities, and this kind of project probably skirts that line a little bit.

In my mind it would be great if this worked out, and someone could prepare bulk orders for organisations that help blind people.

Does anyone find this avenue appealing?

I am totally ok with just continuing this as a spare time project, but maybe a crowd funded initiative could really speed this up.

[Anono Moose]

Without reading all the posts, it sounds like a great idea.

I'm an MSEE, and now actively retired - volunteer is my middle name.
I would think of open source software, with paypal and flittr.com funding
from the interested community.

I've worked with the blind since 1995, and people who would want this
"work experience" for their CV/Resume should come bounding out of the woodwork.

Glasses On!

[johnf]

http://www.kickstarter.com/projects/254 ... ion-sensor
is a fearsome little camera board with object recognition (2 core processor) and potential for facial recognition.
It outputs (kind) and (position) to Arduino, as a robot head device.
What about this CPU-saving peripheral working with the Pi? john

[PranavLal]

Hi Mikey11,

I appreciate your frustration with linker errors. The commandline I use to compile and link the project is below.
g++ -I /usr/include -O3 hificode_OpenCV.cpp -o vl01.bin -lopencv_core -lopencv_imgproc -lopencv_highgui

Why are you trying time laps functions etc.? Let us try and get the existing opencv code to work.
As for funding, my question is one of sustainability. We can certainly try a crowd funded model but how do we sustain it? I doubt selling new units will allow for that.

Pranav

[johnf]

The crowdsourcing option is attractive , but the need remains for a financially profitable outcome. Hard to imagine how The vOICe can be made into a commercially successful business.
However, there are other ways to address the problem. Blindness, worldwide, is a major issue and there are assistive bodies ranging from UNESCO and Medecins Sans Frontiers at the front, to Rotary and Lions Clubs and churches raising funds in the comfortable West. Ultra-wealthy patrons like Branson and Gates disburse millions every year.
The problem is, how to capture the attention of possible sponsors? john

[PranavLal]

Hi John,

The only way I know of doing this is to get a working version out first and then ask for funding. As for profitable, people should pay for learning how to see. <chuckle


Pranav

[tuxun]

Hi, dude!
For my post I come with an idea but not enough time for realize it. I think it could be a COMMUNITY PROJECT so I'll just launch the thing.
I'm a french with two sleeping RPIs. I love programming because it can helpl people in lots of way.
Today, I watched this videohttp://www.dailymotion.com/video/xw8ygi ... oheac_tech

A blind people(sorry for my poor english in this case) explain how he use his Iphone to solve common problem for blind people.
1.BARCODE recognize and search (for cooking)
2. Vocal screen reader (to read a menu at restaurant)
3.Braiile keyboard (for better reading a book than speechly?)
4.internet search (he search on the web when the next bus arrive at a bus stop)
Etc... I think RPI is a good alternative to Iphone by his connectivity but I can't endorse any project. I'm going to india at the start of the next year for growing myself...
please take this idea as you want and help the world! (maybe create a new thread resuming this)
tuxun
(cc)copyme(cc)

[PranavLal]

Tuxon,
The idea here is to create the vOICe for the raspberry pi. The rest of what you suggest can be added. However, the pi does not have a portable control interface so these areas are very much secondary. The focus is on optimizing the vOICe on the pi. Feel free to help if you are a C / C++ programmer.

Pranav

[tuxun]

Should I better try to port it on UDOO?
Is there a git or svn repo?
I got lot of theory, but miss pratice in programing. and my lasts tries was about web coding....

[PranavLal]

Hi,

The raspberry pi is mainstream so I am interested in sticking with that hardware. You can find the code at my blog at http://techesoterica.com. Iam trying to attach it with this message as well.

Pranav

[mikey11]

Greets all,

after a long hiatus wherein I built a garage with a loft, and had the arrival of my first born, I have come back to this project.

This time I bear a gift.

I've written a python script that does the following:

spawns and kills raspistill processes

the raspistill process writes to the ram drive /dev/shm at a defined interval

I've setup a keyscanning loop to let you cycle through the exposure, awb, image effects, and metering mode.

the next step is to get the simple.cpp compiled on the pi, and have it access /dev/shm/picture.jpg to convert it to a soundscape.

The keyscan process is a separate process, the simple.cpp process is a separate process.

Once both are running independently, vOICe on the rpi is functional.

I have obeserved the CPU usage with the keyscanning and raspistill processes functioning. It still looks like less than 10% CPU which will then be available for simple.cpp

Cheers,

Next steps: determine which flags can be dropped from exposure,awb,image effects, and metering mode. See if the region of interest settings will act as a digital zoom, and see about changing rates for image aquisition and sound playback.

Code: Select all

# Key scanner for vOICe on Raspberry PI
# This script scans the keyboard for input
# If a valid keypress is detected, it kills the currently running raspistill process
# The keypress is interpreted, from one of four categories:
# exposure mode, AWB mode (light type), image effect, and metering mode
# The keypress will cycle through the possible type within the mode and then wrap back to the first
# additional keys will be defined for other relevant flags as the project progresses
# The raspistill launch string is then built incorporating the latest flag change
# It launches raspistill as a timelapse with a lapse duration of 300ms, and a timeout of five minutes
# It writes the captured files to the ramdrive for the vOICe process to grab from
# Then it returns to looking for a keypress

import subprocess #used for starting and kill raspistill process
import curses #used for keyboard scanning


stdscr = curses.initscr() #set curses to terminal screen
curses.cbreak() #engage keyscanning mode. This WILL NOT WORK in IDLE you have to launch from a terminal
stdscr.keypad(1) #allows use of keypad

x = 1 # just a random variable

#all the string components will have a leading space to make concatenation easier

width = ' -w 320' #image width
height = ' -h 200' #image height
time_lapse = ' -tl 1000' #timelapse in milliseconds
time = ' -t 300000' #length of time for timelapse = 5 minutes or 300 seconds or 300000 ms
colour_effect = ' -cfx 128:128' #Makes the image monochrome implement this once other things are working
preview = ' -p 100,100,320,200' # preview windows for use in debugging, remove for production
path = ' -o /dev/shm/picture.jpg' #output is to the 16mb ramdrive built into raspbian this saves SD card writes and latency

exposure_list = [' off',' auto',' night',' nightpreview',' backlight',' spotlight',' sports',' snow',' beach',' verylong',' fixedfps',' antishake',' fireworks']
awb_list = [' off',' auto',' sun',' cloudshade',' tungsten',' fluorescent',' incandescent',' flash',' horizon']
image_effect_list = [' none',' negative',' solarise',' sketch',' denoise',' oilpaint',' hatch',' gpen',' pastel',' watercolour',' film',' blur',' saturation',' colourswap',' washedout',' posterise',' colourpoint',' colourbalance',' cartoon']
metering_method_list = [' average',' spot',' backlit',' matrix']
exposure_pos = 1 #Set exposure to auto
awb_pos = 1 #set AWB to auto
image_effect_pos = 0 #set image effect to none
metering_method_pos = 0 #set metering method to average
raspistill_execute = '' # set the execute sting to nothing first



# roi = ['roi 0,0,1,1','roi 0.5,0.5,0.25,0.25'] # I believe this will act as a digital zoom by using less of the sensor

key = '' #set initial keypress value to blank

        
def KillRaspistillProcess():
    subprocess.call(["sudo", "pkill", "raspistill"], shell=False) # subprocess allows for the kill command to go through and come back to keyscanning
    return()

def LaunchNewRaspistill():
    global exposure
    global metering_method
    global awb
    global image_effect
    exposure = " -ex" + exposure_list[exposure_pos]
    awb = " -awb" + awb_list[awb_pos]
    image_effect = " -ifx" + image_effect_list[image_effect_pos]
    metering_method = " -mm" + metering_method_list[metering_method_pos]
    raspistill_execute = "sudo raspistill"+ width + height+ time_lapse + time + preview+ path + exposure + awb + image_effect + metering_method#concatenate the string
    print (raspistill_execute)#print the string for debug purposes
    #execute the string with shell = true
    subprocess.Popen([raspistill_execute], shell=True) #take the current launch string and execute the timelapse using raspistill
    return()

def CycleExposure():
    global exposure_pos
    exposure_pos = exposure_pos + 1 # increase the current exposure element position in the list
    if exposure_pos == 13: # There are 12 exposure elements in the list. If this is exceeded, we must return to position 0
        exposure_pos = 0 # return to position 0 is list length has been exceeded
    else:
        exposure_pos = exposure_pos # if the list length has not been exceeded, proceed with the current setting
    exposure = " -ex" + exposure_list[exposure_pos] #Setting exposure for string building
    print ("cycle exposure")
    return()
def CycleAWB():
    global awb_pos
    awb_pos = awb_pos +1
    if awb_pos == 9: #awb has 8 elements. see CycleExposure() for comments
        awb_pos = 0
    else:
        awb_pos = awb_pos
    awb = " -awb" + awb_list[awb_pos] 
    print("cycle awb")
    return()
def CycleImageEffect():
    global image_effect_pos
    image_effect_pos = image_effect_pos + 1
    if image_effect_pos == 20:
        image_effect_pos = 0
    else:
        image_effect_pos = image_effect_pos
    image_effect = " -ifx" + image_effect_list[image_effect_pos] # more preset for string
    print ("cycle image effect")
    return()
def CycleMeteringMethod():
    global metering_method_pos
    metering_method_pos = metering_method_pos + 1
    if metering_method_pos == 4:
        metering_method_pos = 0
    else:
        metering_method_pos = metering_method_pos
    metering_method = " -mm" + metering_method_list[metering_method_pos] # more preset for string
    print ("cycle metering method")
    return()

# Main keyscanning loop

while key != ord('q'):
            key=stdscr.getch()
            stdscr.addch(20,25,key)
            stdscr.refresh()
            if key == ord('1'):
                KillRaspistillProcess()
                CycleExposure()
                LaunchNewRaspistill()
            elif key == ord('2'):
                KillRaspistillProcess()
                CycleAWB()
                LaunchNewRaspistill()
            elif key == ord('3'):
                KillRaspistillProcess()
                CycleImageEffect()
                LaunchNewRaspistill()
            elif key == ord('4'):
                KillRaspistillProcess()
                CycleMeteringMethod()
                LaunchNewRaspistill()
            else: x=x+1
            print ("out of loop")
curses.endwin()
print("also out of loop")

and if anyone wants to get handy on the simple.cpp and start aiming it at /dev/shm/picture.jpg, that would be excellent.

If not, that's my next target when I get the time.

Cheers,

Mike

[PranavLal]

Mike,

Congratulations on your first born! I have got the python script and will try it today. In the mean time, here is the C code that should capture the image from the ramdrive.

Code: Select all

/* C program for soundscape generation. (C) P.B.L. Meijer 1996 */

#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc/imgproc.hpp>

using namespace cv;

#include <math.h>
#include <stdio.h>


const char*  FNAME="h01.wav";    /* User-defined parameters   */

const int N=64;   /* Resolution, i.e., # rows and columns   */
const int FL=500;   /* Lowest  frequency (Hz) in soundscape */
const int FH=5000;   /* Highest frequency (Hz)                 */
const int  FS=44100;   /* Sample  frequency (Hz)                 */
const double T=1.05;   /* Image to sound conversion time (s)     */

// non-const, so they can get toggled at startup
int D=1;   /* Linear|Exponential=0|1 distribution    */
int HIFI=1;   /* 8-bit|16-bit=0|1 sound quality         */
int STEREO=1;   /* Mono|Stereo=0|1 sound selection        */
int DELAY=1;   /* Nodelay|Delay=0|1 model   (STEREO=1)   */
int FADE=1;   /* Relative fade Yes|No=0|1  (STEREO=1)   */
int DIFFR=1;   /* Diffraction Yes|No=0|1    (STEREO=1)   */
int BSPL=1;   /* Rectangular|B-spline=0|1 time window   */

const int BW=0;   /* 16|2-level=0|1 grey format in *P[]     */
#define C_ALLOC(number, type) ((type *) calloc((number),sizeof(type)) )
const double TwoPi=6.283185307179586476925287;
//get an image from the raspberry pi camera
void takePiImage()
{//start of routine
char cmd[500];
sprintf(cmd,"/opt/vc/bin/raspistill -t 1 -n -w 176 -h 144 -o h01.jpg");
system(cmd);
}//end of routine
int playSound( char *filename ) { 
    char command[256]; int status; 
    /* create command to execute */ 
    sprintf( command, "aplay -i %s", filename ); /* play sound */ status = system( command ); 
    return status; 
  //  return PlaySound(filename,0,SND_FILENAME);
}



FILE *fp; unsigned long ir=0L, ia=9301L, ic=49297L, im=233280L;
void wi(unsigned int i) { int b1,b0; b0=i%256; b1=(i-b0)/256; fputc(b0,fp); fputc(b1,fp); }
void wl(long l) { unsigned int i1,i0; i0=l%65536L; i1=(l-i0)/65536L; wi(i0); wi(i1); }
double rnd(void){ ir = (ir*ia+ic) % im; return ir / (1.0*im); }

int main(int argc, char *argv[])
{
    int cam_id = 0;  // 1st available opencv camera
    // override flags from cmdline : prog.exe [cam_id] [linear] [nodelay], etc. args can be in any order
    for ( int a=1; a<argc; a++ )
    {
        if ( ! strcmp(argv[a],"linear") )        { D = 0; }
        else if ( ! strcmp(argv[a],"nohifi") )   { HIFI = 0; }
        else if ( ! strcmp(argv[a],"nostereo") ) { STEREO = 0; }
        else if ( ! strcmp(argv[a],"nodelay") )  { DELAY = 0; }
        else if ( ! strcmp(argv[a],"nofade") )   { FADE = 0; }
        else if ( ! strcmp(argv[a],"nodiffr") )  { DIFFR = 0; }
        else if ( ! strcmp(argv[a],"nobspl") )   { BSPL = 0; }
        else cam_id = atoi(argv[a]);
    }

    //beginning of main function
    //start of opencv variables
    VideoCapture cap; //handle camera capture
    Mat gray,frame; //image matrices
	//end of opencv variables
	//variables for soundscape generation (initialization)
    int i, j, d=D, ss;
    long k=0L, l, ns=2L*(long)(0.5*FS*T), m=ns/N;
    double **A, a, t, dt=1.0/FS, *w, *phi0, s, y, yp, z, tau1, tau2, x, theta,
      scale=0.5/sqrt(double(N)), q, q2, r, sl, sr, tl, tr, yl, ypl, yr, ypr,
         zl, zr, hrtf, hrtfl, hrtfr, v=340.0, /* v = speed of sound (m/s) */
          hs=0.20;  /* hs = characteristic acoustical size of head (m) */
    w    = C_ALLOC(N, double);
    phi0 = C_ALLOC(N, double);
    A    = C_ALLOC(N, double *);

    for (i=0; i<N; i++) A[i] = C_ALLOC(N, double);  /* N x N pixel matrix */
    //end of variables for soundscape generation (initialization)
    /* Set lin|exp (0|1) frequency distribution and random initial phase */
    if (d) for (i=0; i<N; i++) w[i] = TwoPi * FL * pow(1.0* FH/FL,1.0*i/(N-1));
    else   for (i=0; i<N; i++) w[i] = TwoPi * FL + TwoPi * (FH-FL)   *i/(N-1) ;
    for (i=0; i<N; i++) phi0[i] = TwoPi * rnd();
    //end of frequency randomization
    /*start video capture and image downsampling for feeding into matrix contained in variable A*/
    // device ids
//converting to a file based capturing system
    bool ok = true;
    //create a main while loop that handles keyboard input
    while(ok)
    {//start of keyboard while loop  
//take the image to sonify
takePiImage();
//read image file from disk for sonification
frame=imread("/dev/shm/picture.jpg",CV_LOAD_IMAGE_UNCHANGED);  
        //save image after height and width reduction to disk
        //imwrite("h01Shrunk.jpg",frame);

        if ( frame.empty() )
        {
            fprintf(stderr,"No image.");
            return 1;
        }
        //temp overwrite frame with image for testing
        //frame=imread("lineflat.jpg",CV_LOAD_IMAGE_COLOR);
        //reduce to gray scale
        Mat tmp;
        cvtColor(frame,tmp,CV_BGR2GRAY);
        //actually resize to NxN
        if ( frame.rows != N )
            resize( tmp, gray, Size(N,N) );
        else
            gray=tmp;

        //save grayscale image to disk for debugging
    //imwrite("h01Gray.jpg",gray);
    //feed data into matrix overwriting the hard coded image
    for(int rws=0;rws<N;rws++)
    {//start of matrix population for loop for rows
        for(int cols=0;cols<N;cols++)
        {//start of matrix population for loop for columns
            double mVal=gray.at<uchar>(rws,cols)/16;
            A[rws][cols]=mVal;
        }//end of matrix population for loop for columns

    }//end of matrix population for loop for rows


    /*end video capture and image downsampling for feeding into matrix contained in variable A*/
    
    // moved this into the loop, so you can toggle flags
    long sso=HIFI?0L:128L, ssm=HIFI?32768L:128L;
    int HIST = (1+HIFI)*(1+STEREO);

    /* Write 8/16-bit mono/stereo .wav file, using rectangular time window */
    fp = fopen(FNAME,"wb"); fprintf(fp,"RIFF"); wl(ns*HIST+36L);
    fprintf(fp,"WAVEfmt "); wl(16L); wi(1); wi(STEREO?2:1); wl(0L+FS);
    wl(0L+FS*HIST); wi(HIST); wi(HIFI?16:8); fprintf(fp,"data"); wl(ns*HIST);
    tau1 = 0.5 / w[N-1]; tau2 = 0.25 * tau1*tau1;
    y = yl = yr = z = zl = zr = 0.0;
    /* Not optimized for speed */
    while (k < ns && !STEREO) {
        if (BSPL) { q = 1.0 * (k%m)/(m-1); q2 = 0.5*q*q; }
        j = k / m; if (j>N-1) j=N-1; s = 0.0; t = k * dt;
        if (k < ns/(5*N)) s = (2.0*rnd()-1.0) / scale;  /* "click" */
        else for (i=0; i<N; i++) {
         if (BSPL) {  /* Quadratic B-spline for smooth C1 time window */
            if (j==0) a = (1.0-q2)*A[i][j]+q2*A[i][j+1];
            else if (j==N-1) a = (q2-q+0.5)*A[i][j-1]+(0.5+q-q2)*A[i][j];
            else a = (q2-q+0.5)*A[i][j-1]+(0.5+q-q*q)*A[i][j]+q2*A[i][j+1];
         }
         else a = A[i][j];
         s += a * sin(w[i] * t + phi0[i]);
        }
        yp = y; y = tau1/dt + tau2/(dt*dt);
        y  = (s + y * yp + tau2/dt * z) / (1.0 + y); z = (y - yp) / dt;
        l  = sso + 0.5 + scale * ssm * y; /* y = 2nd order filtered s */
        if (l >= sso-1+ssm) l = sso-1+ssm; if (l < sso-ssm) l = sso-ssm;
        ss = (unsigned int) l;
        if (HIFI) wi(ss); else fputc(ss,fp);
        k++;
   }
   while (k < ns && STEREO) {
      if (BSPL) { q = 1.0 * (k%m)/(m-1); q2 = 0.5*q*q; }
      j = k / m; if (j>N-1) j=N-1;
      r = 1.0 * k/(ns-1);  /* Binaural attenuation/delay parameter */
      theta = (r-0.5) * TwoPi/3; x = 0.5 * hs * (theta + sin(theta));
      tl = tr = k * dt; if (DELAY) tr += x / v; /* Time delay model */
      x  = fabs(x); sl = sr = 0.0; hrtfl = hrtfr = 1.0;
      for (i=0; i<N; i++) {
         if (DIFFR) {
            /* First order frequency-dependent azimuth diffraction model */
            if (TwoPi*v/w[i] > x) hrtf = 1.0; else hrtf = TwoPi*v/(x*w[i]);
            if (theta < 0.0) { hrtfl =  1.0; hrtfr = hrtf; }
            else             { hrtfl = hrtf; hrtfr =  1.0; }
         }
         if (FADE) {
            /* Simple frequency-independent relative fade model */
	    hrtfl *= (1.0-0.7*r);
	    hrtfr *= (0.3+0.7*r);
         }
         if (BSPL) {
            if (j==0) a = (1.0-q2)*A[i][j]+q2*A[i][j+1];
            else if (j==N-1) a = (q2-q+0.5)*A[i][j-1]+(0.5+q-q2)*A[i][j];
            else a = (q2-q+0.5)*A[i][j-1]+(0.5+q-q*q)*A[i][j]+q2*A[i][j+1];
         }
         else a = A[i][j];
         sl += hrtfl * a * sin(w[i] * tl + phi0[i]);
         sr += hrtfr * a * sin(w[i] * tr + phi0[i]);
      }
      if (k < ns/(5*N)) sl = (2.0*rnd()-1.0) / scale;  /* Left "click" */
      if (tl < 0.0) sl = 0.0;
      if (tr < 0.0) sr = 0.0;
      ypl = yl; yl = tau1/dt + tau2/(dt*dt);
      yl  = (sl + yl * ypl + tau2/dt * zl) / (1.0 + yl); zl = (yl - ypl) / dt;
      ypr = yr; yr = tau1/dt + tau2/(dt*dt);
      yr  = (sr + yr * ypr + tau2/dt * zr) / (1.0 + yr); zr = (yr - ypr) / dt;
      l   = sso + 0.5 + scale * ssm * yl;
      if (l >= sso-1+ssm) l = sso-1+ssm; if (l < sso-ssm) l = sso-ssm;
      ss  = (unsigned int) l;
      if (HIFI) wi(ss); else fputc(ss,fp);  /* Left channel */
      l   = sso + 0.5 + scale * ssm * yr;
      if (l >= sso-1+ssm) l = sso-1+ssm; if (l < sso-ssm) l = sso-ssm;
      ss  = (unsigned int) l;
      if (HIFI) wi(ss); else fputc(ss,fp);  /* Right channel */
      k++;
   }
   fclose(fp);
    //play the soundscape using aplay
    int pps=playSound("h01.wav");
    int rFile=remove("h01.jpg");
     rFile=remove("h01.wav");

    k=0; //reset the sample count to allow while loop to execute
}//end of keyboard while loop
printf("normal exit. \n");
 return(0);
}//end of main function

 

[mikey11]

Hi Pranav,

I haven't gotten to the C program yet, but I did look at my script further. There are some immediate things that need improvement.

the script is writing to picture.jpg. I notice that this results in incomplete JPG's on an instantaneous basis much of the time. I may need to loop it to create 10 pictures sequentially, open up a completed one with the soundscape generator, delete the contents of /dev/shm then play the soundscape. In the time it takes the soundscape to play, new complete JPG's will have been created in dev/shm.

rinse, repeat

just storing a links here for later consumption
http://www.raspberrypi.org/phpBB3/viewt ... 1&p=417302
http://www.raspberrypi.org/forum/viewto ... 43&t=50075

[mikey11]

I got a lot more working now.

will update soon.

I need to know how to spawn a subprocess from cpp to run a command line function to delete files. if I can figure that out, I might have it sorted.

So, I have it running, but I don't actually hear any sound, and it seems to playback a wave every 5 - 7 seconds.

I have HDMI hooked up, and a headset plugged in, but don't hear anything on either source. Any ideas?

HDMI killed the headset and didnt play. I got the sound working by unplugging the monitor. almost have it solved. Once I have something working, I will put both codes up as I have had to make changes to both.

[mikey11]

After much more playing around trying to get an easy way to make the two separate processes play nice, I gave up for tonight.

I have a gut feeling that the soundscape generation is maxing out the pi's cpu and resulting in the 5-7 second wait between frames.

I'm not sure there is too much more that I can do at the moment. I will think about ways to tackle that problem, and if I come up with anything, this place will be the first to know.

[mikey11]

I'm not certain, but I think offloading sound playback to a USB solution might free up the CPU even more.

I'm currently researching if this will help. Some solutions are cheap, so staying sub $100 is still possible:

http://www.adafruit.com/products/1475#Description

and feedback in numerous online forums about different USB sound solutions for the rpi indicate that they smooth out sound playback, and eliminate pops and hisses present with the onboard sound.

I just want to know if it will help with the CPU usage because a frame every 5-7 seconds is not viable.

some quote from some forum:

Overclocking

With the raspi-config tool or by editing /boot/config.txt directly it is possible to overclock various parts of the RPi (CPU, GPU, SDRAM). It is recommended to only use the overclock options offered by raspi-config. A common issue with overclocking is SD card corruption. You can prevent this from happening by never using a core_freq (GPU frequency) value higher than 250. If you really want to use a higher core_freq value you could either use a SD card that doesn't show this behavior or put your root filesystem on an USB drive. Overclocking provides more processing power and since it doesn't void your warranty and you don't need to add fans or a heatsink you could consider it a must for doing audio related things on your RPi.

I'm going to give this a go, and see if it helps sometime this week.


Another thing I've come across is that the audio playback on the rPi is only 11 bit at 48 kHz. I think the sound scape generator is making 16 bit files.

I am not too handy with the c++ code, but I think that represents an opportunity for significant processor savings. I will read more about this.

I found this post very informative:

http://wiki.linuxaudio.org/wiki/raspberrypi

some information suggests omxplayer can do sound playback through some utilization of the GPU. If the GPU is not offlimits due to the camera, or is being underutilized and the two processes can work simultaneously on it the info I found says omxplayer can do flawless sound playback with only 12% CPU utilization. This is something worth exploring.

[PranavLal]

Hi Mike,

I have tried using a USB soundcard. No go. I tend to keep things simple. It is not so much the playing that is maxing the pi but the actual soundscape generation. That requires optimization which I do not know how to do. As for tying your processes together, I used a python script or a shell script. I cannot remember which one. The script worked as follows.
1. It ran the camera capture bit.
2. It then ran the sonification code.


The whole thing was in a loop.

By the way, do you have a debugger such as gdb working? If yes, step through the C code. You will find the soundscape generation really taking time.

Pranav

[mikey11]

Pranav,

running the raspistill process to generate a single frame in a loop introduces a 0.8 second delay in addition to the processing and sound playback. This is why I use the timelapse function. The timelapse function incurs the penalty only when starting the raspistill process. Afterwards, frames can be grabbed at ~100-200ms frequency for extended periods of time.

I am unfamiliar with gdb or really, using C in general. I find that language finicky, and everything I've ever tried to do has not worked.

As for synching up the processes, it won't happen easily. The reason why is that raspistill is out of my control, but it is the single most efficient way to generate image captures without bunging up the cpu.

If you use raspistill and only capture to one filename, the file is actually unavailable for large portions of time when the C routine is looking for it.

However, raspistill will capture to sequentially increasing filenames using the flag %04d in the filename.

what I need to do is take the most recently generated, complete file, and then have the C code process it.

My plan to achieve this is to have the python loop I made look in /dev/shm, and rename the most recently completed file to /dev/shm/picture0001.jpg

the C process will look for /dev/shm/picture0001.jpg after loading the file, but before the soundscape is generated, the C process will drop down and delete all files in /dev/shm. This lets raspistill keep populating the directory with new images (picture0002-picture9999.jpg) while the soundscape is being played back. As long as the capture frequency is faster than the sound playback, this should be golden. so I am setting the timelapse to 300ms for testing purposes.

This will make the python script, raspistill, and the C process play nicely.

Now, to address the C process.

You indicate that the majority of time is spent on calculations preparing the sound file.

I've identified some areas for improvement already. I don't know how to implement all of these yet, but some I do know how to do.

1. raspistill can make the captures in grayscale using GPU functions. the C process can stop converting to grayscale. The GPU will be faster, and free up more CPU time for sound generation. I will implement this.

2. The sound generation currently says that it is making a 16 bit file. While I understand that 16 bit sound would provide advantages to the listener over 8 bit, I also have found in forum after forum that the rPi is only capable of 48kHz 11 bit sound. If the sound routines can be partially 'lo-fi' to match the rPi, the CPU will be less engaged in creating the sound file. I don't know how to implement this yet, but it looks like the most proximal goal for reducing CPU load, thus increasing playback frequency.

There are other things about the C process that I want to look at. I want it to be headless (ie. no keyboard interaction, and no screen output/interference). I want it to run constantly in the background.

After these optimizations have been made, I want to see how things round out. Can we get the playback every 5-7 seconds down to once every 3 seconds?

I would be over the moon if we got there, and it would be a huge milestone on the way to a once/sec frequency.

Last but not least:

http://raspberrypi.stackexchange.com/qu ... nt-support

I need to confirm that I have the hard float engaged, and that the math being done in the C process is taking advantage of it. If it isn't, then this is also key to making things sing.

Here is how to check:
http://www.raspberrypi.org/phpBB3/viewt ... 33&t=25350

but I won't know for a while due to obligations.