Arp
Posts: 26
Joined: Wed Sep 11, 2013 1:26 pm
Location: NRW, Germany

Saving Timelapse directly on remote server?

Thu Jan 09, 2014 6:37 pm

Hi,

I plan to let raspivid run over the weekend. I want it to directly save the images onto my owncloud server. I would create a folder and mount it as webdav folder so everything goes automatically into the cloud. Now, I dont know how fast the internet connection will be. Could take a couple of seconds. My question is, when raspivid captures an image, does the waiting start after the file is saved, or is this done in parallel.
For example, if I capture every 10 seconds, but it takes 15 seconds to save the file on the remote folder, does it actually wait 15 seconds, or will it start the next exposure while the old one is saving?

Or would it be better to have a second script checking if there is a new file and start uploading it in parallel?

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: Saving Timelapse directly on remote server?

Thu Jan 09, 2014 7:28 pm

If you do the math, or maybe just intuition who knows, it cannot work.

Honsolgen
Posts: 20
Joined: Mon Nov 25, 2013 8:35 am
Location: Honsolgen - Germany
Contact: Website

Re: Saving Timelapse directly on remote server?

Thu Jan 09, 2014 8:48 pm

raspivid && images?

Denk nochmal drüber nach Meister aus NRW oder ist da schon Karneval ? ...

User avatar
waveform80
Posts: 305
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK

Re: Saving Timelapse directly on remote server?

Thu Jan 09, 2014 10:08 pm

Arp wrote:Hi,

I plan to let raspivid run over the weekend. I want it to directly save the images onto my owncloud server. I would create a folder and mount it as webdav folder so everything goes automatically into the cloud. Now, I dont know how fast the internet connection will be. Could take a couple of seconds. My question is, when raspivid captures an image, does the waiting start after the file is saved, or is this done in parallel.
For example, if I capture every 10 seconds, but it takes 15 seconds to save the file on the remote folder, does it actually wait 15 seconds, or will it start the next exposure while the old one is saving?
I assume we're talking about raspistill (for images) rather than raspivid (which is for video). Anyway, the timelapse mode in raspistill simply keeps track of a running clock and captures when it reaches the appropriate time. It also knows when it's missed a frame (have a look here for the details).

Incidentally, when you say it'll take a couple of seconds to upload I assume we're not talking full resolution captures here - those'll take a good bit more than a few seconds to upload! I'm also not sure how reliable a network file-system (I'm guessing you're using something like NFS or SSHFS?) will be over that sort of time-period.

As an alternative you might want to have a look at some of the recipes for the picamera library (full disclosure: this is a shameless plug; I'm the author) - in particular the ones to do with sending images over a network socket.

Dave.

towolf
Posts: 421
Joined: Fri Jan 18, 2013 2:11 pm

Re: Saving Timelapse directly on remote server?

Fri Jan 10, 2014 12:47 am

Well, personally I’ve been writing full 5mp jpegs per second (or rather 1.2s on average) onto NFS for a month or so. But that is on Gigabit Ethernet. That amounts to 120G of jpegs per day that are cooked down to H264 every morning over a few hours. All because the FoV is not wide enough in video mode.

User avatar
waveform80
Posts: 305
Joined: Mon Sep 23, 2013 1:28 pm
Location: Manchester, UK

Re: Saving Timelapse directly on remote server?

Fri Jan 10, 2014 2:05 am

towolf wrote:Well, personally I’ve been writing full 5mp jpegs per second (or rather 1.2s on average) onto NFS for a month or so. But that is on Gigabit Ethernet. That amounts to 120G of jpegs per day that are cooked down to H264 every morning over a few hours. All because the FoV is not wide enough in video mode.
Yes, must admit I was assuming the OP was on wifi - but ethernet should handle it.

Going slightly off-topic for a sec, I've recently got full-FoV video working in picamera too (it'll be in release 1.0). It does require scaling down from the full-res to something smaller, and it only runs at 15fps, but it does record happily (recipe here).

Dave.

Arp
Posts: 26
Joined: Wed Sep 11, 2013 1:26 pm
Location: NRW, Germany

Re: Saving Timelapse directly on remote server?

Fri Jan 10, 2014 9:50 am

Sorry. I meant raspistill, of course.

@waveform80
Yes, full resolution. ~2.3 MB per image. But I will do this from my office window (has a nice view :)) and we have a decent internet connection, ranging from 500 kb/s to >2 mb/s, depending on the overall traffic. So, uploading a full res image to my webserver would not take more than a couple of seconds. But it *could* take more seconds than my timelaps interval.

I'm thinking about a python script that runs in parallel and always checks if there is one more image (to make sure it does not start upload an image that was not yet completely saved) and then starts uploading.

poing
Posts: 1131
Joined: Thu Mar 08, 2012 3:32 pm

Re: Saving Timelapse directly on remote server?

Fri Jan 10, 2014 12:06 pm

the images have a .jpg~ extension while written, so no problem with half-written ones.

Try something like this (not tested, cut and pasted from a longer script), just fill in where the images are saved by raspistill and where they should be copied:

Code: Select all

#!/usr/bin/python

import os
import time

# define constants
incomingdir=""
targetdir=""

# start infinite loop
while True:

	# see if new images have arrived in incomongdir and copy them to targetdir
	for files in os.listdir(incomingdir):
		if files.endswith((".jpg", ".JPG")):
			print files
			shutil.copy(incomingdir+"/"+files, targetdir+"/"+files)
			time.sleep(.5)

Needles to say, if you constantly take images at a 10s interval and copying constantly takes 15s per image you have a problem ;-) But this script should buffer temporary slowing of the connection.

Arp
Posts: 26
Joined: Wed Sep 11, 2013 1:26 pm
Location: NRW, Germany

Re: Saving Timelapse directly on remote server?

Fri Jan 10, 2014 4:01 pm

Hi,

Yes, my script looked very similar, but there was a problem. I wanted to copy to folder, which was mounted as a webdav folder do my owncloud server. I was able to connect, but every cp command from within the python script would create a 0 byte file, and the actual copying of the data was done when the script was ended (by force). I dont know if this because of python, or owncloud, or webdav. For the moment, I solved it by using scp and uploading the files to my office machine. I need the upload, because it will be roughly 50 GB of image data over the weekend, and my SD card has only 14 GB :)

Honsolgen
Posts: 20
Joined: Mon Nov 25, 2013 8:35 am
Location: Honsolgen - Germany
Contact: Website

Re: Saving Timelapse directly on remote server?

Sat Jan 11, 2014 11:08 am

try the -q switch.
my daylight shots are done with -q 13 ... the files are much smaller and I do not see a visible difference in image quality with the 2592x1944 resolution.

http://adr.dyndns.org:82/webcam/raspberry-pi/#/infos

Return to “Camera board”