kmabry2007
Posts: 8
Joined: Tue Apr 24, 2018 1:59 am

automating writing sd cards and USB drives with a PI computer only.

Thu Apr 26, 2018 7:21 pm

Hello everyone! This is my first post; thank you for having me. Like most of the beginners, I am pouring though the boards searching and learning. Since I only have a raspberry pi 3 as my only functioning pc right now., I faced a dilemma I did not have a windows machine accessible to use for writing sd cards and usb devices to install operating systems. I found the sub-heading under installations to be a great, rich resource. However, I hated the fact that I had to do my research over and over for each writing, dependent on my situation. That is why I wrote the ImageRite program. It is a python based program that will utilize the included dd function to write the image of a Pi operating system (a fresh install only) to an sd card, or usb device. Inspired by the pi foundations own instructions, I wrote two programs located here. https://github.com/kmabry2007/ImageRite Please note that I would recommend sd card sizes between 8 and 32 GB in size, similarly for USB drives. My thanks to the fine guidelines from the raspberry pi foundation, and I hope that if you use my program(s) that they would enhance your Pi enjoyment!!

Sincerly,

kmabry2007

User avatar
KLL
Posts: 1453
Joined: Wed Jan 09, 2013 3:05 pm
Location: thailand
Contact: Website

Re: automating writing sd cards and USB drives with a PI computer only.

Fri Apr 27, 2018 7:11 am

hi @kmabry2007
-A-
i understand that there is need for RPI internal tools, what not need a external PC.
  • -1-the SD card copier tool was a real HIT
  • -2-but it miss the creation of .img or .img.zip files.
  • -3- the @RonR backup tool might help out here.
  • -4- the new setup from .img .img.zip
    like you did is also needed, might even call it RPI_etcher.
-B-
sorry not tested it until now

-C-
but i take a look at the code, and it looks like you are fighting with the variable space in python
( again sorry i not play python2, now use python3 )
-C1- your structure is like:

Code: Select all

#!/usr/bin/env python3
#see at https://github.com/kmabry2007/ImageRite/blob/master/ImageRite.py
# i want check his structure:


myvar='mytext'

def my_sub1():
        print("sub1: "+myvar)
        return

def my_sub2(myvar):
        print("sub2: "+myvar)
        return myvar

def main(myvar):
        print("main: "+myvar)
        myvar = (input("set myvar: "))
        my_sub1()
        my_sub2(myvar)

if __name__ == '__main__':
        main(myvar)



"""makes:
./fromPY2.py
main: mytext
set myvar: TEST
sub1: mytext
sub2: TEST
"""
-C2- but you could simply use ( first step only change structure) and put variable and function declaration inside main:

Code: Select all

#!/usr/bin/env python3
def main():
        myvar='mytext'

        def my_sub1():
                print("sub1: "+myvar)
                return

        def my_sub2(myvar):
                print("sub2: "+myvar)
                return myvar

        print("main: "+myvar)
        myvar = (input("set myvar: "))
        my_sub1()
        my_sub2(myvar)

if __name__ == '__main__':
        main()

"""makes:
./clean.py
main: mytext
set myvar: TEST
sub1: TEST
sub2: TEST
"""
-C3- add:
if you make a function ( procedure or what it is called )
what changes variables, only then you must use return like

def sub(a):
return a

and also you must use the returned value with
a = sub(a)

Code: Select all

 cat work.py
#!/usr/bin/env python3
def main():
        myvar='mytext'

        def my_sub(myvar):
                print("-1- sub: "+myvar)
                myvar = "myNEWtext_from_sub"
                print("-2- sub: "+myvar)
                return myvar

        print("-1- main: "+myvar)
        myvar = (input("set myvar: "))
        print("-2- main: "+myvar)
        my_sub(myvar)
        print("-3- main: "+myvar)
        myvar = my_sub(myvar)
        print("-4- main: "+myvar)

if __name__ == '__main__':
        main()

"""makes:
./work.py
-1- main: mytext
set myvar: TEST
-2- main: TEST
-1- sub: TEST
-2- sub: myNEWtext_from_sub
-3- main: TEST
-1- sub: TEST
-2- sub: myNEWtext_from_sub
-4- main: myNEWtext_from_sub
"""
hope i could help.
Last edited by KLL on Sat Apr 28, 2018 3:02 pm, edited 2 times in total.

kmabry2007
Posts: 8
Joined: Tue Apr 24, 2018 1:59 am

Re: automating writing sd cards and USB drives with a PI computer only.

Fri Apr 27, 2018 7:31 am

:D thanks for the input!! I would love to restructure and tidy up my code. I will be working your suggestion soon in my next release. The main problem I found in writing the program was passing arguments between functions. Variables that were defined in a functioning were not getting passed! As in disappearing, because they were not setup in main function.

Again thank you for the input, and I will update my code soon!

User avatar
KLL
Posts: 1453
Joined: Wed Jan 09, 2013 3:05 pm
Location: thailand
Contact: Website

Re: automating writing sd cards and USB drives with a PI computer only.

Fri Apr 27, 2018 7:43 am

and as it is a command line tool a next step could be to learn/use this

kmabry2007
Posts: 8
Joined: Tue Apr 24, 2018 1:59 am

Re: automating writing sd cards and USB drives with a PI computer only.

Fri Apr 27, 2018 6:23 pm

Wow! you have given me much to think about, but in a GREAT way!! I do not see how it works yet, but I will learn!! Thank you so much. If this post needs to be moved to another thread, I certainly understand... :D Oh.. and I think I understand the first example, now. So this will be a great learning tool for me to work on my coding. Not very much structured coding experience here. So Into the tutorials I will search!!

- kmabry2007

kmabry2007
Posts: 8
Joined: Tue Apr 24, 2018 1:59 am

Re: automating writing sd cards and USB drives with a PI computer only.

Fri May 04, 2018 6:23 am

Well, KLL I think I have implemented your suggestions! ImageRite_v1.4.py is in testing and listed on https://github.com/kmabry2007/ImageRite
I broke from the released master and did some upgrading. I will merge it all into the master branch once testing has completed. Also added the ability to write .iso files as well. Now it is more automated and streamlined with only input and output files as arguments

usage

ImageRite_v1.4.py -i <inputfile> -o <outputfile>

enjoy!

-kmabry2007

jahboater
Posts: 4923
Joined: Wed Feb 04, 2015 6:38 pm

Re: automating writing sd cards and USB drives with a PI computer only.

Fri May 04, 2018 7:29 am

Some comments ....
What does the "truncate" do here and why are you using it?

To compare raw .img files I would use "cmp", not "diff" which does a line by line compare of text files.
If you could somehow get the SHA-256 checksum down and use "sha256sum" to verify the downloaded .zip file it would be good.

Why do you think that 4M is a good block size? I know it is common, but "cp" is designed specifically for the job and uses 128KB blocks.
"cp" also sets fadvise( FADV_SEQUENTIAL ) to optimize the read prefetch. Because you want to overlap the reads with the writes, I don't think having too large a block size is a good idea - in the worst case if you set the block size to the size of the file it would be very slow as no overlap would be possible at all.

sudo cp xxxx.img /dev/sda

is very much simpler than dd and slightly faster last time I measured it.
You still need a "sync" at the end, in case someone pulls out the SD card too soon.
You can also "unzip" to stdout and direct that straight to the SD card.

The script does seem grossly over-complicated !

kmabry2007
Posts: 8
Joined: Tue Apr 24, 2018 1:59 am

Re: automating writing sd cards and USB drives with a PI computer only.

Sun May 06, 2018 6:03 am

The 4M is warranted here due to speed of writing involved. Also, that is suggested by the guide presented here under 'installations'. This was meant as an exercise to model and automate this process for beginners to learn from. I personally have learned much from this experience.

I have upgraded the code to streamline it and v1.4 utilizes command line arguments of an input file and an output file only. It is in testing at this time.

Why truncate?
When the image is written, it not the same size as the original file. It is the size of the disk it is written into. Truncate removes the extra size and allows for a proper diff to be performed. Again other methods could have been used but I chose to mirror the 'installation ' guide.

- kmabry2007

jahboater
Posts: 4923
Joined: Wed Feb 04, 2015 6:38 pm

Re: automating writing sd cards and USB drives with a PI computer only.

Sun May 06, 2018 8:31 am

kmabry2007 wrote:
Sun May 06, 2018 6:03 am
Why truncate?
When the image is written, it not the same size as the original file. It is the size of the disk it is written into. Truncate removes the extra size and allows for a proper diff to be performed. Again other methods could have been used but I chose to mirror the 'installation ' guide.
Yes, interesting.
There is no file system on the raw disk.
Does it still fail if you use "cmp" which is better for binary files?

kmabry2007
Posts: 8
Joined: Tue Apr 24, 2018 1:59 am

Re: automating writing sd cards and USB drives with a PI computer only.

Tue May 08, 2018 6:48 am

I haven't had time to test that yet. Thanks for the idea!

-kmabry2007

kmabry2007
Posts: 8
Joined: Tue Apr 24, 2018 1:59 am

Re: automating writing sd cards and USB drives with a PI computer only.

Sun May 20, 2018 1:50 am

Hi everyone!

I have published version 1.5 of the ImageRite program on github!

https://github.com/kmabry2007/ImageRite

It has been tested and updated from version 1.4. it utilizes the Command line argument structure as in version 1.4 and has added the ability to write .iso files to usb drive for use in upgrading a pc from "cough" Windows 10 to debian with the raspberry pi desktop. Try it and send me feedback to
K5AID@hughes.net

- kmabry2007

jahboater
Posts: 4923
Joined: Wed Feb 04, 2015 6:38 pm

Re: automating writing sd cards and USB drives with a PI computer only.

Sun May 20, 2018 6:31 am

Thanks for posting this.

I see you set "str" in line 114 and then set it again at line 117 which doesn't look right.

kmabry2007
Posts: 8
Joined: Tue Apr 24, 2018 1:59 am

Re: automating writing sd cards and USB drives with a PI computer only.

Tue May 22, 2018 9:08 pm

Hey buddy! Thank you for your keen eye!
I have removed the extraneous lines of code.
This is the correct line. It does the unzip and redirects
It to write simultaneously to the sd card or USB

str = "unzip -p "+inputfile+ "| sudo dd bs=4M of= "+outputfile+" status=progress conv=fsync"
print("Unzipping and writing "+inputfile+" to "+outputfile)
print().

Thanks again!

- kmabry2007

Return to “Beginners”