Azsorlex26
Posts: 16
Joined: Sat Feb 10, 2018 8:41 am
Location: Australia

How do I alter files using commands?

Mon Jul 23, 2018 11:51 pm

Hi there. I'm attempting to make a script that will fully set up my raspberry pi from a clean installation of Raspbian Lite. This mostly involves using automatic commands like sudo apt-get update or sudo apt-get install ..., however the setup process also involves me altering files such as /etc/fstab and /etc/dhcpcd.conf. I know that files can be created and have text added to them by using echo/printf '[file content]' > [file name], and have text added on to existing files by using echo/printf '[file content]' >> [existing file's name], but my questions are: how can certain lines be edited or have text inserted at them, and how can you delete lines from files using commands?

User avatar
The Traveler
Posts: 360
Joined: Sat Oct 21, 2017 3:48 pm

Re: How do I alter files using commands?

Tue Jul 24, 2018 12:09 am

Couple of ways to do it: https://www.linuxquestions.org/question ... on-377340/

Basic shell scripting: https://en.wikibooks.org/wiki/Bash_Shell_Scripting and how to "glue" together linux command line programs to form larger, more complex applications. It's really the heart of the OS if you find yourself needing to do more at the command line level.
Retired IT professional, C programmer and "beardie weirdie".
RPi interests: Developing an Infinite Improbability Drive
“Thinking outside of the box allows you to get rewards outside of your reach.” Matshona Dhliwayo

User avatar
Paeryn
Posts: 2632
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: How do I alter files using commands?

Tue Jul 24, 2018 12:34 am

One tool of choice for making non-interactive edits of files is sed (Stream EDitor). Read it's man page (man sed) and/or ask your favourite search engine for guides on how to use it. You can look at the raspi-config shell script to see how they use it in modifying /boot/config.txt

There are of couse numerous other programs that you can use as well depending on what type of changes you are wanting to make, mix and match them as you see fit.
She who travels light — forgot something.

DirkS
Posts: 9895
Joined: Tue Jun 19, 2012 9:46 pm
Location: Essex, UK

Re: How do I alter files using commands?

Tue Jul 24, 2018 2:24 am

You could also have a look at pi-gen (https://github.com/RPi-Distro/pi-gen)
RPF uses it to create their Raspbian images.

I use a customised version to create my Lite images, with added programs, activate SSH, add wpa_supplicant.conf, etc.

tpyo kingg
Posts: 583
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: How do I alter files using commands?

Tue Jul 24, 2018 7:43 am

To expand on the sed option, look at the a, i, c, and d commands. Additionally the -i option allows you to save a backup copy.

The i command can insert anywhere. Here the i inserts some lines at the head of the file, before line 1:

Code: Select all

sed '1i\fallback to static profile on eth0\ninterface eth0\nfallback static_eth0' /etc/dhcpcd.conf
The a command works about the same but appends instead. Here the $ pattern stands for the last line in the file:

Code: Select all

sed '$i\fallback to static profile on eth0\ninterface eth0\nfallback static_eth0' /etc/dhcpcd.conf
The c command can replace a line.

Code: Select all

sed '/^slaac private/c\slaac hwaddr' /etc/dhcpcd.conf
You'll have to look around for guides and tutorials that match your level, but eventually the Sed - An Introduction and Tutorial by Bruce Barnett is a good place to end up.

If things start getting too complex, however, then it might be time to escalate to perl which excels at text processing and has many modules for parsing different configuration files specifically.

Azsorlex26
Posts: 16
Joined: Sat Feb 10, 2018 8:41 am
Location: Australia

Re: How do I alter files using commands?

Tue Jul 24, 2018 12:37 pm

tpyo kingg wrote:
Tue Jul 24, 2018 7:43 am
To expand on the sed option, look at the a, i, c, and d commands. Additionally the -i option allows you to save a backup copy.

The i command can insert anywhere. Here the i inserts some lines at the head of the file, before line 1:
Code: Select all

sed '1i\fallback to static profile on eth0\ninterface eth0\nfallback static_eth0' /etc/dhcpcd.conf
The a command works about the same but appends instead. Here the $ pattern stands for the last line in the file:
Code: Select all

sed '$i\fallback to static profile on eth0\ninterface eth0\nfallback static_eth0' /etc/dhcpcd.conf
The c command can replace a line.
Code: Select all

sed '/^slaac private/c\slaac hwaddr' /etc/dhcpcd.conf
I'm still not getting this. The sources I've found aren't entirely making sense. From what I understand, sed doesn't write to the file, which is what I need. I was testing a method for inserting text into a file, and this is what I was able to come up with:

Code: Select all

#!/bin/bash

sed '4i\Testing' target > target1
rm target
mv target1 target
It essentially takes the info of target and writes the sed output to a new file. The original is then deleted and the new file is renamed to the original. But knowing me, I've only come up with an inefficient solution. What's the best way of doing this? Also, I couldn't figure out how to delete from files. What I want to do is delete the last five lines of a file.
Paeryn wrote:
Tue Jul 24, 2018 12:34 am
You can look at the raspi-config shell script to see how they use it in modifying /boot/config.txt
Where do I find that? Also, I found this source for editing the raspi-config options: https://raspberrypi.stackexchange.com/q ... nfig-setup, and the only one I don't get is the password changing one. What would be an example of this?
Last edited by Azsorlex26 on Tue Jul 24, 2018 12:55 pm, edited 1 time in total.

Ernst
Posts: 1213
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: How do I alter files using commands?

Tue Jul 24, 2018 12:48 pm

A very interesting option that works with many unix commands is "--help"

Code: Select all

sed --help
will show that there is an option "-i"

Code: Select all

  -i[SUFFIX], --in-place[=SUFFIX]
                 edit files in place (makes backup if SUFFIX supplied)
with this knowledge we can replace your script:

Code: Select all

#!/bin/bash

sed '4i\Testing' target > target1
rm target
mv target1 target
with the one liner:

Code: Select all

 sed -i '4i\Testing' target 
The road to insanity is paved with static ip addresses

tpyo kingg
Posts: 583
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: How do I alter files using commands?

Tue Jul 24, 2018 12:55 pm

Azsorlex26 wrote:
Tue Jul 24, 2018 12:37 pm
I'm still not getting this. The sources I've found aren't entirely making sense. From what I understand, sed doesn't write to the file, which is what I need. I was testing a method for inserting text into a file, and this is what I was able to come up with:

Code: Select all

#!/bin/bash

sed '4i\Testing' target > target1
rm target
mv target1 target
It essentially takes the info of target and writes the sed output to a new file. The original is then deleted and the new file is renamed to the original. But knowing me, I've only come up with an inefficient solution. What's the best way of doing this? Also, I couldn't figure out how to delete from files. What I want to do is delete the last five lines of a file.
The redirect which you are using there is a good way to save the output. I'd say it is the best for this situation, especially once pipes come into play. If you are just using sed alone then the -i option is sometimes good.

Remember that the guiding idea is that each component does one thing very well and you chain them together like Lego blocks to build something complex. For that you can use a data pipe |

If you want to delete the last five lines of the file and do no other modification to the file, then you can bring in 'tac' which flips the file around. Then you can delete the first five lines of that and flip it around again.

Code: Select all

tac target | sed '1,5d' | tac > target1;
sed is good at a few things and is rather genius considering how many decades ago it was written. However, it has some limitations and counting backwards is one thing it can't do. The reason for that is no matter how you slice it and which language you use, your script will have to read through the whole file once to know how many lines there are before it can start deleting. So reversing the input first with tac allows it to count forward.

You can also have sed delete a span of lines starting at a pattern and then continuing through to the next pattern or else the end of the file.

Code: Select all

sed '/^startpat/,$d' < infile.txt > outfile.txt
Or

Code: Select all

sed -i.orig '/^startpat/,$d' < infile.txt > outfile.txt

Azsorlex26
Posts: 16
Joined: Sat Feb 10, 2018 8:41 am
Location: Australia

Re: How do I alter files using commands?

Tue Jul 24, 2018 1:17 pm

tpyo kingg wrote:
Tue Jul 24, 2018 12:55 pm
The redirect which you are using there is a good way to save the output. I'd say it is the best for this situation, especially once pipes come into play. If you are just using sed alone then the -i option is sometimes good.
Going off what you said, I was able to simplify the insertion command to:

Code: Select all

sed -i '4i\Testing' target
So yes, my initial implementation was inefficient. Thanks for your help.
tpyo kingg wrote:
Tue Jul 24, 2018 12:55 pm

Code: Select all

tac target | sed '1,5d' | tac > target1;
The way you've put this, it seems that the only way of implementing deletion is through my initial method of writing a new file, deleting the old one, and then renaming the new one. Is this true? I've already tried experimenting with this but got nothing.

User avatar
Paeryn
Posts: 2632
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: How do I alter files using commands?

Tue Jul 24, 2018 1:42 pm

Azsorlex26 wrote:
Tue Jul 24, 2018 12:37 pm
I'm still not getting this. The sources I've found aren't entirely making sense. From what I understand, sed doesn't write to the file, which is what I need. I was testing a method for inserting text into a file, and this is what I was able to come up with:

Code: Select all

#!/bin/bash

sed '4i\Testing' target > target1
rm target
mv target1 target
It essentially takes the info of target and writes the sed output to a new file. The original is then deleted and the new file is renamed to the original. But knowing me, I've only come up with an inefficient solution. What's the best way of doing this? Also, I couldn't figure out how to delete from files. What I want to do is delete the last five lines of a file.
Paeryn wrote:
Tue Jul 24, 2018 12:34 am
You can look at the raspi-config shell script to see how they use it in modifying /boot/config.txt
Where do I find that? Also, I found this source for editing the raspi-config options: https://raspberrypi.stackexchange.com/q ... nfig-setup, and the only one I don't get is the password changing one. What would be an example of this?
By default sed writes the resultant stream to its stdout, if you want to edit the file in-line then pass -i to it. To save a backup of the original at the same time the characters directly following the -i are appended to the filename, e.g. use -i.bak to append the suffix .bak for the backup filename, note do not put a space between the i and the suffix.

Code: Select all

## target will be edited and target.bak will be a copy of the original version of target
sed -i.bak '4i\Testing' target
To delete the last 5 lines I'd use head, the standard gnu head allows negative values to mean "until n lines from the end"

Code: Select all

head -n -5 target > target.new
mv target.new target
sed can do it but it isn't necessarily obvious

Code: Select all

sed -i.bak -n ':a;1,5!{P;N;D};N;ba' target
To find the location of a program on your system you can use which, it shows you the full path to the program.

Code: Select all

pi@rpi3:~/Public $ which raspi-config
/usr/bin/raspi-config
She who travels light — forgot something.

tpyo kingg
Posts: 583
Joined: Mon Apr 09, 2018 5:26 pm
Location: N. Finland

Re: How do I alter files using commands?

Tue Jul 24, 2018 6:20 pm

Paeryn wrote:
Tue Jul 24, 2018 1:42 pm
sed can do it but it isn't necessarily obvious

Code: Select all

sed -i.bak -n ':a;1,5!{P;N;D};N;ba' target
Neat. That's a different approach. It looks like a FIFO queue 5 deep which takes advantage of P and D only working up to the next new line character.

User avatar
Paeryn
Posts: 2632
Joined: Wed Nov 23, 2011 1:10 am
Location: Sheffield, England

Re: How do I alter files using commands?

Tue Jul 24, 2018 7:49 pm

tpyo kingg wrote:
Tue Jul 24, 2018 6:20 pm
Paeryn wrote:
Tue Jul 24, 2018 1:42 pm
sed can do it but it isn't necessarily obvious

Code: Select all

sed -i.bak -n ':a;1,5!{P;N;D};N;ba' target
Neat. That's a different approach. It looks like a FIFO queue 5 deep which takes advantage of P and D only working up to the next new line character.
Basically, yes (though it's a six line FIFO). Description of how it works for those not versed in sed :-
sed reads each line in turn into the pattern buffer and executes the expression given on it. This code only uses that for the initial line, every other line is read in manually with the N command by the expression because otherwise the pattern buffer will be overwritten each line which we do not want (we want to add to it rather than replace it).
  • :a This is just a label that we jump back to later.
  • 1,5!{P;N;D} the 1,5! executes the {...} block only on lines other than 1 to 5 (so from line 6 onward).
    • P Print the first line of the pattern buffer.
    • N Append the next line of input to the pattern buffer.
    • D Delete the first line of the pattern buffer and go back to the start.
  • N Append the next line to the pattern buffer.
  • ba Branch back to label a.
Those final two commands N;ba serve to load the buffer with the first six lines of the input, they will only be executed on the first 5 lines because after that the previous block executes which either jumps back to the start or exits totally.
When there are no lines left for N to read it will terminate so the last five lines, which have currently only been read into the pattern buffer, never get output.
She who travels light — forgot something.

jbudd
Posts: 944
Joined: Mon Dec 16, 2013 10:23 am

Re: How do I alter files using commands?

Wed Jul 25, 2018 12:54 am

Paeryn's head -n -5 target >target.new; mv target.new target looks the most elegant solution.

Here is another approach using ed, an editor so old it was designed to work with a teleprinter!
Probably it's only advantage is that it modifies the original file directly.

Code: Select all

ed target <<- !
$
d
d
d
d
d
w
q
!
or

Code: Select all

ed target <<- !
$-5,$ d
w
q
!

Return to “Beginners”