Tech-tech Musings: sed

Ah, sed. What a wonderfully confusing command you are~

This was one of those commands that honestly intimidated me a bit when I was in class for my Linux certification. It’s odd, has weird syntax, can be confusing to get working, and is often unneeded for basic command line tasks that most people will be doing.

But I’ve found ‘sed’ to be a really, really handy command and I sincerely enjoy using it! I’d like to explain why in this little post and hopefully give some folks an idea on how they can use it.

First, let’s take a look at sed’s man page.

NAME
sed – stream editor for filtering and transforming text

SYNOPSIS
sed [OPTION]… {script-only-if-no-other-script} [input-file]…

DESCRIPTION
Sed is a stream editor. A stream editor is used to perform basic text transformations on an input stream (a file or input from a pipe‐
line). While in some ways similar to an editor which permits scripted edits (such as ed), sed works by making only one pass over the
input(s), and is consequently more efficient. But it is sed’s ability to filter text in a pipeline which particularly distinguishes it
from other types of editors.

So, from this we understand that ‘sed’ means “stream editor.” It’s designed to perform text edits of a “stream” on the Linux command line. “Well, that’s cool and all Jesse, but what even is a stream?” What even, indeed. Well, to understand that, you need to understand that the Linux command line has these three magical things called STDIN, STDOUT, and STDERR.

STDIN is easy enough to understand. This is “standard input,” the stuff you type into the command line to make things go, like commands or responses to a prompt/program. You type it in, press enter, and it’s input into the computer.

STDOUT is also fairly straightfoward. This is “standard output.” Any time that you run a command, you will generally get some form of text output to your command line, be it general information (like a directory listing), file contents, system information, or some sort of output that you designed your program to put out.

STDERR is a little weirder. As you might have gathered by the trend so far, this is “standard error.” This is the text that will be printed to your terminal if there was some sort of error with your command, program, or what not. The reason that STDOUT and STDERR are separated is so that if you are doing something with the text of STDOUT in your command, and then you have a problem with your command or your manipulation of STDOUT, the error stuff will remain unmanipulated in its own separate stream.

Aha! So there’s the concept now! STDOUT and STDERR are essentially text streams to the command line. It’s the text that flows to your screen, which can then be manipulated to whatever end in your command that you’re running, depending on what you’re trying to do. They can be redirected as input to files, modified by other commands, or just left to sit on the screen. I’m going to show you some examples below.

 

Example of STDOUT:

“free -m” — this command will tell me the memory usage of my system (in megabytes) at the given moment I run it. My typing it in and pressing enter is STDIN. All the goo that comes out is STDOUT.

[prannon@prannondesk ~]$ free -m
total used free shared buffers cached
Mem: 9991 9843 147 97 165 595
-/+ buffers/cache: 9082 908
Swap: 5063 14 5049

Example of STDERR:

Now I’m going to run this command with some invalid options. These options weren’t built into the program, so whatever I get out of it should throw some errors. The goo that comes out of this is STDERR.

[prannon@prannondesk ~]$ free -sdf
free: seconds argument `df’ failed

“Well, that’s all nice Jesse, I guess I get what text streams are now. But weren’t we going to talk about sed or something?”

Yeahsure. Let’s talk about that! So, one of the things that we can do on the Linux command line is take the STDOUT of one command and make it the STDIN of another command. We do this via a process called “piping.”

“PIPING???”

Yes, piping. This thing… >>   |

It’s that weird character that’s in that key just above your enter key on your keyboard. No one ever really uses it for anything, but we use this all the time to link different commands together. The pipe basically takes the STDOUT of a previous command and turns it into the STDIN of another command. Let me show you~

I’m going to use the “free -m” command from above, and pipe it to some weirdness that’ll take the word “free” in the output, and then run that as a command in and of itself. Watch…

[prannon@prannondesk ~]$ free -m
total used free shared buffers cached
Mem: 9991 9807 183 117 158 631
-/+ buffers/cache: 9017 974
Swap: 5063 18 5045
[prannon@prannondesk ~]$ free -m | grep -o free
free
[prannon@prannondesk ~]$ free -m | grep -o free | bash
total used free shared buffers cached
Mem: 10231128 10043332 187796 120588 162648 646428
-/+ buffers/cache: 9234256 996872
Swap: 5185532 18464 5167068

“Woah! What did you just do????”

So, I took free -m. The STDOUT of that command includes the word “free,” in it, which just so happens to be the same name as the command I just ran. I piped the STDOUT to the STDIN of another command that grabbed just that one word. The STDOUT of that command is “free.” I then piped that one word into the STDIN of BASH, which is the shell that I’m using. BASH sees that, thinks it’s a command, and then runs it as though I had typed it in myself.

So, here we have the idea that we can manipulate the text streams that come out of commands that we run. Now I think we can actually talk about what ‘sed’ does and what it’s useful for!

Let’s say that I want to take the STDOUT of one command, and then change it up? I can use sed to make changes to the formatting, or the spelling, or the words used in STDOUT. All I have to do is pipe the STDOUT stream into the sed command, specify my options and parameters, and then the STDOUT of sed will be whatever I wanted it to be.

Sed’s syntax works something like this…

sed ‘s/something in the original stream/whatever you want that changed to/g’

First you call to the command, and then you can have single or double quotations. Then “s,” then some dividers of some kind (commonly a forward slash or “/.” You have the option of using different dividers if it makes it easier for you to read), and then “g.” The “g” part of optional. If you don’t specify the “g,” then your change will only take place in the first instance in the stream. If you do specify the “g,” it basically makes your specified change “global.”

Neat! You can feed a stream into sed, specify what you want changed within sed, and then get what you wanted as the STDOUT of sed. Let’s see this in practice…

I’m going to use “free -m” again, but I’m going to change the first instance of the letter ‘e’ to the letter ‘a.’ I’m going to use % as my divider, since it’s easier on the eyes than /.

[prannon@prannondesk ~]$ free -m | sed ‘s%e%a%’
total usad free shared buffers cached
Mam: 9991 9830 161 143 135 689
-/+ buffars/cache: 9004 986
Swap: 5063 20 5043

Now I’m going to change EVERY instance of the letter ‘e’ to the letter ‘a.’

[prannon@prannondesk ~]$ free -m | sed ‘s%e%a%g
total usad fraa sharad buffars cachad
Mam: 9991 9834 156 143 135 689
-/+ buffars/cacha: 9009 981
Swap: 5063 20 5043

Now I’m going to be completely immature and change every instance of the word “buffers” to “poopie.”

[prannon@prannondesk ~]$ free -m | sed ‘s%buffers%poopie%g’
total used free shared poopie cached
Mem: 9991 9849 141 143 134 689
-/+ poopie/cache: 9025 966
Swap: 5063 20 5043

“Ok, that’s neat, and I guess I understand what the command is doing, but how is this even useful?”

Well… I like to think that half of Linux administration is just in modifying configuration files. There is an awful lot of configuration file modifications to be made when you’re using a Linux based OS. Sed finds its greatest utility when you’re scripting, trying to find an easy way to quickly script some changes to a configuration file, or a series of configuration files that all need the same change.

In my case, I have a program on my computer that I use to screencast my desktop. That program has a configuration file, and when I turn it on I use a script to do so. Depending on how I run the program, I may need to have some changes made to my configuration file, either to enable sound or to disable it. I don’t want to have to access my file and make changes to it every single time I run this script, so I use sed to make the changes for me.

#If the user selects to have sound, rewrite the ffserver.conf file, stop the firewall, and start ffmpeg with the appropriate options.

sed -ri ‘s/#+NoAudio/NoAudio/’ /etc/ffserver.conf
sed -ri ‘s/NoAudio/#NoAudio/’ /etc/ffserver.conf
sed -ri ‘s/#+AudioBitRate/AudioBitRate/’ /etc/ffserver.conf
sed -ri ‘s/#+AudioChannels/AudioChannels/’ /etc/ffserver.conf
sed -ri ‘s/#+AudioSampleRate/AudioSampleRate/’ /etc/ffserver.conf
echo -e “\n\nStarting Stream w/ Sound!\n\n”
systemctl stop firewalld.service
nohup ffserver &
nohup ffmpeg -video_size 1920×1080 -framerate 20 -f x11grab -i :0.0 -f alsa -ac 2 -i pulse http://localhost:9537/screencast.ffm &

In this case, I’ve added the ‘r’ and ‘i’ options to sed, which tell it to understand regular expressions (a topic for another time) and to write whatever changes it makes to the stream to the file I’ve given as its input (that file at the very end). The # you see in this example is basically a marker. It tells the program that whatever follows is to be ignored. We call this a “comment,” since most of the time the # precedes some sort of comment made on the code of the program.

A very simple edit, but it plays a very big role in my script, and it wouldn’t be easily possible without the ‘sed’ command. 🙂 Hopefully this’ll give you aspiring admins out there some ideas.