This file documents version 4.1.5 of GNU `sed', a stream editor.

   Copyright (C) 1998, 1999, 2001, 2002, 2003, 2004 Free Software
Foundation, Inc.

   This document is released under the terms of the GNU Free
Documentation License as published by the Free Software Foundation;
either version 1.1, or (at your option) any later version.

   You should have received a copy of the GNU Free Documentation
License along with GNU `sed'; see the file `COPYING.DOC'.  If not,
write to the Free Software Foundation, 59 Temple Place - Suite 330,
Boston, MA 02110-1301, USA.

   There are no Cover Texts and no Invariant Sections; this text, along
with its equivalent in the printed manual, constitutes the Title Page.

Print `bash' Environment
========================

   This script strips the definition of the shell functions from the
output of the `set' Bourne-shell command.

     #!/bin/sh

     set | sed -n '
     :x

     # if no occurrence of "=()" print and load next line
     /=()/! { p; b; }
     / () $/! { p; b; }

     # possible start of functions section
     # save the line in case this is a var like FOO="() "
     h

     # if the next line has a brace, we quit because
     # nothing comes after functions
     n
     /^{/ q

     # print the old line
     x; p

     # work on the new line now
     x; bx
     '

Reverse Characters of Lines
===========================

   This script can be used to reverse the position of characters in
lines.  The technique moves two characters at a time, hence it is
faster than more intuitive implementations.

   Note the `tx' command before the definition of the label.  This is
often needed to reset the flag that is tested by the `t' command.

   Imaginative readers will find uses for this script.  An example is
reversing the output of `banner'.(1)

     #!/usr/bin/sed -f

     /../! b

     # Reverse a line.  Begin embedding the line between two newlines
     s/^.*$/\
     &\
     /

     # Move first character at the end.  The regexp matches until
     # there are zero or one characters between the markers
     tx
     :x
     s/\(\n.\)\(.*\)\(.\n\)/\3\2\1/
     tx

     # Remove the newline markers
     s/\n//g

   ---------- Footnotes ----------

   (1) This requires another script to pad the output of banner; for
example

     #! /bin/sh

     banner -w $1 $2 $3 $4 |
       sed -e :a -e '/^.\{0,'$1'\}$/ { s/$/ /; ba; }' |
       ~/sedscripts/reverseline.sed

Reverse Lines of Files
======================

   This one begins a series of totally useless (yet interesting)
scripts emulating various Unix commands.  This, in particular, is a
`tac' workalike.

   Note that on implementations other than GNU `sed' this script might
easily overflow internal buffers.

     #!/usr/bin/sed -nf

     # reverse all lines of input, i.e. first line became last, ...

     # from the second line, the buffer (which contains all previous lines)
     # is *appended* to current line, so, the order will be reversed
     1! G

     # on the last line we're done -- print everything
     $ p

     # store everything on the buffer again
     h

Numbering Lines
===============

   This script replaces `cat -n'; in fact it formats its output exactly
like GNU `cat' does.

   Of course this is completely useless and for two reasons:  first,
because somebody else did it in C, second, because the following
Bourne-shell script could be used for the same purpose and would be
much faster:

     #! /bin/sh
     sed -e "=" $@ | sed -e '
       s/^/      /
       N
       s/^ *\(......\)\n/\1  /
     '

   It uses `sed' to print the line number, then groups lines two by two
using `N'.  Of course, this script does not teach as much as the one
presented below.

   The algorithm used for incrementing uses both buffers, so the line
is printed as soon as possible and then discarded.  The number is split
so that changing digits go in a buffer and unchanged ones go in the
other; the changed digits are modified in a single step (using a `y'
command).  The line number for the next line is then composed and
stored in the hold space, to be used in the next iteration.

     #!/usr/bin/sed -nf

     # Prime the pump on the first line
     x
     /^$/ s/^.*$/1/

     # Add the correct line number before the pattern
     G
     h

     # Format it and print it
     s/^/      /
     s/^ *\(......\)\n/\1  /p

     # Get the line number from hold space; add a zero
     # if we're going to add a digit on the next line
     g
     s/\n.*$//
     /^9*$/ s/^/0/

     # separate changing/unchanged digits with an x
     s/.9*$/x&/

     # keep changing digits in hold space
     h
     s/^.*x//
     y/0123456789/1234567890/
     x

     # keep unchanged digits in pattern space
     s/x.*$//

     # compose the new number, remove the newline implicitly added by G
     G
     s/\n//
     h

Numbering Non-blank Lines
=========================

   Emulating `cat -b' is almost the same as `cat -n'--we only have to
select which lines are to be numbered and which are not.

   The part that is common to this script and the previous one is not
commented to show how important it is to comment `sed' scripts
properly...

     #!/usr/bin/sed -nf

     /^$/ {
       p
       b
     }

     # Same as cat -n from now
     x
     /^$/ s/^.*$/1/
     G
     h
     s/^/      /
     s/^ *\(......\)\n/\1  /p
     x
     s/\n.*$//
     /^9*$/ s/^/0/
     s/.9*$/x&/
     h
     s/^.*x//
     y/0123456789/1234567890/
     x
     s/x.*$//
     G
     s/\n//
     h

Counting Characters
===================

   This script shows another way to do arithmetic with `sed'.  In this
case we have to add possibly large numbers, so implementing this by
successive increments would not be feasible (and possibly even more
complicated to contrive than this script).

   The approach is to map numbers to letters, kind of an abacus
implemented with `sed'.  `a's are units, `b's are tens and so on: we
simply add the number of characters on the current line as units, and
then propagate the carry to tens, hundreds, and so on.

   As usual, running totals are kept in hold space.

   On the last line, we convert the abacus form back to decimal.  For
the sake of variety, this is done with a loop rather than with some 80
`s' commands(1): first we convert units, removing `a's from the number;
then we rotate letters so that tens become `a's, and so on until no
more letters remain.

     #!/usr/bin/sed -nf

     # Add n+1 a's to hold space (+1 is for the newline)
     s/./a/g
     H
     x
     s/\n/a/

     # Do the carry.  The t's and b's are not necessary,
     # but they do speed up the thing
     t a
     : a;  s/aaaaaaaaaa/b/g; t b; b done
     : b;  s/bbbbbbbbbb/c/g; t c; b done
     : c;  s/cccccccccc/d/g; t d; b done
     : d;  s/dddddddddd/e/g; t e; b done
     : e;  s/eeeeeeeeee/f/g; t f; b done
     : f;  s/ffffffffff/g/g; t g; b done
     : g;  s/gggggggggg/h/g; t h; b done
     : h;  s/hhhhhhhhhh//g

     : done
     $! {
       h
       b
     }

     # On the last line, convert back to decimal

     : loop
     /a/! s/[b-h]*/&0/
     s/aaaaaaaaa/9/
     s/aaaaaaaa/8/
     s/aaaaaaa/7/
     s/aaaaaa/6/
     s/aaaaa/5/
     s/aaaa/4/
     s/aaa/3/
     s/aa/2/
     s/a/1/

     : next
     y/bcdefgh/abcdefg/
     /[a-h]/ b loop
     p

   ---------- Footnotes ----------

   (1) Some implementations have a limit of 199 commands per script

Counting Words
==============

   This script is almost the same as the previous one, once each of the
words on the line is converted to a single `a' (in the previous script
each letter was changed to an `a').

   It is interesting that real `wc' programs have optimized loops for
`wc -c', so they are much slower at counting words rather than
characters.  This script's bottleneck, instead, is arithmetic, and
hence the word-counting one is faster (it has to manage smaller
numbers).

   Again, the common parts are not commented to show the importance of
commenting `sed' scripts.

     #!/usr/bin/sed -nf

     # Convert words to a's
     s/[ tab][ tab]*/ /g
     s/^/ /
     s/ [^ ][^ ]*/a /g
     s/ //g

     # Append them to hold space
     H
     x
     s/\n//

     # From here on it is the same as in wc -c.
     /aaaaaaaaaa/! bx;   s/aaaaaaaaaa/b/g
     /bbbbbbbbbb/! bx;   s/bbbbbbbbbb/c/g
     /cccccccccc/! bx;   s/cccccccccc/d/g
     /dddddddddd/! bx;   s/dddddddddd/e/g
     /eeeeeeeeee/! bx;   s/eeeeeeeeee/f/g
     /ffffffffff/! bx;   s/ffffffffff/g/g
     /gggggggggg/! bx;   s/gggggggggg/h/g
     s/hhhhhhhhhh//g
     :x
     $! { h; b; }
     :y
     /a/! s/[b-h]*/&0/
     s/aaaaaaaaa/9/
     s/aaaaaaaa/8/
     s/aaaaaaa/7/
     s/aaaaaa/6/
     s/aaaaa/5/
     s/aaaa/4/
     s/aaa/3/
     s/aa/2/
     s/a/1/
     y/bcdefgh/abcdefg/
     /[a-h]/ by
     p

Counting Lines
==============

   No strange things are done now, because `sed' gives us `wc -l'
functionality for free!!! Look:

     #!/usr/bin/sed -nf
     $=

Printing the First Lines
========================

   This script is probably the simplest useful `sed' script.  It
displays the first 10 lines of input; the number of displayed lines is
right before the `q' command.

     #!/usr/bin/sed -f
     10q

Printing the Last Lines
=======================

   Printing the last N lines rather than the first is more complex but
indeed possible.  N is encoded in the second line, before the bang
character.

   This script is similar to the `tac' script in that it keeps the
final output in the hold space and prints it at the end:

     #!/usr/bin/sed -nf

     1! {; H; g; }
     1,10 !s/[^\n]*\n//
     $p
     h

   Mainly, the scripts keeps a window of 10 lines and slides it by
adding a line and deleting the oldest (the substitution command on the
second line works like a `D' command but does not restart the loop).

   The "sliding window" technique is a very powerful way to write
efficient and complex `sed' scripts, because commands like `P' would
require a lot of work if implemented manually.

   To introduce the technique, which is fully demonstrated in the rest
of this chapter and is based on the `N', `P' and `D' commands, here is
an implementation of `tail' using a simple "sliding window."

   This looks complicated but in fact the working is the same as the
last script: after we have kicked in the appropriate number of lines,
however, we stop using the hold space to keep inter-line state, and
instead use `N' and `D' to slide pattern space by one line:

     #!/usr/bin/sed -f

     1h
     2,10 {; H; g; }
     $q
     1,9d
     N
     D

   Note how the first, second and fourth line are inactive after the
first ten lines of input.  After that, all the script does is: exiting
on the last line of input, appending the next input line to pattern
space, and removing the first line.

Make Duplicate Lines Unique
===========================

   This is an example of the art of using the `N', `P' and `D'
commands, probably the most difficult to master.

     #!/usr/bin/sed -f
     h

     :b
     # On the last line, print and exit
     $b
     N
     /^\(.*\)\n\1$/ {
         # The two lines are identical.  Undo the effect of
         # the n command.
         g
         bb
     }

     # If the `N' command had added the last line, print and exit
     $b

     # The lines are different; print the first and go
     # back working on the second.
     P
     D

   As you can see, we mantain a 2-line window using `P' and `D'.  This
technique is often used in advanced `sed' scripts.

Print Duplicated Lines of Input
===============================

   This script prints only duplicated lines, like `uniq -d'.

     #!/usr/bin/sed -nf

     $b
     N
     /^\(.*\)\n\1$/ {
         # Print the first of the duplicated lines
         s/.*\n//
         p

         # Loop until we get a different line
         :b
         $b
         N
         /^\(.*\)\n\1$/ {
             s/.*\n//
             bb
         }
     }

     # The last line cannot be followed by duplicates
     $b

     # Found a different one.  Leave it alone in the pattern space
     # and go back to the top, hunting its duplicates
     D

Remove All Duplicated Lines
===========================

   This script prints only unique lines, like `uniq -u'.

     #!/usr/bin/sed -f

     # Search for a duplicate line --- until that, print what you find.
     $b
     N
     /^\(.*\)\n\1$/ ! {
         P
         D
     }

     :c
     # Got two equal lines in pattern space.  At the
     # end of the file we simply exit
     $d

     # Else, we keep reading lines with `N' until we
     # find a different one
     s/.*\n//
     N
     /^\(.*\)\n\1$/ {
         bc
     }

     # Remove the last instance of the duplicate line
     # and go back to the top
     D

Squeezing Blank Lines
=====================

   As a final example, here are three scripts, of increasing complexity
and speed, that implement the same function as `cat -s', that is
squeezing blank lines.

   The first leaves a blank line at the beginning and end if there are
some already.

     #!/usr/bin/sed -f

     # on empty lines, join with next
     # Note there is a star in the regexp
     :x
     /^\n*$/ {
     N
     bx
     }

     # now, squeeze all '\n', this can be also done by:
     # s/^\(\n\)*/\1/
     s/\n*/\
     /

   This one is a bit more complex and removes all empty lines at the
beginning.  It does leave a single blank line at end if one was there.

     #!/usr/bin/sed -f

     # delete all leading empty lines
     1,/^./{
     /./!d
     }

     # on an empty line we remove it and all the following
     # empty lines, but one
     :x
     /./!{
     N
     s/^\n$//
     tx
     }

   This removes leading and trailing blank lines.  It is also the
fastest.  Note that loops are completely done with `n' and `b', without
relying on `sed' to restart the the script automatically at the end of
a line.

     #!/usr/bin/sed -nf

     # delete all (leading) blanks
     /./!d

     # get here: so there is a non empty
     :x
     # print it
     p
     # get next
     n
     # got chars? print it again, etc...
     /./bx

     # no, don't have chars: got an empty line
     :z
     # get next, if last line we finish here so no trailing
     # empty lines are written
     n
     # also empty? then ignore it, and get next... this will
     # remove ALL empty lines
     /./!bz

     # all empty lines were deleted/ignored, but we have a non empty.  As
     # what we want to do is to squeeze, insert a blank line artificially
     i\

     bx

GNU `sed''s Limitations and Non-limitations
*******************************************

   For those who want to write portable `sed' scripts, be aware that
some implementations have been known to limit line lengths (for the
pattern and hold spaces) to be no more than 4000 bytes.  The POSIX
standard specifies that conforming `sed' implementations shall support
at least 8192 byte line lengths.  GNU `sed' has no built-in limit on
line length; as long as it can `malloc()' more (virtual) memory, you
can feed or construct lines as long as you like.

   However, recursion is used to handle subpatterns and indefinite
repetition.  This means that the available stack space may limit the
size of the buffer that can be processed by certain patterns.

Other Resources for Learning About `sed'
****************************************

   In addition to several books that have been written about `sed'
(either specifically or as chapters in books which discuss shell
programming), one can find out more about `sed' (including suggestions
of a few books) from the FAQ for the `sed-users' mailing list,
available from any of:
      `http://www.student.northpark.edu/pemente/sed/sedfaq.html'
      `http://sed.sf.net/grabbag/tutorials/sedfaq.html'

   Also of interest are
`http://www.student.northpark.edu/pemente/sed/index.htm' and
`http://sed.sf.net/grabbag', which include `sed' tutorials and other
`sed'-related goodies.

   The `sed-users' mailing list itself maintained by Sven Guckes.  To
subscribe, visit `http://groups.yahoo.com' and search for the
`sed-users' mailing list.

Reporting Bugs
**************

   Email bug reports to <bonzini@gnu.org>.  Be sure to include the word
"sed" somewhere in the `Subject:' field.  Also, please include the
output of `sed --version' in the body of your report if at all possible.

   Please do not send a bug report like this:

     while building frobme-1.3.4
     $ configure
     error--> sed: file sedscr line 1: Unknown option to 's'

   If GNU `sed' doesn't configure your favorite package, take a few
extra minutes to identify the specific problem and make a stand-alone
test case.  Unlike other programs such as C compilers, making such test
cases for `sed' is quite simple.

   A stand-alone test case includes all the data necessary to perform
the test, and the specific invocation of `sed' that causes the problem.
The smaller a stand-alone test case is, the better.  A test case should
not involve something as far removed from `sed' as "try to configure
frobme-1.3.4".  Yes, that is in principle enough information to look
for the bug, but that is not a very practical prospect.

   Here are a few commonly reported bugs that are not bugs.

`N' command on the last line
     Most versions of `sed' exit without printing anything when the `N'
     command is issued on the last line of a file.  GNU `sed' prints
     pattern space before exiting unless of course the `-n' command
     switch has been specified.  This choice is by design.

     For example, the behavior of
          sed N foo bar

     would depend on whether foo has an even or an odd number of
     lines(1).  Or, when writing a script to read the next few lines
     following a pattern match, traditional implementations of `sed'
     would force you to write something like
          /foo/{ $!N; $!N; $!N; $!N; $!N; $!N; $!N; $!N; $!N }

     instead of just
          /foo/{ N;N;N;N;N;N;N;N;N; }

     In any case, the simplest workaround is to use `$d;N' in scripts
     that rely on the traditional behavior, or to set the
     `POSIXLY_CORRECT' variable to a non-empty value.

Regex syntax clashes (problems with backslashes)
     `sed' uses the POSIX basic regular expression syntax.  According to
     the standard, the meaning of some escape sequences is undefined in
     this syntax;  notable in the case of `sed' are `\|', `\+', `\?',
     `\`', `\'', `\<', `\>', `\b', `\B', `\w', and `\W'.

     As in all GNU programs that use POSIX basic regular expressions,
     `sed' interprets these escape sequences as special characters.
     So, `x\+' matches one or more occurrences of `x'.  `abc\|def'
     matches either `abc' or `def'.

     This syntax may cause problems when running scripts written for
     other `sed's.  Some `sed' programs have been written with the
     assumption that `\|' and `\+' match the literal characters `|' and
     `+'.  Such scripts must be modified by removing the spurious
     backslashes if they are to be used with modern implementations of
     `sed', like GNU `sed'.

     On the other hand, some scripts use s|abc\|def||g to remove
     occurrences of _either_ `abc' or `def'.  While this worked until
     `sed' 4.0.x, newer versions interpret this as removing the string
     `abc|def'.  This is again undefined behavior according to POSIX,
     and this interpretation is arguably more robust: older `sed's, for
     example, required that the regex matcher parsed `\/' as `/' in the
     common case of escaping a slash, which is again undefined
     behavior; the new behavior avoids this, and this is good because
     the regex matcher is only partially under our control.

     In addition, this version of `sed' supports several escape
     characters (some of which are multi-character) to insert
     non-printable characters in scripts (`\a', `\c', `\d', `\o', `\r',
     `\t', `\v', `\x').  These can cause similar problems with scripts
     written for other `sed's.

`-i' clobbers read-only files
     In short, `sed -i' will let you delete the contents of a read-only
     file, and in general the `-i' option (see Invocation)
     lets you clobber protected files.  This is not a bug, but
     rather a consequence of how the Unix filesystem works.

     The permissions on a file say what can happen to the data in that
     file, while the permissions on a directory say what can happen to
     the list of files in that directory.  `sed -i' will not ever open
     for writing  a file that is already on disk.  Rather, it will work
     on a temporary file that is finally renamed to the original name:
     if you rename or delete files, you're actually modifying the
     contents of the directory, so the operation depends on the
     permissions of the directory, not of the file.  For this same
     reason, `sed' does not let you use `-i' on a writeable file in a
     read-only directory (but unbelievably nobody reports that as a
     bug...).

`0a' does not work (gives an error)
     There is no line 0.  0 is a special address that is only used to
     treat addresses like `0,/RE/' as active when the script starts: if
     you write `1,/abc/d' and the first line includes the word `abc',
     then that match would be ignored because address ranges must span
     at least two lines (barring the end of the file); but what you
     probably wanted is to delete every line up to the first one
     including `abc', and this is obtained with `0,/abc/d'.

`[a-z]' is case insensitive
     You are encountering problems with locales.  POSIX mandates that
     `[a-z]' uses the current locale's collation order - in C parlance,
     that means using `strcoll(3)' instead of `strcmp(3)'.  Some
     locales have a case-insensitive collation order, others don't: one
     of those that have problems is Estonian.

     Another problem is that `[a-z]' tries to use collation symbols.
     This only happens if you are on the GNU system, using GNU libc's
     regular expression matcher instead of compiling the one supplied
     with GNU sed.  In a Danish locale, for example, the regular
     expression `^[a-z]$' matches the string `aa', because this is a
     single collating symbol that comes after `a' and before `b'; `ll'
     behaves similarly in Spanish locales, or `ij' in Dutch locales.

     To work around these problems, which may cause bugs in shell
     scripts, set the `LC_COLLATE' and `LC_CTYPE' environment variables
     to `C'.

   ---------- Footnotes ----------

   (1) which is the actual "bug" that prompted the change in behavior

Extended regular expressions
****************************

   The only difference between basic and extended regular expressions
is in the behavior of a few characters: `?', `+', parentheses, and
braces (`{}').  While basic regular expressions require these to be
escaped if you want them to behave as special characters, when using
extended regular expressions you must escape them if you want them _to
match a literal character_.

Examples:
`abc?'
     becomes `abc\?' when using extended regular expressions.  It
     matches the literal string `abc?'.

`c\+'
     becomes `c+' when using extended regular expressions.  It matches
     one or more `c's.

`a\{3,\}'
     becomes `a{3,}' when using extended regular expressions.  It
     matches three or more `a's.

`\(abc\)\{2,3\}'
     becomes `(abc){2,3}' when using extended regular expressions.  It
     matches either `abcabc' or `abcabcabc'.

`\(abc*\)\1'
     becomes `(abc*)\1' when using extended regular expressions.
     Backreferences must still be escaped when using extended regular
     expressions.

Concept Index
*************

   This is a general index of all issues discussed in this manual, with
the exception of the `sed' commands and command-line options.

Command and Option Index
************************

   This is an alphabetical list of all `sed' commands and command-line
options.