Sed: Multiline Substitutions with Hold Space and Pattern Space

1. Why single line sed commands hit a wall

The classic call sed 's/old/new/' file.txt works strictly line by line: sed reads one line into the pattern space, applies the command, prints the result and reads the next line. For simple text replacements that is perfectly sufficient. But as soon as a replacement spans multiple lines, say an XML element that runs across three lines, or a multiline comment block in a config file, this single line approach fails completely, because sed simply no longer knows the previous line while processing the next one.

This is exactly where sed multiline substitutions come in with the so called hold space. sed internally keeps two buffers: the pattern space, which holds the line currently being processed, and the hold space, an additional storage area where lines can be parked. With targeted commands, lines can be moved between both buffers, merged and processed together. That turns sed into a full featured tool for sed multiline substitutions, without ever needing to switch to Perl or Python.

In practice this problem shows up regularly in deployment scripts that patch Nginx configurations, in cleaning up log files with multiline stack traces, or in automated rewriting of Docker Compose blocks. Once you understand sed multiline substitutions, these tasks can be solved in a single command line instead of writing an entire script in another language.

2. Understanding hold space: N, h, H, g, G, x

The six commands N, h, H, g, G and x form the foundation of every sed multiline substitutions task. N appends the next line to the current pattern space, separated by an embedded newline. h copies the pattern space into the hold space and overwrites its content, while H appends the pattern space to the existing hold space content. Conversely, g copies the hold space into the pattern space, G appends it, and x swaps both buffers completely.

A typical pattern for sed multiline substitutions: 1h copies the first line into the hold space, H appends all following lines, and at the end of the file ${g;p} retrieves the whole gathered text and prints it. This pattern is often used to load an entire file into the hold space and then work on the complete content with a single substitution command, instead of proceeding line by line.

#!/usr/bin/env bash
# sed hold space basics — copy, append, swap
set -euo pipefail

# Load the entire file into hold space, then work on it as one block
sed -n '1h; 1!H; ${g; p}' config.conf > /tmp/whole-file-as-block.txt

# Practical example: join every two lines into one (key + value pairs)
printf 'name\nAcme Corp\nport\n8080\n' | sed 'N; s/\n/=/'
# Output:
# name=Acme Corp
# port=8080

# Swap pattern and hold space to compare current line with previous one
sed -n 'x; $p; x; 1!p' access.log | tail -n +2

3. Joining lines on purpose: the N loop with :loop and b

The command N on its own only pulls a single additional line into the pattern space. To join an arbitrary number of lines, say a whole multiline comment block or a stack trace of unknown length, you need a loop. The sed multiline substitutions pattern for this defines a label with :loop, pulls in the next line with N, and jumps back with b loop as long as a condition holds, for example as long as the line does not end with a certain pattern.

Handling the last line carefully matters here: when N is called at the end of the file and no further line exists, GNU sed behaves differently from POSIX sed. GNU sed still prints the current pattern space in this case and ends the script, while strict POSIX sed aborts the script without printing anything. Anyone writing portable scripts for sed multiline substitutions should catch this explicitly with $!N, so that no N is attempted at the very last line.

#!/usr/bin/env bash
set -euo pipefail

# Join an unknown number of continuation lines ending in backslash
# into a single logical line — typical for shell or Makefile continuations
sed -e ':loop' -e '/\\$/{N; s/\\\n//; b loop}' Makefile.fragment

# Collapse multi-line log entries that start with a timestamp
# into single lines, joining continuation lines with a space
sed -E ':loop
/^[0-9]{4}-[0-9]{2}-[0-9]{2}/!{
  N
  s/\n(  )/ /
  b loop
}' app.log

4. Replacing multiline blocks: address ranges with /START/,/END/

Address ranges are the most direct tool for sed multiline substitutions when there is a clearly recognizable start and a clearly recognizable end. The syntax /START pattern/,/END pattern/ defines a range within which a command gets applied. This allows deleting, replacing or marking entire blocks of a config file without knowing the exact line numbers, which is essential for scripts meant to run on input files of varying length.

A common use case for sed multiline substitutions is replacing an entire server block in an Nginx configuration, or a section between two marker comments like # BEGIN AUTOGENERATED and # END AUTOGENERATED. The range is first deleted with d, then the new content is inserted back at the right place with a or r. This technique is more robust than a purely line number based solution, because it works no matter how the rest of the file changes around it.

#!/usr/bin/env bash
set -euo pipefail

# Replace an entire marked block between two comment markers
# with fresh content from a separate file
sed -e '/# BEGIN AUTOGENERATED/,/# END AUTOGENERATED/{
  /# BEGIN AUTOGENERATED/r generated-block.txt
  /# BEGIN AUTOGENERATED/!{/# END AUTOGENERATED/!d}
}' nginx.conf > nginx.conf.new

# Delete a whole XML element spanning multiple lines
sed '/<deprecated-feature>/,/<\/deprecated-feature>/d' features.xml

# Print only what is inside a marked range (inclusive)
sed -n '/^-- migration:2024_08/,/^-- migration:2024_09/p' schema.sql

5. Practical examples: config files, XML fragments, log blocks

In deployment scripts, sed multiline substitutions are often used to swap out entire environment variable blocks in Docker Compose files without breaking the surrounding YAML structure. A typical pattern: the whole environment: block of a specific service definition gets deleted via an address range and replaced with freshly generated lines built from an .env file. This works reliably as long as the indentation and structure of the YAML file stay consistent.

For log files containing multiline Java or PHP stack traces, the N loop is the tool of choice to merge related lines into a single record before passing them on to grep or awk. Without this intermediate step, every stack trace line would be treated as its own log entry, which skews analyses like error counts per exception type. sed multiline substitutions solve this problem by merging the whole block into one line before further processing.

A third practical example concerns XML and HTML fragments in templates where a single tag is spread across multiple lines with attributes. Instead of fragile regular expressions that only cover one line, you read in as many lines as needed with N until the closing > is found, and then apply the substitution to the complete, joined tag block. That is far more robust than trying to catch a multiline tag with a single line expression.

6. sed -z for null byte separated whole file processing

The option -z (also --null-data on GNU sed) changes the record separator from newline to the null byte. That means sed effectively treats the entire input as a single record, as long as it contains no null bytes, which drastically simplifies sed multiline substitutions that cross line boundaries, because the dot . in a regular expression then also matches newlines.

This technique is particularly useful for substitutions where a pattern is guaranteed not to start and end within a single line, for example a multiline comment block in source code, or an HTML comment spanning several lines. Without -z you would need to build an N loop for that, with -z a single s/// expression with DOTALL like behavior or an explicit newline in the search pattern is enough.

#!/usr/bin/env bash
set -euo pipefail

# Remove a multi-line HTML comment block spanning several lines
# treating the whole file as one record (null-separated)
sed -z 's/<!--.*-->//g' template.html > template.clean.html

# Replace a multi-line license header at the top of every source file
find src -name '*.php' -print0 | while IFS= read -r -d '' file; do
  sed -z -i 's/^\/\*.*\*\///' "$file"
done

# Note: -z reads the whole file into memory — avoid on very large files

7. Performance and pitfalls with large files

The big advantage of sed multiline substitutions using hold space over -z is memory usage: hold space and pattern space keep working line by line, or block by block, while -z loads the entire file into memory. For log files in the gigabyte range, -z can therefore cause noticeable slowdowns or even memory pressure, while a cleanly constructed N loop keeps working stream based and only holds as many lines in memory as the current block requires.

A second performance pitfall is an unbounded N loop without a stop condition: if the check for the end of the block is missing, sed keeps appending lines until either the pattern is found or the file ends. With malformed input data where the end marker is missing, this can lead to the entire rest of the file landing in a single, enormous pattern space. An additional safety net with a line count limit or an explicit error message for a missing end marker prevents this behavior in production scripts.

8. Debugging sed scripts: -n, the l command and dry runs

When debugging complex sed multiline substitutions, the combination of -n (suppresses automatic output) and the l command, which makes the current pattern space visible including all non printable characters and embedded newlines, is invaluable. An embedded newline shows up as \n in the text, so you immediately see whether several lines were actually merged into the pattern space or whether the N loop terminated prematurely.

A second practical trick is to first run the script with p instead of an actual substitution, to check which lines even fall into the relevant address range, before working destructively with d or s///. Anyone using sed multiline substitutions in production scripts should always test against a copy of the file first, and only change the original in place with -i after verification, ideally with -i.bak to get an automatic backup.

#!/usr/bin/env bash
set -euo pipefail

# Debug: show embedded newlines and control characters explicitly
sed -n 'N; l' two-lines.txt

# Dry run: only print the matched range, do not delete or replace yet
sed -n '/# BEGIN/,/# END/p' nginx.conf

# Safe in-place edit with automatic backup
sed -i.bak -e '/# BEGIN AUTOGENERATED/,/# END AUTOGENERATED/d' nginx.conf
diff nginx.conf.bak nginx.conf

9. sed compared: hold space vs. awk vs. perl for multiline tasks

Not every multiline text task is equally well suited for sed multiline substitutions. Beyond a certain complexity, for example with nested blocks or when counting logic is required, it pays off to look at alternatives with clearer control structures.

For clearly bounded marker blocks and simple line joins, sed multiline substitutions remain the leanest solution, because no additional interpreter is needed and the command runs in every deployment environment without further dependencies. But once counting logic, nested structures or complex conditions come into play, switching to awk or perl is the more pragmatic decision, because these cases can only be modeled in plain sed with disproportionate effort.

10. Summary

sed multiline substitutions solve a problem the standard line by line mode of sed simply cannot handle: blocks spanning multiple lines need to be looked at as a whole to be correctly replaced, deleted or marked. Hold space and pattern space provide the necessary tools for this with N, h, H, g, G and x, address ranges with /START/,/END/ make marker based blocks robust against changes elsewhere in the file.

For very large files it is worth looking at memory usage: stream based N loops are more economical than sed -z, which reads in the entire file. Anyone using sed multiline substitutions in production scripts should consistently test first with -n and the l command before working destructively on the original with -i. Beyond a certain complexity with nested structures, switching to awk or perl is the more pragmatic choice.

11. FAQ: Sed multiline substitutions