python – IOError: [Errno 32] Broken pipe when piping: `prog.py | othercmd`

python – IOError: [Errno 32] Broken pipe when piping: `prog.py | othercmd`

The problem is due to SIGPIPE handling. You can solve this problem using the following code:

from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL) 

Update: As pointed out in the comments, python docs already have a good answer.

See here for background on this solution. Better answer here.

To bring information from the many helpful answers together, with some additional information:

• Standard Unix signal SIGPIPE is sent to a process writing to a pipe when theres no process reading from the pipe (anymore).

  • This is not necessarily an error condition; some Unix utilities such as head by design stop reading prematurely from a pipe, once theyve received enough data.
  • Therefore, an easy way to provoke this error is to pipe to head^[1]; e.g.:
    • python -c for x in range(10000): print(x) | head -n 1

• By default – i.e., if the writing process does not explicitly trap SIGPIPE – the writing process is simply terminated, and its exit code is set to 141, which is calculated as 128 (to signal termination by signal in general) + 13 (SIGPIPEs specific signal number).

• However, by design Python itself traps SIGPIPE and translates it into a Python BrokenPipeError (Python 3) / IOError (Python 2) instance with errno value errno.EPIPE.

  • Note: If you use a Unix emulation environment on Windows, the error may surface differently – see this answer.

• If a Python script does not catch the exception, Python outputs error message BrokenPipeError: [Errno 32] Broken pipe (Python 3, possibly twice, with Exception ignored in: <_io.TextIOWrapper name=<stdout> mode=w encoding=utf-8> sandwiched in between) / IOError: [Errno 32] Broken pipe (Python 2) and terminates the script with exit code 1^[2] – this is the symptom Johannes (the OP) saw.

Windows considerations (SIGPIPE is a Unix-only signal)

• If your script needs to run directly on Windows too, you may have to conditionally bypass code that references SIGPIPE, as shown in this answer.

• If your script runs in a Unix subsystem on Windows, the SIGPIPE signal may surface differently than on Unix – see this answer.

---

There are two ways to solve this problem:

Generally, it is not advisable to silence this exception, as it may signal a severe error condition, depending on your scripts purpose, such as the receiving end of a network socket unexpectedly closing.

• However, if your script is a command-line utility, where quiet termination may not only be acceptable but preferred so as to play nicely with the standard head utility, for instance, you can abort quietly as follows, using signal.signal() to install the platforms default signal handler (which behaves as described above), as also shown in akhans answer (works in both Python 3 and 2):

# ONLY SUITABLE FOR COMMAND-LINE UTILITIES

# Install the default signal handler.
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE, SIG_DFL)

# Start printing many lines.
# If this gets interrupted with SIGPIPE, 
# the script aborts quietly, and the process exit code is set to
# 141 (128 + SIGPIPE)
for x in range(10000): print(x)

• Otherwise, if you want to handle the SIGPIPE-triggered exception yourself (works in both Python 3 and 2, adapted from the docs):

import sys, os, errno

try:

  # Start printing many lines.
  for x in range(10000): print(x)

  # IMPORTANT: Flush stdout here, to ensure that the 
  # SIGPIPE-triggered exception can be caught.
  sys.stdout.flush()

except IOError as e: 
  # Note: Python 3 has the more specific BrokenPipeError,
  #       but this way the code works in Python 2 too.
  if e.errno != errno.EPIPE: raise e # Unrelated error, re-throw.

  # Python flushes standard streams on exit; redirect remaining output
  # to devnull to avoid another BrokenPipeError at shutdown
  devnull = os.open(os.devnull, os.O_WRONLY)
  os.dup2(devnull, sys.stdout.fileno())

  # ... perform other handling.
  # Note: You cant write to stdout here.
  #       (print() and sys.stdout.write wont work)
  #       However, sys.stderr.write() can be used.
  sys.stderr.write(SIGPIPE received, terminating.n)

  # Finally, exit with an exit code of choice.
  sys.exit(141)

---

^{[1] Note that in bash you will by default only see heads exit code – which is 0 – reflected in $? afterwards. Use echo ${PIPESTATUS[0]} to see Pythons exit code.}

^{[2] Curiously, on macOS 10.15.7 (Catalina), with Python 3.9.2 (but not 2.x), I see exit code 120, but the docs say 1, and thats what I also see on Linux.}

python – IOError: [Errno 32] Broken pipe when piping: `prog.py | othercmd`

I havent reproduced the issue, but perhaps this method would solve it: (writing line by line to stdout rather than using print)

import sys
with open(a.txt, r) as f1:
    for line in f1:
        sys.stdout.write(line)

---

You could catch the broken pipe? This writes the file to stdout line by line until the pipe is closed.

import sys, errno
try:
    with open(a.txt, r) as f1:
        for line in f1:
            sys.stdout.write(line)
except IOError as e:
    if e.errno == errno.EPIPE:
        # Handle error

You also need to make sure that othercommand is reading from the pipe before it gets too big – https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer