How Org Babel Processes the Output in Shell's Sessions

See
- org-babel,

For the sake of this document, let’s use $ as prompt to ease understanding what happens.

export PS1="$"

The code of org-babel-sh-evaluate looks like

(org-babel-comint-with-output
        (session org-babel-sh-eoe-output t body)
  (dolist (line (append (split-string (org-trim body) "\n")
                                        (list org-babel-sh-eoe-indicator)))
        (insert line)
        (comint-send-input nil t)
        (while (save-excursion
                         (goto-char comint-last-input-end)
                         (not (re-search-forward
                               comint-prompt-regexp nil t)))
          (accept-process-output
           (get-buffer-process (current-buffer))))))

for each input line
1. it sends the input line to comint
2. if waits for the prompt to be displayed

It calls comint-send-input for each line, nested in org-babel-comint-with-output, that overloads the comint-output-filter-functions with the following code:

(comint-output-filter-functions
 (cons (lambda (text) (setq string-buffer (concat string-buffer text)))
           comint-output-filter-functions))

The output are simply appended to string-buffer as they arrive.

In the end of the function, org-babel-comint-with-output does

(split-string string-buffer comint-prompt-regexp)

The whole output string is split using the comint-prompt-regexp, whose value is ^[^#$%>\n]*[#$%>] *. The starting ^ means that the prompt is meant to always be at the beginning of a line.

This means it make the following hypothesis:

For each input line, there is an output (possibly “”, possibly spanning several lines) and then a new line and a prompt.

Because comint sends back the output of the underlying process, the output is always followed by the prompt. So this makes total sense.

Splitting using the prompt regexp would be sensible to. For instance, with

echo a
echo b

the respective output would be

a\n$
b\n$
org_babel_sh_eoe\n$

The last one is the sentinel value sent by org-babel to find out whether the source block is finished.

Or, once concatenated:

a
$b
$org_babel_sh_eoe
$

And then, splitting using ^$ would result in

a
b
org_babel_sh_eoe

Actually, the code of org-babel-sh-evaluate removes the last two elements of the result before concatenating it, as you can see in the call to butlast with the parameter 2 below. It also concatenates the result using the \n character.

(mapconcat
 #'org-babel-sh-strip-weird-long-prompt
 (mapcar
  #'org-trim
  (butlast
   (org-babel-comint-with-output
       (session org-babel-sh-eoe-output t body)
     (dolist (line (append (split-string (org-trim body) "\n")
                           (list org-babel-sh-eoe-indicator)))
       (insert line)
       (comint-send-input nil t)
       (while (save-excursion
                (goto-char comint-last-input-end)
                (not (re-search-forward
                      comint-prompt-regexp nil t)))
         (accept-process-output
          (get-buffer-process (current-buffer))))))
   2))
 "\n")

so it eventually gets:

a
b

Which is the expected output

Let’s see it in action

echo a
echo b

a
b

So far so good.

But, when some command are noop, what happens?

Imagine the code is now

echo a
# noop
echo b

Then the output becomes

a\n$
$
b\n$
org-babel-sh-eoe-indicator\n$

Which after concatenation becomes:

a
$$b
$org_babel_sh_eoe
$

Once split, it gives

a
$b
org-babel-sh-eoe

Then, after removing the last 2 elements and concatenating with \n, we should have:

a
$b

Let’s see if this assumption is correct:

echo a
# noop
echo b

a
$b

Ok, great, we understood the code. But unfortunately this is not the expected output, we would want:

a
b

Actually, as we have seen in the hypothesis above, the code expect each line of output to be terminated with a newline.

Guess the following program:

echo -n a
echo b

The output becomes

a$
b\n$
org-babel-sh-eoe\n$

then concatenated

a$b
$org_babel_sh_eoe
$

Then the a$ part is interpreted as a prompt, then after split and trim, we get

The first empty line is due to the fact that splitting a string starting with the separator gives a first element being the empty string.

Let’s try it.

echo -n a
echo b

Indeed, the a character got lost.

Similarly, we can also make the last output be glued to the sentinel value and be missed.

echo a
echo -n b

b is not printed because the string was

a
$b$org-babel-sh-eoe
$

split into

a
b$org-babel-sh-eoe

and trimmed into

Konubinix' opinionated web of thoughts

How Org Babel Processes the Output in Shell's Sessions

Notes linking here

Permalink