Konubinix' opinionated web of thoughts

Make Org-Babel-Sh-Evaluate Handle Noop Lines

Fleeting

See how to org babel process the output in shell’s sessions for the analysis. Also I assume we have set an explicit prompt in comint-prompt-regexp, as explained here.

As a reminder, the problem is that org-babel, due to the fact it relies on comint-prompt-regexp in sh, expects all the prompts in the output to have a newline before them. In the case of noop line, only the prompt is output but no newline.

In the following example:

  • echo a
  • # noop
  • echo b

comint gives

  • a\n:theprompt:
  • :theprompt:
  • b\n:theprompt:
  • org_babel_sh_eoe\n:theprompt:

Or, once concatenated:

a
:theprompt::theprompt:b
:theprompt:org_babel_sh_eoe
:theprompt:

And, after split and trim

  • a

  • b

  • org_babel_sh_eoe

And thus in the end we should get

a

b

Let’s try it to make sure we got it right.

echo a
# noop
echo b

a

b

Ok, it looks like we understand correctly what happens.

Notice that, by setting the prompt exactly, like suggested in here, we already have sensible output. Without this fix, the output would be:

a
$b

One solution could be to find out the output associated to the noop and remove the associated output prompt. If the case of the example, we would get

  • a\n:theprompt:
  • b\n:theprompt:
  • org_babel_sh_eoe\n:theprompt:

Or, once concatenated:

a
:theprompt:b
:theprompt:org_babel_sh_eoe
:theprompt:

And finally we would get

a
b

To remove the noop commands, we could rewrite the comint-output-filter-functions part to add this behavior.

Because I did not want to mess too much with the function, I created a patch that runs a custom function instead of the inline code.

The patch is available on ipfs, at the location https://ipfs.iohttps//konubinix.eu/ipfs/bafkreidhcwrcjycffb3z6wlso2ltuwvp43vo3dsxx7jbzmwglihy5sz37e?filename=ob-comint.el

Note that because I patched a macro, you need to reload it but also recompile all the files that used it.

This patch makes org-babel-comint-with-output use the function konix/org-babel-comint-with-output/comint-output-filter-function to filter the comint output.

We could have the exact same behavior as the current one by defining the following

(defun konix/org-babel-comint-with-output/comint-output-filter-function (string-buffer text)
  (concat string-buffer text)
  )

Now, let’s try to erase the lines that match exactly the prompt. Thanks to our previous work, comint-prompt-regexp is no more a regexp and this is quite easy to write:

(defun konix/org-babel-comint-with-output/comint-output-filter-function (string-buffer text)
  (when (string-equal comint-prompt-regexp text)
    (setq text "")
    )
  (concat string-buffer text)
  )

Then we get

echo a
# noop
echo b

a
b

Now, let’s see if we to mix some commands not outputting a newline as well as noop to challenge this solution.

  • echo -n a
  • # noop
  • echo b

would result in the output

  • a:theprompt:
  • :theprompt:
  • b\n:theprompt:
  • org_babel_sh_eoe\n:theprompt:

The noop workaround would remove the second line, then

  • a:theprompt:
  • b\n:theprompt:
  • org_babel_sh_eoe\n:theprompt:

and after concatenation

a:theprompt:b
:theprompt:org_babel_sh_eoe
:theprompt:

and then, after splitting and stripping

  • a
  • b

Let’s try it for real

echo -n a
# noop
echo b
# noop
echo c
echo -n d

a
b
c
d

As you can see, the outputs are correctly shown now.

Actually, there is one subtlety that needs to be taken care of: the prompt may come some time after the last output, giving the following pattern:

  • some output
  • :theprompt:

For example, the following program don’t return anything

echo a && sleep 2

That case may be captured by simply remembering whether the last command was finished, meaning it ended by the prompt.

(defvar konix/org-babel-comint-with-output/comint-output-filter-function/not-finished-yet nil)

(defun konix/org-babel-comint-with-output/comint-output-filter-function (string-buffer text)
  ;; see https://konubinix.eu/braindump/posts/25b52cc8-71f8-420f-9161-5c60030cede9/
  ;; for an explanation
  (when (and
         (string-equal ":theprompt:" text)
         (not konix/org-babel-comint-with-output/comint-output-filter-function/not-finished-yet)
         )
    (setq text "")
    )
  (setq konix/org-babel-comint-with-output/comint-output-filter-function/not-finished-yet
        (not (string-suffix-p text ":theprompt:"))
        )
  (concat string-buffer text)
  )

Now, the delay between the output and the prompt is not a problem anymore.

echo a && sleep 2

a

Notes linking here