Playwright/Test + Ts-Node = Wrong Stack Traces

Playwright is an awesome browser automation library. typescript is a nice programming language and using ts-node with --require ts-node/register is becoming one obvious way of running node programs. It is very practical to run a typescript program while using the power of playwright.

playwright/test provides nice helpers, like expect that can assert the page toBeVisible.

But, as soon as you import playwright/test, the stack trace becomes messy. This makes debugging very annoying.

Note that I investigated the issue only with a CommonJS project, but a quick look at the code of both ts-node and playwright suggests that the issue is the same.

Let’s see how ts-node and playwright/test transpile typescript code on the fly. Then, let’s see them work together and explain how they conflict. Finally, let’s discuss some work around this issue.

in ts-node

You enable the on-the-fly transpilation of typescript by requiring ts-node/register. Its code installs the appropriate `require.extension[“ts”]` to tell nodejs how to deal with typescript files, transpiling them on the fly and inlining the source map code in the output code.

Then, it installs the sourceMapSupport from @cspotcode to deal with the stack trace. This code is barely customized, as it finds out the source map data from the output content by itself as we would expect.

Looking at the code of @cspotcode/source-map-support, we can see that it provides a way to make it have an effect only once, even of run several times. To me, this is a good design choice, but this will impact us later in this article.

in playwright

playwright/test loads the project files in a particular way. The code run globally in each file performs the needed registering for playwright to run its tests.

To deal with typescript, it does not use require. Instead, it contains a special function called requireOrImport.

This function does the following:

1. it calls a function called installTransform that uses pirates to install babel transpilation hooks
2. it loads the file, during that time, the hooks:
   1. put the temporary source map in /tmp/playwright-transform-cache-<PID>/file.js.map
   2. put the temporary compiled content in /tmp/playwright-transform-cache-<PID>/file.js
   3. remembers internally (in compilationCache.js:sourceMaps) the association file->map
3. the previous step is recursively run for each file loaded, hence the whole tree of required files is transpiled.
4. revert the transpilation hooks put at step 2.

At the step 2. of the above flow, the module compilationCache.js is loaded. This module has the side effect of installing a source map support compatible with how the files where transpiled. This code looks for the sourceMaps variables. If there is no match, it simply returns, inferring there is no source map.

We can see that the lifecycle of the transpilation hooks and the one of the source map support is not the same at all. We can have the source map support left setup while the hooks are reverted.

It would be nice that playwright would install and uninstall the source map support when the transpilation is installed and uninstalled, because it makes no sense to have one and not the other.

put together

Let’s get our hands dirty.

Let’s start by installing ts-node and playwright.

npm i ts-node @playwright/test

added 21 packages in 3s

Imagine you run the following typescript code.

import { expect } from "@playwright/test"
let b = expect // to avoid tree shaking
class A{} // to increase the likelyhood that the js file is longer than the typescript file
throw new Error()

This code will throw an error at line 4.

Let’s try it, using ts-node to load the typescript content.

node --require ts-node/register code.ts

Error:
    at Object.<anonymous> (/home/sam/tmp/tmp.gaWFjR7cm1/code.ts:7:7)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module.m._compile (/home/sam/tmp/tmp.gaWFjR7cm1/node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Object.require.extensions.<computed> [as .ts] (/home/sam/tmp/tmp.gaWFjR7cm1/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Function.Module._load (node:internal/modules/cjs/loader:958:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47

It shows an error at line 7 instead at line 4.

Based on the previous description, we can explain what happens:

1. ts-node/register sets up the transpilation code as well as the associated source map support
2. our code is loaded, triggering the loading of @playwright/test that loads modules that load module… that load ./compilationCache.js. This installs its own source map support, overriding the one from ts-node
3. note that nothing called requireOrImport. Therefore compilationCache.js:sourceMap is left empty.
4. the error is thrown
5. the source map support of playwright is triggered and does nothing because compilationCache.js:sourceMap is empty.
6. this leads to this stack trace, with the lines of the generated javascript content rather than the typescript content.

workarounds

We want to have the transpilation hooks and the source maps support that comes with it. It could be either the one of playwright, or the one of ts-node.

trying to use the code of ts-node

As we could see previously, it is the transpilation hooks of ts-node that are used when we simply run node. Therefore, we only need to focus of the sourcemap support.

It is to be noted that the source map support used in ts-node is meant to be called only once. Therefore, when we initially install it via --require ts-node/register, it prevents us from being able to reinstall it later.

Moreover, when using playwright/test, the source map support of playwright will overwrite the one of ts-node. Also, because it is done in a global instruction when compilationCache is loaded, this happens only once in the lifetime of the program.

What we need is a way to first let playwright install its sourceMap, and then overwrite it.

This is actually quite simple using a commonjs shim scripts that first loads playwright and then loads ts-node/register. Fortunately, the compilationCache module that installs the source map support is exported, so the code of the shim then is simply:

require("@playwright/test/lib/transform/compilationCache");
require("ts-node").register();

This code can be required the same way we required ts-node/register.

node --require ./shim-tsnode code.ts

/home/sam/tmp/tmp.gaWFjR7cm1/code.ts:4
throw new Error()
      ^
Error
    at Object.<anonymous> (/home/sam/tmp/tmp.gaWFjR7cm1/code.ts:4:7)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module.m._compile (/home/sam/tmp/tmp.gaWFjR7cm1/node_modules/ts-node/src/index.ts:1618:23)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Object.require.extensions.<computed> [as .ts] (/home/sam/tmp/tmp.gaWFjR7cm1/node_modules/ts-node/src/index.ts:1621:12)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Function.Module._load (node:internal/modules/cjs/loader:958:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47

Now the lines are correctly shown!

using the code of playwright

In that case, there is nothing to be done about the source map support, but we need to setup its transpilation hooks.

We know that this is done using installTransform. It would be nice to simply load transform and load this code. Like this.

require("@playwright/test/lib/transform/transform").installTransform();

Unfortunately, this code is not exported, so we cannot actually use it.

Let’s try it anyway for the sake of the analysis, by patching the installed playwright/test to export installTransform.

pushd node_modules/@playwright/test
cat<<EOF | patch
--- ./node_modules/@playwright/test/package.json
+++ ./node_modules/@playwright/test/package.json
@@ -20,6 +20,7 @@
     "./lib/cli": "./lib/cli.js",
     "./lib/transform/babelBundle": "./lib/transform/babelBundle.js",
     "./lib/transform/compilationCache": "./lib/transform/compilationCache.js",
+    "./lib/transform/transform": "./lib/transform/transform.js",
     "./lib/transform/esmLoader": "./lib/transform/esmLoader.js",
     "./lib/internalsForTest": "./lib/internalsForTest.js",
     "./lib/plugins": "./lib/plugins/index.js",
EOF
popd

pushd node_modules/@playwright/test/lib/transform
cat<<EOF | patch
--- transform.js
+++ transform.js
@@ -8,6 +8,7 @@
 exports.setTransformConfig = setTransformConfig;
 exports.shouldTransform = shouldTransform;
 exports.transformConfig = transformConfig;
+exports.installTransform = installTransform
 exports.transformHook = transformHook;
 exports.wrapFunctionWithLocation = wrapFunctionWithLocation;
 var _crypto = _interopRequireDefault(require("crypto"));
EOF
popd

~/tmp/tmp.1c1NtlwuW4/node_modules/@playwright/test ~/tmp/tmp.1c1NtlwuW4
patching file package.json
~/tmp/tmp.1c1NtlwuW4
org_babel_sh_prompt> ~/tmp/tmp.1c1NtlwuW4/node_modules/@playwright/test/lib/transform ~/tmp/tmp.1c1NtlwuW4
patching file transform.js
~/tmp/tmp.1c1NtlwuW4

And see what happens:

node --require ./shim-playwright.js code.ts

/home/sam/tmp/tmp.1c1NtlwuW4/code.ts:6
throw new Error();
^

Error:
    at Object.<anonymous> (/home/sam/tmp/tmp.1c1NtlwuW4/code.ts:4:7)
    at Module._compile (node:internal/modules/cjs/loader:1254:14)
    at Module.f._compile (/home/sam/tmp/tmp.1c1NtlwuW4/node_modules/@playwright/test/lib/utilsBundleImpl.js:16:994)
    at Module._extensions..js (node:internal/modules/cjs/loader:1308:10)
    at Object.i.<computed>.ut._extensions.<computed> [as .ts] (/home/sam/tmp/tmp.1c1NtlwuW4/node_modules/@playwright/test/lib/utilsBundleImpl.js:16:1010)
    at Module.load (node:internal/modules/cjs/loader:1117:32)
    at Function.Module._load (node:internal/modules/cjs/loader:958:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:23:47

Node.js v18.15.0

Here it is! A stack trace generated using the code of playwright.

which should I prefer?

Using ts-node, we have the following pros

1. it is meant to be used that way (SRP),
2. we may eventually remove the shim if in the future playwright deals with this issue more nicely with us, we won’t have anything else to do,

The cons are:

1. it is not meant to be overwritten and then rerun. We may encounter issues if it does not override correctly what has been done by playwright.

Using playwright, on the other hand, has this pro: we don’t use ts-node at all, so we don’t deal with the possible issues of them overlapping. But it also has the downside that we have to patch it. Also we are linked to playwright and the day we change our browser automation tool, we have to think of putting back ts-node.

Actually, if we don’t mind patching playwright, we could remove the code that deals with source map support. With a diff like this:

--- compilationCache.js
+++ compilationCache.js
@@ -50,6 +50,20 @@
const fileDependencies = new Map();
// Dependencies resolved by the external bundler.
const externalDependencies = new Map();
-Error.stackTraceLimit = 200;
-_utilsBundle.sourceMapSupport.install({
-  environment: 'node',
-  handleUncaughtExceptions: false,
-  retrieveSourceMap(source) {
-    if (!sourceMaps.has(source)) return null;
-    const sourceMapPath = sourceMaps.get(source);
-    if (!_fs.default.existsSync(sourceMapPath)) return null;
-    return {
-      map: JSON.parse(_fs.default.readFileSync(sourceMapPath, 'utf-8')),
-      url: source
-    };
-  }
-});
function _innerAddToCompilationCache(filename, options) {
sourceMaps.set(options.moduleUrl || filename, options.sourceMapPath);
memoryCache.set(filename, options);

That way, only ts-node would deal with transpiling and there would be no more overlapping.

conclusion

In the end, there are three dimensions to consider:

1. whether we don’t mind patching something,
2. the likelyhood of two systems interfering with one another,
3. the single-responsibility principle,

There seem to be no clear winner, those seem like pareto optima. Depending of how those criteria are important to you, you might prefer one or another.

In my opinion, eventually, the best scenario would be that playwright fixes that situation of leaving the sourceMap support when the transpilation hooks are uninstalled, but that does not seem like an easy task.

Notes linking here

• to transpile data in nodejs (braindump)

Permalink