Creating a terminal application - Backend

How Terminals work

Maybe not all of you know what goes on behind the scenes when you launch your Terminal.
The term “terminal” is from the 1970s (or even earlier) Wikipedia: Computer terminal. At this time you had a real terminal device that was built in a way so that the input/output part was separated from the computing part. There were different protocols used for those two parts to talk to each other.
A well known terminal has been the VT100 that introduces special escape codes to control e.g. the cursor or status lights.
The protocol used there is still what drives the Terminal of your operating system (at least the Unix based ones).

So given the history a terminal application today is still controlled using the same protocol by the process it talks to.
The terminal application is “only” handling user input and output. It connects to the underlying process (most of the time the default shell of your OS) via special pseudo terminal connections where it sends the user input to the shell process and receives the UI data from the shell process (using the protocol from the old days).

Hosting a shell process with DotNet

The first step in my endeavor to create a terminal application is to host the shell process. This has to be done in a special way to get a proper pseudo terminal connection.
It seems that DotNet has some problems on that end. There is something special in the dotnet process that makes the default fork mechanism that yields the pty (pseudoterminal) endpoints not working.
In a C++ program you would simply call forkpty (see man page) that does a unix style fork and returns the process id of the process launched and a file descriptor for the pty socket.
When you do that in a DotNet application it just doesn’t work.
This is why Miguel had to create a native library that executes the forkpty function to work around that problem. Source code

Of course this is not really nice as it means I have to compile platform specific code in order to make the process start work.
In one of the issues a maintainer of AvalonStudio (that also includes a Terminal component) mentioned another approach to launch the shell process that doesn’t involve native code Github issue.
This approach basically launches the running application again with a special argument that leads to an immediate (before anything else happens) fork. They have to do some special magic with the pty file descriptors so that the terminal application can communicate with the process.

So I had two options: Use Miguels approach to have classical fork but with native code or use the rather hacky approach of AvalonStudio to avoid native code.

I decided to use the AvalonStudio approach.

I implemented the process launch and setup of the pty connection and it worked. When I tried to use zsh as shell I recognized that CTRL+C didn’t stop the currently running process.
After digging around a bit I also recognized that when using bash it complains about a missing control connection.
So this means that the AvalonStudio approach did set up the basic communication channels (entering and receiving data works) but somehow screws up some bits that are needed for the shell to work properly.

So I decided to switch over to Miguels approach because the MacTerminal that he has in his repository had a working control connection.
After implementing it I tried to launch bash using my DotNet application and it crashed with a message like “Qt was not able to release locked mutex”.
I digged around and tried to find the problem but it seems to me that the way Qt is loaded into the DotNet process is not compatible to the way a Unix fork works.

So I had two not working fork solutions. What to do?

I decided to stick with Miguels approach but pull the forking in a separate executable (ConsoleHost) and communicate with the ConsoleHost process via a named pipe connection.

Some evenings later the construction was ready and Miguels approach is able to fork the shell process. Performance is also good (I will cover the real performance bottleneck in one of the next posts).
There is still one quirk: I have to launch the Console Host application via bash. Otherwise I get the same missing control connection message. Not sure why this is.

So the architecture of the terminal application looks like this:

Currently I have the backend working for Mac. Linux will probably work in a similar way and for Windows I expect WinPty to do all that hassle for me (at least is AvalonStudio using it and it seems to work there).

Next step: User interface (also focusing on Mac for now).
Supporting other platforms will be tackled when the basic terminal is working on the Mac.