Dart Isolates and huge amounts of data

Why?

I’m currently experimenting with Dart and Flutter to take another shot at the terminal emulation application. This time not using C# but something completely new.
For this I decided to take a look at Flutter and Dart and try to implement the terminal application using that technology.
Gladly there is already a person in the internet that is working on implementing the xterm protocol for Dart and also has a working Flutter UI that renders the terminal.
https://github.com/TerminalStudio/xterm.dart
This implementation is not yet finished and has a couple of problems but is a really good starting point to dig into a Dart and Flutter based development of a terminal application.
One “problem” of the solution is that the whole terminal logic happens in the UI thread. So I wanted to move that logic into a separate thread and only copy the UI state to the UI thread for the UI to show it.
I did this for the C# based approaches and it speeded up the performance (especially reflow) a lot.

Dart and threads

Dart has a very special way of handling threads. The designers have chosen to use an approach that rules out any race condition by design.

For this they decided that in Dart threads don’t have shared memory and they have to define so called ports for explicit data exchange (the data gets copied).
This is called an Isolate in Dart. So you can create and spawn a new Isolate, specify its “main” method and define the port to communicate with it.

Dart makes sure that the data that is sent to a port is serialized and deserialized on the other end. This way two Isolates never work on the same place in memory. By design.

Then comes the terminal

I tried this with the terminal logic and it worked but the performance gain from moving the xterm logic to a separate thread was completely eaten up by the overhead of copying the whole UI state for each change in the terminal state.

So the amount of data exchanged here was too much for it to be copied each time.

I searched around to find a way of doing “classical” threading in Dart as I was ready to do some locking in favor of getting rid of the memory copying overhead.

In one of the StackOverflow posts (can’t remember which one) I found a comment throwing in the idea of getting raw heap memory, fill that with the data and use the Isolate port only to share the pointer to that raw memory.

Of course this puts you in control and also the responsibility to handle that piece of memory yourself (making sure that it gets freed, no access happens after free, …) but I wanted to give it a shot.

Going raw

Dart allows direct heap interaction with the ffi package. It provides all the APIs needed to get, access and free memory on the heap.

// get heap
final ptr = calloc<Int64>(amount);

// manipulate it
ptr.elementAt(index).value = newValue;

// use it
final condition = ptr.elementAt(index).value == comarisonValue;

// free it
calloc.free(ptr);

I developed an intermediate layer that transforms the data relevant for the UI to a data type that then gets copied over to the UI thread. This datatype internally puts the terminal content (and only the content) into raw heap memory and only holds the pointer(s) to the heap.

Then I tried to run it and directly got an exception: Dart prohibits sending of dart:ffi datatypes between Isolates (for a good reason I suppose).

I already knew that I was violating some unwritten Dart rules so I didn’t feel too bad by using a dirty workaround to bypass that check by getting the raw pointer value (int value), store that and when accessing the heap again re-transform this into a dart:ffi Pointer.

// convert dart:ffi Pointer to a raw address
final rawAddress = ptr.address;

// get a dart:ffi Pointer from a raw address
final ptr = Pointer<Int64>.fromAddress(rawAddress);

Note! Time has moved on and in the meantime Dart has a feature (Finalizers) exposed to the user that allows implementing a destructor-like experience. You can read a bit more about that here

This intermediate layer has to make sure that the memory is freed when it no longer is in use. In other programming languages you would put that logic into a destructor.

Sadly Dart doesn’t know the concept of a destructor so I had to do some basic usage tracking manually. This makes it even more dangerous to use and therefore should only be used when absolutely necessary.

class IntermediateLayer {
  
  final List<int> _rawDataPtr;
  int _usages = 0;
  bool _freed = false;

  // ... a whole bunch of other things

  void addUsage() {
    _usages++;
  }

  void removeUsage() {
    _usages--;
    if (_usages <= 0) {
      free();
    }
  }

  void free() {
    _checkFreed();
    _rawDataPtr.forEach((element) {
      calloc.free(Pointer<Int64>.fromAddress(element));
    });
    _freed = true;
  }

  void _checkFreed() {
    if (_freed) {
      throw Exception("Raw data is already freed but still gets used!");
    }
  }
}

Conclusion

In the end it worked and the performance gain was very huge.

It clearly violates all Dart rules regarding Isolates but I think in that special case (huge amounts of data that have to be passed) there is no other option.

I think those are the places where you can see that Dart is not yet fully meant to be used as a cross platform programming language for mobile and desktop (and the web). They are clearly heading into that direction (with huge steps) but aren’t quite there yet.

I also think that this terminal problem is a very special one. I don’t think that there are many cases where you have to exchange that amount of data for each UI state change like in the terminal scenario.
So 99% of the apps are quite fine with the way Isolates work and profit from the race condition freeness it guarantees.

Maybe this post helps someone that also has to pass big amounts of data between two Isolates in Dart.