@pavel phosh or rather phoc is pretty lightweight in that regard. I'd try a terminal like foot on the PP.
On the Librem5 the issue is that we don't use the advanced features of the external display controller (DCSS) like texture comression (which is a driver limitation in the kernel / mesa) so memory bandwidth becomes tight when driving both displays and hence things can become laggy.