Hyperlink Support in GNU Poke
[02-02-2020]

by Darshit Shah

FOSDEM 2020 is over, and hyperlink support has just landed for GNU Poke!

Wait, Hyperlinks!?

What do hyperlinks, a web concept, mean for GNU Poke, a terminal application?

For many years now, terminal emulators have been detecting http:// URLs in the output of any program and giving the user a chance to click on them and immediately navigate to the corresponding web page. In 2017, Egmont Kob made a proposal for supporting general hyperlinks in terminal emulators. Gnome Terminal, iTerm and a few other terminal emulators have already implemented this proposal in their latest releases. With Egmont's proposal, an application can emit any valid URI and have the terminal emulator take the user to that resource.

A few applications, such as systemd, GNU Coreutls and GNU Wget2 have already incorporated hyperlink support. These applications use hyperlink support for one of two things:

  1. To print out pretty URLs that take you to a specific web page
  2. To use file:// URIs to allow the user to click on a filename and open it using an application of their choice.

While these are nice ways to use the new hyperlink support in terminal emulators, there's so much more that can be done. GNU Poke intends to use hyperlinks to really change how we interact with command line applications. Take a look at this demo video of how such an interaction may look like:

GNU Poke Hyperlinks Demo

As you can see, one can now simply click on commands printed on the screen to execute them. Similarly, you can change the currently mapped file by simply clicking on the filename in the terminal. In the future, a lot more hyperlinks are planned, making navigation within Poke incredibly easy, and all this without using ncurses.

The path to Hyperlinks

When Bruno Haible first introduced to us the concept of hyperlinks in terminal emulators, we were intrigued. This has the potential to be a paradigm shifting change in how we interact with modern terminal emulators. Clicking on the output of ls and playing the video in mpv, or viewing the image in feh, etc. is quite cool. This is what the future looks like!

And yet we had to ask, "What next!?". Opening files and applications is all nice and dandy, but we wanted more. We wanted to drastically change how users can interact with certain applications. To us, these hyperlinks were most promising in the context of non TUI interactive programs.

Yes, most terminal users love their keyboards, but in today's world we almost always have a mouse at hand. And sometimes, clicking is indeed faster than typing.

So, during one of the Rabbit-Herd Hacking Weekends (RHHW), we sat and designed a protocol for interactive applications to use. This is a very simple protocol based on a new URI handler, app:// or appsocket://. I will give a brief description of the protocol here; the full text of the proposal is available in Bruno's email to freedesktop.org.

The app:// protocol]

Bruno's email introduces the appsocket:// protocol. During the first iteration of this protocol, we called it simply app://. Thus, GNU Poke implements the app:// protocol. However, it is identical to the protocol suggested by Bruno. Once standardised, Poke will switch to using the correct version of the name.

Following from RFC3986, the URI syntax for our proposal would be: 

URI              = scheme ":" hier-part

scheme           = "app"
hier-part        = "//" authority payload-abempty
authority        = host ":" port
payload-abempty  = *( "/" segment )

query and fragment are currently not supported in app:// URIs. However, support for those can be added if the need arises.
The authority element removes the userinfo part since the original proposal for hyperlinks did not use it either.

In easier terms, this translates to: 

app://hostname:port/payload

That's it! The protocol is that simple.
The payload is completely application specific. We decided to leave things like security, multiple-clicks, etc. to the application developer since these vary wildly in usage. While, in some application a link should be valid only once, for another it could be reused even across different invocations of the application.

This is what a real hyperlink generated by Poke looks like: 

 app://ankh-morpok:46603/1161/e/.help
 \___/ \_________/ \___/ \__________/
   |        |        |        |
Protocol Hostname   Port   Payload

The application then listens on the given port for incoming connections carrying a payload. It is the terminal emulator's responsibility to open a connection to the port and handover the payload to the application.

Since, no terminal emulators (except emacs, jemarch hacked that during the same weekend), support this protocol yet, we designed a small external utility called app-client to perform the task. XDG compliant terminal emulators can be configured to start app-client for app:// URIs. Steps for configuring the clients are shown later.

As mentioned above, the payload is left to the discretion of the application program. For GNU Poke, the payload is described by the following grammar: 

payload-abempty  = "/" nonce "/" command "/" parameters

nonce            = string
command          = "i"  ; insert into readline prompt
                 / "e"  ; execute a command
parameters       = *(pchar)

That is, the payload is made up of three different components separated by the path separator, "/":

Together, a real payload looks like this: 

  Command
     |
1161/e/.help
\__/   \___/
 |       |
Nonce  Parameters

Nonce

The nonce is a unique string generated for each hyperlink. GNU Poke stores each generated nonce in memory and verifies the nonce for each payload it receives on the open port. This safeguards GNU Poke against:

  1. Commands from a previous invocation of Poke cannot be used in the current execution (in the unlikely event that Poke binds to and listens on the same port again).
  2. Accidental messages (that look like a valid command) sent to the open port by an unrelated program don't trigger Poke into executing arbitrary commands
  3. Malicious applications cannot coax Poke into executing their commands on a mapped file

Ideally, the nonce is a randomly generated string of fixed or varying length. However, the current implementation of hyperlinks support in Poke is rudimentary and only generates a 4-digit random number as the nonce.
Currently, Poke also does not verify if the nonce it receives matches the command it was created for. Hence, it is susceptible to a reuse attack, where a known good nonce can be used to execute an arbitrary number of commands. This is a known issue and we will get to fixing it as soon as time permits.

Command

The command identifies the type of hyperlink it is. Currently, it can be one of two types, insert or execute.

Parameters

parameters are the additional information required by the command.

In case of insert commands, this is the exact text that will be put into the readline prompt.
In case of execute commands, this is the Poke command along with any data that will be executed.

Since the parameters may contain reserved characters, it is preferable to percent encode the parameters before emitting the hyperlink. GNU Poke though, currently takes a shortcut and does not percent encode its hyperlinks. This too will be fixed shortly.

Testing Hyperlink support in GNU poke

Step 1. Compile a new version of libtextstyle

libtextstyle is a library for managing the output of structured and formatted data to the terminal. GNU Poke uses libtextstyle for handling coloured outputs and also for printing hyperlinks.
Since hyperlink support was added fairly recently to libtextstyle, there exists no release with support for printing hyperlinks. Hence to have Poke emit hyperlinks, it is required to build a version of libtextstyle from git. Since libtextstyle is currently distributed as a part of GNU Gettext, one needs to compile the latest git version of that. 

$ git clone https://git.savannah.gnu.org/git/gettext.git
$ cd gettext
$ ./gitsub.sh pull
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install

Or, if you use Arch Linux, then simply install gettext-git using your favourite AUR manager.

Step 2. Get a terminal emulator that supports hyperlinks

Gnome Terminal has support for displaying hyperlinks as do many other emulators that rely on VTE. Check the list here for a mostly up-to-date, non-exhaustive list of emulators that support printing hyperlinks. In the example above, I used Gnome Terminal.

Step 3. Make your terminal emulator understand app:// URIs

Since app:// is a new URI protocol that we designed, common terminal emulators don't know what to do when they encounter such a URI. To work around this problem we use the XDG Desktop Specification and app-client. By setting app-client as the default handler for app:// URIs, the terminal emulator does not need to understand the syntax or semantics of the app:// protocol. It offloads the handling of the URI entirely to app-client. In order to use this, first download and install app-client: 

$ git clone https://gitlab.com/darnir/hyperlink-app-client
$ cd hyperlink-app-client
$ make
$ cp app-client /location/in/$PATH/variable

Next, we need to register the app-client as the handler for the app:// URIs. This is a two-step process:

  1. Copy the app-client.desktop file in the git repository to $HOME/.local/share/applications. This is a XDG Desktop Entry for the app-client. And lets most applications on your system know that this should be used to handle app:// URIs. (See the MimeType field)

  2. However, Gnome Terminal doesn't use xdg-open to start the applications. Instead, it parses the mimeapps.list file manually to find the right application. Edit your mimeapps.list, it is usually located at $HOME/.local/share/applications/mimeapps.list, but it might also be at $XDG_CONFIG_DIR/mimeapps.list, and add the following line to it:
    x-scheme-handler/app=app-client.desktop
    This lets Gnome Terminal know how to open app:// links.

Step 4. Compile Poke!

This is the easy part! Just clone GNU Poke and build it. Instructions and other build requirements are mentioned in the HACKING file.
Remember to pass --enable-hserver during ./configure to enable the hyperlink support.

Back to Applied Pokology Follow up in the mailing list...