[BRLTTY] Possible Tactile Graphics Support in the BRLAPI Protocol

[Elijah Massey]

Hello,
I have started writing an implementation of the brlapi protocol in pure Rust, that directly parses and writes the packets instead of using libbrlapi. One of the things I want to use it for is writing a brlapi server for the Monarch, since I figured out how to display any dot matrix from within an Android app, and capture all key events from the Braille keyboard and panning, arrow, and zoom keys, and how to capture the locations of double taps (when you hold your finger somewhere on the display and press the circle button twice). So after my protocol library is done, I can write a server that manages connections from clients, keeps track of the state of the display, converts key presses to brltty commands, etc, and hook that up to the display and keys of the Monarch. Then BRLTTY could connect to it over WiFi with the ba driver and display multi-line Braille, or Orca or NVDA could connect to it directly and display Braille.

However, this wouldn't expose all the capabilities of the Monarch, because the Monarch can display more than Braille dots. To display multi-line Braille, my server would calculate the amount of spacing needed around the Braille, and leave those dots unused, like the driver for the Dot Pad does. But when displaying tactile graphics, those unused dots are very useful, and there doesn't seem to be a way to expose that over brlapi. I could use raw mode with a custom packet format, but then only one application could use the display at a time, because raw mode gives exclusive control to one client ignoring the priority system and the TTY paths. This solution would also mean that displaying tactile graphics on the Dot Pad would be done differently than displaying them on the Monarch, unless I just implement the Dot Pad protocol for my raw mode. But then if the Cadence or something else has tactile graphics support, then there would be the same problem.

Do you think it would be a good idea to add packets to the brlapi protocol for treating supported displays as arrays of equidistant pins? Maybe a second getDisplaySize packet that returns its width and height in pins, and a writeDots packet that takes a starting x value, a starting y value, a width, a height, and an array of 0s or 1s, which sets a section of the display pins. It might also be good to be able to get the parameters of the Braille spacing somehow, so an application could display both text and graphics letting brltty or my brlapi server handle Braille spacing, and still know enough so they don't overlap with each other. This would allow people to write graphics viewers as brlapi clients that work with all current and future tactile graphics displays, or add image display support to screen readers like Orca or NVDA, or add image support to brltty on Android.

Sent from my iPad