[BRLTTY] Multi-column characters on Linux

[Aura Kelloniemi]

Hi,

On 2021-10-01 at 17:03 -0400, Dave Mielke <Dave at mielke.cc> wrote:
 > On the screen, Linux pads a wide character with a space to the right. This maintains vertical alignment, which is what they care about.

I reasearched this, and Linux does not do anything special with two-column
characters, not even padding them with spaces.

The behaviour you describe is done by editors and some other programs and it
usually happens just, because the editor repositions the cursor after the user
types in characters. So Linux displays the character as one-column wide, but
the editor moves the cursor by two columns, because it thinks that the
character is wide, and therefore it looks like as if there was an intentional
padding space inserted.

You can often demonstrate this by typing several wide characters in an editor
(and see them be interspersed with spaces). Then, if you redraw the screen
(often with CTRL+L), you can see that all the characters you typed are packed
to the left side of the screen and all the padding space is put after the
text.

All the editors that I have tested (emacs, nano, vim) have serious issues with
Linux not behaving as a proper Unicode-aware terminal. All of these editors
have serious issues with cursor positioning and text on the screen becoming
garbled when workign with wide characters.

I actually would like to here, which editor people are using to write and read
languages that utilize two-column characters.

I'm afraid that people who use languages with combining characters really are
in trouble, if they try to use the Linux console. These issues probably cannot
even be worked around withouth switching to anoter terminal emulator. (This is
because Linux prints the combining characters as a diamond, not understanding
how these codepoints should be handled.)

I have tried to work with people from emacs to enable a hack for linux that
would prevent the screen from getting garbled. They are, however, very
reluctant, and they want the Linux folks to fix this. But as you know, Linux
console is sort of deprecated, and these problems will not be fixed.

Here I provide a proof that Linux really does not understand two-column
characters.

This is a Bash session in a bare Linu console:

$ echo $'ab\U0001F64Fxy\rabc'
abcxy

This prints letters a and b followed by a wide emoji, followed by letters x
and y. Then it moves the cursor back to the beginning of line with \r and
writes letters a b and c. These should override the first two letters and the
first half of the emoji. This leaves the letters x and y in tact.

But as you see, the c letter here overrides the whole emoji. If the emoji
really was wide, then the output would be

$ echo $'ab\U0001F64Fxy\rabc'
abc xy

Here the space represents the right half of the broken emoji. This later
example is run in a VTE-based terminal that supports Unicode properly.

 > Brltty's Linux screen driver removes that space because retaining it makes
 > it very hard for those whose languages use wide characters, e.g. Chinese,
 > to read. Imagine how reading your language would be much more difficult if
 > there were a space after every single letter.

Yes, that would be unacceptable.

But my problem remains. I really care about the alignment, and my language
does not use wide characters. Therefore I wish that this feature of removing a
space is made a run-time configurable option. (It may happen, that a reader of
Chinise would like to read a table, and they need to temporarily check which
characters align vertically). Also, I'm afraid that this option can result in
the the screen cursor and the cursor shown by BRLTTY to be out of sync.

I would also like to get rid of this space removal right away, because I'm
debugging alignment issues, and I might provide misinformation to developers
because of this. Could you please tell me the location of the code
implementing this, so I can comment it out even before this is made
configurable. (If it is made configurable at all.)

Thanks in advance!

-- 
Aura