CY-X FORUM

After being reminded today how much I hate Unicode, I wrote down some ideas I had for a new text encoding that I hope one day we can get with a new OS, enjoy:

Scripts are separated into different groups that are either 1 or 2 bytes and can have their own semantics (but
multi-point characters should be avoided wherever possible). This allows for 1 byte = 1 character (or 2 bytes = 1
character for certain groups) like Latin-1 and UCS-2 without the UTF-8 hack on top of the abomination that is Unicode.
Given that they are in separate groups, retarded scribble scripts like Arabic can be quarantined with their composed
characters separated from everyone else. Each group can have its own defined semantics where necessary and not have
everything end up being a lookup into a colossal Unicode table.

A modal processor is used to manage multiple groups with the special value of 0xFF to indicated a 4-byte group header:
[0xFF][1-byte group][2-byte size]
This allows for up to 255 different groups which is more than enough and a size value that (1) allows you to skip
through chunks (avoiding the UTF-8 issues of absolute linear scanning) and use direct indexing (unless the group
prohibits it) and (2) partially maintain UTF-8's context-independent values. Memory corruption is localized to the
immediate bytes for UTF-8, whereas a pure modal method could have the entire following stream be corrupted. The size
allows for faster whole-document traversal and localizes any corruption to the immediate group.

Let's call the basic encoding Latin-A which will be 1 byte and backwards-compatible with ASCII (sans useless garbage
like the vertical tab and bell values which hopefully could be replaced with something useful). By default, the encoding
assumed should be Latin-A so that any ASCII config files or other text just work (and then it is also not necessary to
initially change groups for pure ASCII text). Additional groups will be required for some languages like Vietnamese that love
to scribble all over their letters, but they should all be ASCII-compatible if possible (it would probably be a composed
group given the number of possibilities, but the issue is then localized to one Latin group and no one else has to care).
Scripts like Cyrillic and Greek should also be able to fit in to 1-byte groups, whereas scripts like Hanzi and Hangeul
get 2-byte encodings. There can be a maths group that contains all the common letters and symbols used in order to avoid
constant group-switching.

Additionally, redundancy can be added where it is beneficial like in, say, Japanese there can be matching ASCII
characters in numerically-matching positions (ie, 'A' is 65 in both groups, but stored in 1 byte for Latin-A vs 2 bytes
in the Japanese encoding) which allows the group-switch overhead to be bypassed. This does, however, incur a small (but
trivial -> just remove the zero bytes) normalization step for passwords, databases, etc. Certain groups like the maths
one should simply be forbidden in contexts requiring normalization because it would complicate having to normalize
Latin, Greek, etc values to their primary groups.

Benefits vs Unicode:

- Smaller text than UTF-8 (equal for ASCII, close to 33%/50% shorter for almost everything else)
- No retarded having-to-check-every-character-for-its-UTF-8-header (which wastes 30%+ of the bits BTW)
- Quarantines things which require extra processing into distinct groups
- Character-wise processing is much more straightforward (equivalent to ASCII processing for most groups)
- You don't have to look up a massive table just to figure out if something is a space character
- How many bytes is 'á'? :^)
- Character queries and processing are much easier to optimize

Some potential issues:

- Normalization is needed to get to ASCII-subset optimization
- Concatenation requires a (simple) backwards-scan to update the header
- Less corruption resistant
- A 1-byte group like Cyrillic that can't have ASCII as a subset has an expensive group switch for a few Latin letters ("Съешь ещё" = 9B/13B (17B UTF-8), but "СъешьA ещё" = 18B/22B (18B UTF-8) because it requires two group switches). There are potential optimizations for this.