Unicode in titles/usernames

[jdpage]

Right now, page titles are required to match /^[a-z][a-z0-9]*(?:\/[a-z][a-z0-9]*)*$/i, i.e. they must be one or more parts separated by slashes, where each part consists of an English letter followed by zero or more English letters or Arabic numbers. Usernames are further restricted and must match /^[a-z][a-z0-9]*$/i equivalent to one "part" above. (See also issue #33, which will introduce spaces in titles and forever ban underscores.)

This policy is obviously horrifically Anglocentric. I went ahead and implemented it because I wasn't up on Perl 5 Unicode support, and it's easier to expand the character-set allowed for titles than it is to contract it. But the question remains: what should the title format be?

• Research Perl 5 Unicode support (initial reading suggests 5.18 supports it well, throughout)
• Research Sqlite3 Unicode support (seems to be passthrough)
• Choose a title format
• Implementation

References:

• perluniintro
• perlunicode

[jdpage]

• Wikipedia disallows # < > [ ] | { }, disrecommends . /, and bans sequences of three or more consecutive ~ characters.
• Unless we change the link syntax, the : | [ ] characters are likely to cause problems for us.

[jdpage]

My current thinking is that page titles follow this format:

<page-title> ::= <title-part> { "/" <page-title> }
<title-part> ::= <letter-character> { <title-character> }
<title-character> ::= <letter-character>
                    | <decimal-digit-character>
                    | <combining-character>
                    | <formatting-character>
<letter-character> ::= /\pL/  ; uppercase, lowercase, titlecase, modifier, other
<decimal-digit-character> ::= /\p{Nd}/
<combining-character> ::= /[\p{Mn}\p{Mc}]/  ; non-spacing, spacing-combining
<formatting-character> ::= /\p{Cf}/

i.e. they match the regular expression:
/\pL[\pL\p{Nd}\p{Mn}\p{Mc}\p{Cf}]*(?:\/\pL[\pL\p{Nd}\p{Mn}\p{Mc}\p{Cf}]*)*/

Furthermore, all page titles, slugs, what-have-you should be put into NFKC (per http://unicode.org/faq/normalization.html#2) before any processing occurs. This means that homograph attacks will not cause problems. It also means that some titles which (as input by the user) would not be accepted by the above pattern will be transformed into titles that are (e.g. titles containing Roman numerals, which in NFKC are decomposed into ASCII characters, or more practically titles copy-pasted from sources where the actual ffi ligature character was used).

[jdpage]

Note that the above pattern still allows for a #33-style implementation of spaces.