Unicode support

[tamaroning]

Tokenizer

• add pass to check input files are valid as utf-8
  • Refactoring lexer to treat all characters as UTF-8 #2309
  • Add utf-8 validation for input source #2374
• add util function to check XID_Start/XID_Continue to libcpp
  • XID_Start, XID_Continue
    • https://www.unicode.org/Public/14.0.0/ucd/DerivedCoreProperties.txt
  • Tokenize Unicode identifiers #2284
• identifiers
  • Tokenize Unicode identifiers #2284
  • Missing tests for utf-8 identifiers #2338
• lifetime label
  • Tokenize Unicode identifiers #2284
  • Missing tests for utf-8 identifiers #2338
• raw identifiers
  • Refactoring lexer to treat all characters as UTF-8 #2309
  • Missing tests for utf-8 identifiers #2338
• string literal
• char literal
• Unicode esccape \uxxxx
• loop label
  • not implemented yet
• parse whitespaces
  • https://doc.rust-lang.org/reference/whitespace.html?highlight=whitespaces#whitespace
  • Refactor lexer to treat all input characters as UTF-8 #2307
  • Fix lexer to skip utf-8 whitespaces #2339
• normalize identifiers (including labels) to NFC
  • normalize function: Unicode Normalization of Identifiers #2379
  • use it during tokenization: Normalize all identifiers to NFC #2489

Parser

• check for crate_name attributes
  • https://doc.rust-lang.org/reference/crates-and-source-files.html#the-crate_name-attribute
  • Unicode Alphabetic
    • https://www.unicode.org/reports/tr44/#Alphabetic
    • https://www.unicode.org/Public/14.0.0/ucd/DerivedCoreProperties.txt
  • Unicode Numeric (search with Nl; (decimal digits), Nd; (letter-like numeric), or No; (other numeric) )
    • https://doc.rust-lang.org/std/primitive.char.html#method.is_numeric
  • is_numeric and is_alphabetic functions is added via
    • Add header file generator for Unicode normalization and alphanumeric check #2425
  • Modify checker
    • Unicode check for crate_name attribute #2463

Backend

• punycode (v0 mangling)
  • Implement punycode encoding #2533
  • Related to Symbol Mangling #305

Others

• linter
  • https://rust-lang.github.io/rfcs/2457-non-ascii-idents.html
• Fix #[no_mangle]
  • #[no_mangle] should not be applied to non-ASCII items #2548
• Fix legacy name manglling
  • Legacy name mangling does not encode unicode characters #2545

TODOs

need more tests!

[CohenArthur]

Unicode identifiers were stabilized in rust-lang/rust#83799. There is a PR (rust-lang/reference#1022) on the reference to clarify the rules regarding unicode identifiers. There are also some lints to detect confusing cases like mixing different scripts.

originally posted by @bjorn3 in #418