The Delphi Compiler and UTF-8 Encoded Source Code Files With no BOM

While the Delphi compiler can handle both ANSI and Unicode source code files, there are scenarios causing trouble, namely UTF-8 files with no BOM. The solution is simple, once you know about the issue.

Last week a (large) customer sent me an email indicating he was experiencing issues when compiling the same project on different machines. Turned out the difference was in the source code files format and the root cause was a unit saved as UTF-8 but without a BOM. The reason? One of the developers is using Visual Studio Code... and the solution is a changing that or using compiler flag. But before I get to the solution, let me show you the problem with a very simple test case.

Delphi and Source Files Encoding

To test the scenario, we (it was one of the architects who came up with the simple scenario) created a simple VCL application with code like the following:

      var
  strEuro : String = 'Euro=€';

procedure TForm16.Button1Click(Sender: TObject);
begin
  Button1.Caption := 'Hello ' + strEuro;
end;

By default, the editor in the Delphi IDE uses ANSI encoding, and everything works fine. You can use the editor context menu, pick the File Format submenu, select UTF8 (I know, missing -), and everything keeps working as expected. Notice, though the length in bytes of the string changes, as you need multiple bytes to represent the Euro symbol in UTF-8.

Enter Visual Studio Code

While many modern editors use UTF-8 as their standard file format, a nice option we are considering to default to also in RAD Studio, Visual Studio Code (or VSC) is one of the few that prefers using UTF-8 with no BOM. The BOM (Byte Order Marker) is a sequence of bytes (3 bytes for for UTF-8) that marks the file to make it simple for an editor or a tool processing the file (like a compiler) to figure out the internal format. Given the similarities in many cases, you might have to read an entire file to see if it is UTF-8 or ANSI encoded.

When you open a Delphi unit in VSC, it keeps the formatting. If it is UTF-8 with BOM, it remains as such. But for a new file the default is UTF-8 with no BOM. And any file can be saved with that format, as you can see in the status bar options:

Now if you do this, compile the application again, you'll get a nice caption for your button, rather than the Euro symbol:

Use a BOM or a Compiler Flag

Once you realize the issue, the solution is not that complex to achieve.

On one side, you can make sure your editors save the UTF-8 files with a BOM. The compiler sees it, and all works fine.

On the other side, you can tell the compiler to consider the source code file as UTF-8, regardless of the BOM. You can do this using the compiler flag:

--codepage:65001

or setting the codepage in the compiler options to that value:

Good Work Around, Future Solution

I hope this will help you avoid a similar issue. We are researching making the codepage detection automatic, so that the compiler could automatically pick up the right option. But not something we plan to addressing shortly... as the R&D team is busy with the coming release.

FreePascal has a similar command-line option (-Fc) for specifying a source file's codepage. But it also has a {$CODEPAGE} compiler directive (https://www.freepascal.org/docs-html/current/prog/progsu87.html) that lets the code itself set a codepage of subsequent string literals at compile time. Perhaps you might also consider adding a similar solution to Delphi?

Marco, I don't agree with your claim that Visual Studio Code is one of only a few editors that don't add a BOM. The fact is, you're not supposed to have a BOM with UTF-8; it's for UTF-16. From Wikipedia, referring to the Unicode Standard documentation: "The Unicode Standard neither requires nor recommends the use of the BOM for UTF-8, but warns that it may be encountered at the start of a file as a transcoding artifact " What source tells you that the purpose of a BOM is to help an editor figure out internal formats? The very name, "Byte Order Mark", is about byte ordering for UTF-16. From the specification document: "A character encoding scheme consists of a specified character encoding form plus a specification of how the code units are serialized into bytes. The Unicode Standard also specifies the use of an initial byte order mark (BOM) to explicitly differentiate big-endian or little-endian data in some of the Unicode encoding schemes.... For UTF-8, the encoding scheme consists merely of the UTF-8 code units (= bytes) in sequence. Hence, there is no issue of big- versus little-endian byte order for data represented in UTF-8." I checked Kate (Linux/Windows), Eclipse, GEdit (Linux), Sublime Text... none add a BOM by default. In fact, in all of them, setting encoding to "UTF-8" means "UTF-8 without BOM". Sticking a BOM into the file when it's not required seems to be a phenomenon of some Windows-only programs. It can cause problems, e.g.: "Programming language parsers not explicitly designed for UTF-8 can often handle UTF-8 in string constants and comments, but will choke on encountering an UTF-8 BOM at the start of the file. Programs that identify file types by leading characters may fail to identify the file if a UTF-8 BOM is present even if the user of the file could skip the BOM. An example is the Unix shebang syntax. Programs that insert information at the start of a file will break use of the BOM to identify UTF-8 (one example is offline browsers that add the originating URL to the start of the file" The problem here would be Delphi expecting a BOM, not editors that omit it for UTF-8 files.

As others pointed out, BOM is really not used in UTF-8 files, and its use discouraged. I'd suggest to use a compiler directive if an explicit encoding is used by a file, just like Python allows some editors syntax in comments to detect the encoding without parsing the whole file (and avoid ambiguities). A compiler switch is global, a directive inside the file would be local.

While almost all of my computers have Ansi code page 1252 (Latin 1), one of my machines managed to acquire Ansi code page 65001 (UTF-8) by accident. If I use the command line compiler on this machine, it uses UTF-8 for source files by default. So adding a compiler flag like "--codepage:1252" is neccessary - or you might be in for a surprise. The runtime environment uses windows API GetACP (get Ansi code page) to set global variable "DefaultSystemCodePage" (that happens in the initialization part of system.pas). DefaultSystemCodePage is used as the default code page for AnsiString, regardless of wich code page was set as a compiler flag. This means - if you use AnsiString or AnsiChar constants - that the output of your program will vary according to the current Ansi code page of your computer. You should set DefaultSystemCodePage to the desired value (e.g. 1252) in the initialization part of a unit that is used by all your other units to prevent this from happening.

Marco Tech Blog