\nAs is customary, the first day of CLR Week is a warm-up.\nActually, today’s question is a BCL question, not a CLR question,\nbut only the nitpickers will bother to notice.\n<\/P>\n
\n\nCan somebody explain why StreamReader can’t read apostrophes?\nI have a text file, and I read from it the way you would expect:\n<\/P>\n
\nStreamReader sr = new StreamReader(“myfile.txt”);\nConsole.WriteLine(sr.ReadToEnd());\nsr.Close();\n<\/PRE>\n\nI expect this to print the contents of the file\nto the console, and it does—almost.\nEverything looks great except that all the apostrophes are gone!\n<\/P>\n<\/BLOCKQUOTE>\n
\nYou don’t have to have very strong psychic powers to figure this one out.\n<\/P>\n
\nHere’s a hint: In some versions of this question,\nthe problem is with accented letters.\n<\/P>\n
\nYour first psychic conclusion is that the text file is probably an ANSI\ntext file.\nBut\n\nStreamReader defaults to UTF-8<\/A>, not ANSI.\nOne version of this question actually came right out and asked,\n“Why can’t StreamReader read apostrophes from my ANSI text file?”\nThe alternate version of the\nquestion already contains a false hidden assumption:\nStreamReader can’t read apostrophes from an ANSI text file\nbecause StreamReader (by default) doesn’t read ANSI text files at all!\n<\/P>\n
\n
\nBut that shouldn’t be a factor, since the apostrophe is encoded\nthe same in ANSI and UTF-8, right?\n<\/P>\n
\nThat’s your second clue.\nOnly the apostrophe is affected.\nWhat’s so special about the apostrophe?\n(The bonus hint should tip you off:\nWhat’s so special about accented letters?\nWhat property do they share with the apostrophe?)\n<\/P>\n
\nThere are apostrophes and there are apostrophes,\nand it’s those “weird” apostrophes that are the issue here.\nCode points\nU+2018 (‘) and\nU+2019 (’)\noccupy positions 0x91 and 0x92, respectively,\nin code page 1252,\nand these “weird” apostrophes are all illegal lead bytes in\nUTF-8 encoding.\nAnd the default behavior for the\n\nEncoding.UTF8Encoding<\/A>\nencoding is to ignore invalid byte sequences.\nNote that StreamReader does not raise an exception when incorrectly-encoded\ntext is encountered.\nIt just ignores the bad byte and continues as best it can,\nfollowing\n\nBurak’s advice<\/A>.\n<\/P>\n
\nResult:\nStreamReader appears to ignore apostrophes and accented letters.\n<\/P>\n
\nThere are therefore multiple issues here.\nFirst,\nyou may want to look at why\nyour ANSI text file is using those weird apostrophes.\nMaybe it’s intentional, but I suspect it isn’t.\nSecond, if you’re going to be reading ANSI text,\nyou can’t use a default StreamReader, since a default StreamReader\ndoesn’t read ANSI text.\nYou need to set the encoding to
System.Text.Encoding.Default<\/CODE>\nif you want to read ANSI text.\nAnd third, why are you using ANSI text in the first place?\nANSI text files are not universally transportable, since the ANSI code\npage changes from system to system.\nShouldn’t you be using UTF-8 text files in the first place?\n<\/P>\n\nAt any rate, the solution is to decide on an encoding and to specify\nthat encoding when creating the StreamReader.\n<\/P>\n