Comments (9)
FYI:
https://msdn.microsoft.com/en-us/library/yeby3zcb.aspx
FILE *fopen(
const char *filename,
const char *mode
);
FILE *_wfopen(
const wchar_t *filename,
const wchar_t *mode
);
Unicode Support
fopen supports Unicode file streams. To open a Unicode file, pass a ccs flag that specifies the desired encoding to fopen, as follows.
FILE *fp = fopen("newfile.txt", "rt+, ccs=encoding");
Allowed values of encoding are UNICODE, UTF-8, and UTF-16LE.
When a file is opened in Unicode mode, input functions translate the data that's read from the file into UTF-16 data stored as type wchar_t. Functions that write to a file opened in Unicode mode expect buffers that contain UTF-16 data stored as type wchar_t. If the file is encoded as UTF-8, then UTF-16 data is translated into UTF-8 when it is written, and the file's UTF-8-encoded content is translated into UTF-16 when it is read. An attempt to read or write an odd number of bytes in Unicode mode causes a parameter validation error. To read or write data that's stored in your program as UTF-8, use a text or binary file mode instead of a Unicode mode. You are responsible for any required encoding translation.
If the file already exists and is opened for reading or appending, the Byte Order Mark (BOM), if it present in the file, determines the encoding. The BOM encoding takes precedence over the encoding that is specified by the ccs flag. The ccs encoding is only used when no BOM is present or the file is a new file.
from pegtl.
Are you saying that fopen
on Windows only accepts ASCII, but not Unicode encoded as UTF-8? What does wfopen
get as argument, a string or a wide string? What type and encoding?
from pegtl.
fopen
takes filename in Windows (ANSI) code page, which is ASCII + 128 extra characters.
_wfopen
takes wchar_t *
which is either UCS2 or UTF-16 (I don't know).
I don't have a lot of experience with it, but I came across one example of a library handling this issue: zlib has an extra function gzopen_w
which does the same as gzopen
but takes wchar_t*
as an argument and exists only on Windows.
from pegtl.
Using _wfopen()
instead of fopen()
is easy, the problem is that we also store the filename as source
, i.e. name-of-the-source-of-the-parsed-data.
This can currently be a const char *
, or a std::string
, so that we can construct the std::string
in the position
in the parse_error
exception class.
The question is, (how) can we use _wfopen()
without supporting wchar_t
or std::wstring
as source
?
Can we convert the wide characters provided to _fopen()
to a regular string - or do we have to bite the bullet and enable wide strings in parse_error
etc...?
from pegtl.
@skyrich62 Thanks for the link; at the moment we are just looking at the encoding of the file name, not the content, but we'll keep the possibility of an on-the-fly conversion in mind.
from pegtl.
As a work-around for this: Would it be sufficient to add a ctor taking the FILE*
directly and accept an additional "normal" std::string
as the source? It makes usage slightly more verbose, but I think it fits better with the rest of the design.
@wojdyr Would that be OK with you or does it cause additional problems? Note that our class takes ownership of the FILE*
, e.g. it closes it during destruction.
from pegtl.
Yes, it would work fine for me.
from pegtl.
That makes things easier, read_input
now has a constructor that takes a FILE*
as first argument.
from pegtl.
Thanks!
from pegtl.
Related Issues (20)
- [>=3.2.1] Compile error with GCC 9.2 — error: must '#include <typeinfo>' before using 'typeid' HOT 6
- How do I capture each substring at run time HOT 3
- MSVC: error C2338: static_assert failed: 'internal::dependent_true< T > && ( begin != std::string_view::npos ) HOT 3
- Example grammar proto3 does not accept enum fields starting from zero HOT 2
- data type of input? byte? character? HOT 3
- <ciso646> is removed in C++20 and should not be included HOT 1
- Feature Request: Add defines to exclude headers to improve compile time HOT 4
- parse_tree needs to be optimized HOT 8
- Does "pegtl" support the operation of binary data serialization/deserialization? HOT 2
- Why parsing succeeds? HOT 3
- Why do I have an infinite loop? HOT 4
- parser_tree.cpp example not compiling in VS 2022, as of PEGTL 3.2.6 HOT 4
- Order independence of rules HOT 7
- list_tail<> invokes action for trailing separator twice? HOT 10
- Parsing Binary Data Encounters Left recursion Problem HOT 9
- Any consideration of ghc::filesystem HOT 8
- Can't get custom error messages to work. HOT 8
- Backreferences and grammar tracing/analyzing. HOT 5
- vs2022 编译错误 x64-windows-static\include\tao\pegtl\parse.hpp(45,38): error C2062: 意外的类型“unknown-type”
- Issues with change_action_and_state HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pegtl.