libzim is not able to deal correctly with unicode path on Windows about libzim HOT 3 CLOSED

Comment by Brion:

Can this be done with the standard POSIX file APIs? Or do we need to use Win32 APIs for file access? :P

(Use of legacy 8-bit encodings in POSIX APIs is a problem for PHP, and thus MediaWiki, when running on Windows servers as well.)

Sounds like you gotta use the Win32 wide-char APIs to get Unicode filename support... at least that's what glib does on Windows for this reason -- https://developer.gnome.org/glib/2.26/glib-File-Utilities.html#glib-File-Utilities.description

from libzim.

lvandeve wrote:

Hello,

I have done a little bit of research on this (I must admit it was on a Linux computer however).

I hope this may help at least with fixing this issue.

Most File, FileImpl, zim::ifstream and zim::streambuf works with string. There is one intermediate string -> const char* -> string conversion in FileImpl, which the attached patch removes (to use string for everything). This to hopefully support the NUL-characters that may appear in the octets of UTF-16 strings put in a 8-bit char string.

In addition, the patch has a comment at the locations in fstream.cpp where probably different Windows API needs to be called.

I think that this way, file.h does not need to be changed to have wchar_t or wstring constructors.

What follows below the line is the patch:

diff --git a/zimlib/include/zim/file.h b/zimlib/include/zim/file.h
index a6ac75b..8aa5f0d 100644
--- a/zimlib/include/zim/file.h
+++ b/zimlib/include/zim/file.h
@@ -39,7 +39,7 @@ namespace zim
File()
{ }
explicit File(const std::string& fname)

•    : impl(new FileImpl(fname.c_str()))
  

•    : impl(new FileImpl(fname))
     { }
  
   const std::string& getFilename() const   { return impl->getFilename(); }
  

diff --git a/zimlib/include/zim/fileimpl.h b/zimlib/include/zim/fileimpl.h
index 1cf584d..ada65ee 100644
--- a/zimlib/include/zim/fileimpl.h
+++ b/zimlib/include/zim/fileimpl.h
@@ -53,7 +53,7 @@ namespace zim
offset_type getOffset(offset_type ptrOffset, size_type idx);

 public:

•  explicit FileImpl(const char* fname);
  

•  explicit FileImpl(const std::string& fname);
  
   time_t getMTime() const   { return zimFile.getMTime(); }
  

diff --git a/zimlib/src/fileimpl.cpp b/zimlib/src/fileimpl.cpp
index 8c072eb..d7a7f4a 100644
--- a/zimlib/src/fileimpl.cpp
+++ b/zimlib/src/fileimpl.cpp
@@ -38,7 +38,7 @@ namespace zim
//////////////////////////////////////////////////////////////////////
// FileImpl
//

• FileImpl::FileImpl(const char* fname)

• FileImpl::FileImpl(const std::string& fname)
  : zimFile(fname),
  direntCache(envValue("ZIM_DIRENTCACHE", DIRENT_CACHE_SIZE)),
  clusterCache(envValue("ZIM_CLUSTERCACHE", CLUSTER_CACHE_SIZE))
  diff --git a/zimlib/src/fstream.cpp b/zimlib/src/fstream.cpp
  index 5ce72f5..e4accf8 100644
  --- a/zimlib/src/fstream.cpp
  +++ b/zimlib/src/fstream.cpp
  @@ -59,6 +59,24 @@ namespace zim
  //
  streambuf::OpenfileInfo::OpenfileInfo(const std::string& fname_)
  : fname(fname_),
  +// I think regular std::string with 8-bit characters
  +// is OK up to this point, that is, in the ctor and fields of File,
  +// FileImpl, ifstream, streambuf etc... (that is, no wchar_t or wstring
  +// necessary there and no API change needed), the string just may have
  +// NULL-characters if bytes of UTF-16 are present, but std::string
  +// supports that (const char* does not).
  +// So c_str() or any C/C++ API that uses C strings should not be used at
  +// this point or anywhere before.
  +// So here a Windows-specific API that supports UTF-16 is needed for
  +// WIN32, and the fname string needs to converted here to what it
  +// needs if necessary. That is still TODO, hence the comment here.
  +// Apparently the fname received here on Windows is UTF-16 encoded with
  +// two chars per wchar_t, if used by Kiwix (the other possibility would
  +// have been that it was UTF-8 encoded and needed to be converted to
  +// UTF-16). Maybe a comment in the File constructor could also be nice
  +// to ensure all users will be consistent with this?
  +// This concludes my investigation, I hope it can help fixing the
  +// issue :)
  #ifdef HAVE_OPEN64
  fd(::open64(fname.c_str(), O_RDONLY | O_LARGEFILE | O_BINARY))
  #else
  @@ -293,6 +311,8 @@ time_t streambuf::getMTime() const
  if (mtime || files.empty())
  return mtime;

+// See comment under streambuf::OpenfileInfo::OpenfileInfo about UTF-16:
+// the same applies here as well to the c_str() call and stat.
const char* fname = files.front()->fname.c_str();

#ifdef HAVE_STAT64

from libzim.

@mgautierfr You will probably need to have a look for kiwix-desktop 2.0

from libzim.