Seeking for ways to regenerate `src/generated/*` files about qevercloud HOT 9 CLOSED

QEverCloud sources are generated from Evernote's Thrift IDL files using the custom code generating parser. The generation happens during the build of that generator, i.e. just do cmake amd make and you'll get the generated headers/sources in your build directory's "generated" folder. Then you just copy them manually to QEverCloud's source tree.

I can foresee that Debian developers would like the generated files be generated during the build of QEverCloud. However, I am actually against it because it would sufficiently complicate the building of QEverCloud (more build-dependencies + more build steps + complication of existing build steps) + Thrift IDL files would typically only change along with Evernote API itself and when that changes, the generated files would most likely have API/ABI breaks compared to previous version making it necessary to do some manual refinement of non-generated QEverCloud sources or at the very least bump the version of the library. Therefore, I really don't think the QEverCloud sources should be treated in the same way as shortened JS files - it's a very different kind of source code generation.

from qevercloud.

Thanks, I will take a look.

P. S. This is Debian; even though things may become a little bit
complicated, those extra work still needs to be done.

在 2016年9月4日週日 17:15，Dmitry Ivanov [email protected] 寫道：

QEverCloud sources are generated from Evernote's Thrift IDL files
https://github.com/evernote/evernote-thrift using the custom code
generating parser https://github.com/d1vanov/QEverCloudGenerator. The
generation happens during the build of that generator, i.e. just do cmake
amd make and you'll get the generated headers/sources in your build
directory's "generated" folder. Then you just copy them manually to
QEverCloud's source tree.

I can foresee that Debian developers would like the generated files be
generated during the build of QEverCloud. However, I am actually against it
because it would sufficiently complicate the building ofbQEverCloud (more
build-dependencies + more build steps + complication of existing build
steps) + Thrift IDL files would typically only change along with Evernote
API itself and when that changes, the generated files would most likely
have API/ABI breaks compared to previous version making it necessary to do
some manual refinement of non-generated QEverCloud sources or at the very
least bump the version of the library. Therefore, I really don't think the
QEverCloud sources should be treated in the same way as shortened JS files

• it's a very different kind of source code generation.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#5 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AIDYFkr06yEwQCoFtfBI4MrcHflm_tRdks5qmowRgaJpZM4J0eod
.

from qevercloud.

P. S. This is Debian; even though things may become a little bit
complicated, those extra work still needs to be done.

So there's no chance at all to avoid the generation step during the build of QEverCloud for Debian? Even if my reasoning above is explained to the Debian developer mentoring this packaging project?

If so, the Debian's patch to QEverCloud might get veeery complex.

from qevercloud.

Thinking of it more, I think the easiest way to work around this rigid Debian policy would be the following:

1. Make the generator generate QEverCloud sources from Thrift IDL at run time instead of build time. It is relatively easy because it's how it used to work before I switched the build system from qmake to CMake, most likely it'd be enough to simply remove the generated subdirectory within the source tree of QEverCloudGenerator + remove the add_subdirectory(generated) statement from the top level CMakeLists.txt. Then the build step would just build the parser which one would then need to run in order to generate the QEverCloud sources from Thrift IDL files. From QEverCloudGenerator's source tree the generation procedure would look like this then:
   mkdir build cd build cmake ../ make mkdir generated ./thrift_parser/thrift_parser ../thrift/src generated
   After this one would just need to copy the generated header and source files to relevant locations within the QEverCloud's source tree.
   For this I'd accept a PR because such a change in the generator's workflow makes a good sense to me.
2. For Debian you'd also need to remove the prepackaged lemon parser's sources from QEverCloudGenerator and instead use lemon parser from the corresponding Debian package. For this I'd accept a PR as well, however, only if it'd be generic enough i.e. if it won't use any hardcoded paths to lemon executable but instead would use CMake's find_package module. Something like the following would be required in thrift_parser/CMakeLists.txt:
   find_package(LEMON REQUIRED) LEMON_TARGET(ThriftParser thrift_lemon.y thrift_lemon.h thrift_lemon.cpp)
   Note that lemon would be required only as a build dependency for the generator, not as a runtime dependency.
3. Then you'd need to package QEverCloudGenerator for Debian. Hopefully after the two previous steps it wouldn't be very painful.
4. After that you could finally do the following patch to QEverCloud:
   1. Remove generated sources and headers
   2. Add Thrift IDL files

In addition to this, you'd need to add QEverCloudGenerator as a build dependency to QEverCloud package + add pre-build steps in debian/rules: before even starting to build QEverCloud (i.e. before executing CMake for it) you'd need to execute QEverCloudGenerator on these Thrift IDL files and put the sources/header it would generate to the relevant folders of QEverCloud. Then you could execute the remaining build steps as necessary.

The suggested patch in #4 would affect only the Debian package building but not the QEverCloud's own build system which I consider a good thing. In more general sense I still don't think it's of any use to run the generator on the Thrift IDL files during each build as they are not something that changes often or something that anyone except Evernote people would even want to change - serioulsy, it's just an abstract description of their API.

Sorry if this suggestion is less than you hoped for but it sounds like too much work for me to handle myself right now. On the bright side, if/when this is over, you'd be quite proficient in packaging hard stuff to Debian 😃

from qevercloud.

Take it easy, things won't be too difficult.

I could make QEverCloudGenerator a git submodule of QEverCloud in the debian source package to prevent the emergence of another debian package called qevercloud-generator.

I looked through the generator's code and it looks fine. On the first glance, using cmake one can easily build the needed cpp headers and files. Personally I don't think we need to switch to debian packaged lemon tool, but switching looks easy as well.

One more question: could you provide with the copyright status of thrift_parser/ directory, especially the thrift_lemon.y file? Thanks.

from qevercloud.

Ok, that way seems simpler, indeed.

The generator's sources including those in thrift_parser directory, were originally made by mgsxx and distributed under the MIT license. Note, however, that thrift directory (which is a git submodule in my generator's source tree) containing the actual Thrift IDL files belongs to Evernote, they distribute those files under the BSD-like license.

from qevercloud.

@hosiet, just FYI, Evernote guys are actually slowly update their APIs, see this PR for Thrift IDL files. I've tried to run the existing generator on the updated IDL files, no luck so far due to more complicated data model in their new API - some structs contain exception objects, some exceptions reference structs etc. I'd have to update the generator to handle this new stuff. And such quirks are very likely to occur with the future versions of their API. So "just replacing the Thrift IDL files and regenerating the new sources from them" is likely to not work in the real world.

The new API also means I'd have to release another major version of QEverCloud eventually - it would contain a ton of API breaks but since it's Evernote who controls that, the max I can do is to try minimizing the damage by ensuring the two major versions of the library can coexist within a single system. I don't think Evernote people are going to shut down the current API (1.25) any time soon as that would require updating all of their official SDKs (zero done at the moment) + give some time to thirdparty developers for adopting them.

from qevercloud.

I see. It seems that the shutdown of the old API will not take place in the near future, so we have some time. The biggest problem may be making nixnote2 adopt the new QEverCloud library.

FYI, I updated the qevercloud package to use the QEverCloudGenerator here (https://github.com/hosiet/qevercloud) as a "technical preview". The package contains some dirty hacks, though. You may take a look.

from qevercloud.

I've taken a look at it. Yeah, it's surely not "cmake && make" thing anymore 😃 Despite my feelings about the policy enforcing this, the work done so far looks impressive.

from qevercloud.