Unify with boost-feedstock about boost-cpp-feedstock HOT 21 CLOSED

I think before we try this for boost-cpp, we should figure out a way for qt. Although we have agreed upon manual uploads, look at the mess in qt. https://anaconda.org/conda-forge/qt/files some build numbers have only one platform. Some are built with cb3 and some with cb2. No build logs. If and that's a big if qt became a good example about building packages manually, we can have a conversation about boost-cpp. With the mess that is qt, I'm strongly against moving boost to a manual upload approach.

from boost-cpp-feedstock.

Sorry, the CI time was longer because we were building the C++ libraries multiple times because there were no multiple outputs supported then.

from boost-cpp-feedstock.

Sure, I'll give it a shot. Fingers crossed. Might be a couple of days, but I'll try to get it done soon.

The main thing that is taking time is resolving merge conflicts. I have about 15 out 150 left to do. I will probably circle back around and re-do some of them later - ones where there is a large difference in how each system does things, such as handling X11 dependencies or blas. Where there is a large gap like that, I have tried to use variants to toggle between the two ways of doing things. For example, ffmpeg in defaults has no GPL stuff, while CF does have that. There's a new variable in the config called GPL_ok to switch between the two. I expect we'll have one value in defaults (0), and CF will have the other (1).

Then I'll need to build each recipe to make sure that it still works. Finally, I'll put up a PR from AnacondaRecipes to CF for each repo.

I think first I'll try to push up all the recipes to branches on respective feedstocks on anaconda recipes. Because I may have screwed up the merge, they can't go right into master branches on AnacondaRecipes. If anyone wants to join the testing/debugging effort, we should work out a way for you to "claim" a given feedstock.

from boost-cpp-feedstock.

Part of the issue is we were starting to run into time limits on CI when this was one feedstock.

from boost-cpp-feedstock.

Yeah, that's definitely an issue. We need to discuss this at a future meeting. Since we now have different recipes by necessity of build time requirements, that will create friction. Is it better to keep that friction, or is it better to have the friction of needing to do the builds externally (like we do for things like Qt)?

Asking as a topic for discussion in future meeting, more than discussion here, per se.

from boost-cpp-feedstock.

Loosing boost as a feedstock would be a huge loss for me personally and those I work with.

Independently many things depend on boost in conda-forge. So we would be exchanging one kind of friction for another.

With qt I understand the situation a bit more clearly as there really isn't another option. However building boost with different options is pretty trivial. Can you please elaborate a bit more on the friction this causes for you (Anaconda)?

from boost-cpp-feedstock.

The friction is that there is no shared recipe. We are not going to undo our merge into a single recipe - it saves us a lot of build time and greatly simplifies maintenance of boost/boost-python. With no shared recipe, it'll be harder (but by no means impossible) to keep in sync and keep everything compatible.

You aren't losing it as a feedstock. You're losing the build on CI. Yes, there's friction either way. Either way, the cost is in human time - humans building packages, or humans ensuring that recipes on either side are compatible.

from boost-cpp-feedstock.

You're losing the build on CI.

To be clear, this is the concern (and to my way of thinking that is a huge part of what a feedstock is otherwise it is just a recipe). If you have an alternative in mind that would still allow us access to a build worker of some kind, would be happy to discuss that. Right now it's not clear to me that option exists outside of the free CIs.

from boost-cpp-feedstock.

We all have computers. Those are build workers. That's how we treat Qt, because it's the only option there. I think you're hung up on the chain-of-custody and proof of build logs. That's OK, it is a good thing to be concerned about. There really is no perfect solution right now. I believe that the burden of reconciling recipes is greater than the burden of some trusted entity building and uploading packages outside of the standard free CI workflow. The burden of the former is manual labor in tracking updates, as well as potential binary incompatibility, and the myriad complaints of broken software that result. I see the burden of the latter is what I mentioned before about concern over trust, plus perhaps some additional overhead in debugging, especially for people without ready access to windows or mac build machines (although free win VMs are readily obtainable).

I also believe that the non-ci builds can be augmented with steps that may help your conscience, such as uploading build logs and some sort of cryptographic signing. Note that cryptographic signing currently would be something external to conda/conda-build - we are working on that feature, but it is not currently available.

I highly doubt that we will reach consensus by commenting here, and recommend that this be brought up at the next meeting, and perhaps for a vote by the community some time later.

from boost-cpp-feedstock.

So IMHO the community that conda-forge has is really vital to solving these issues surrounding Boost as it is used in all corners of the C++ world, which is why having CI here that builds and deploys on demand really helps as we can iterate quickly on fixing those issues while drawing on many people's expertise. IOW CI is only part of the discussion here, but it is a vital part.

Sure, happy to continue this in a meeting. Just wanted to let you know this is a sensitive issue for us.

from boost-cpp-feedstock.

I'm not trying to take away people's power to build their own packages. Yes, that is essential. CI is a means to that end. It is not the only means to that end. Let's think about this problem in terms of constraints:

1. Where software must be built (public/private)? What is the build env that we expect?
2. What artifacts must be present for a build to be trusted? Just the package, or logs, too? Signatures or other verification mechanism?
3. What implications are for users to be able to build and debug their recipes on any given platform?
4. What is the desired SLA, in terms of time for user to get a response to their request for a build, not including build time?

Feel free to add your own.

There are many possibilities, but matching the current CI exactly is unnecessarily constraining. For example, there can be a build farm that we (CF) set up that is private to Conda-forge core maintainers, based on c3i. If it's private to conda-forge that would get around our current limitations with c3i that we expose too many secrets. This would not connect well with the github workflow. It would absolutely achieve the build/deploy goals, though.

I think we can agree that having software that just works is paramount.

You know well that C++ is incredibly sensitive to small differences in configuration. It is worth the time to figure out alternative build/deploy strategies if it mitigates risk in having software break. We will not realistically be able to maintain binary compatibility between CF and Anaconda if we have a piecemeal collection of manually synchronized recipes, especially as core as boost is. We've been there. We are there right now. It does not work.

from boost-cpp-feedstock.

That's fair.

I'm not proposing a manual upload approach exactly. I'm proposing a non-CI build/upload approach. It can be automated, and c3i is one way to automate it. With c3i, build/upload is automatic, but triggering builds is not. Setting that up and maintaining it is work, though, and Anaconda can't shoulder that alone. As you have wisely pointed out, you already have a problem. We can help you solve it, but we will not solve it for you and then have you either accept or reject depending on if you like it.

In the meantime, I won't submit PRs from AnacondaRecipes to conda-forge for any boost recipes. If anyone else would like to split our recipe changes out into the different feedstocks, they are welcome to do so. I will not build packages for boost as part of the bootstrap effort, since our recipes will not match. I will use defaults packages to build against boost dependencies, with a similar version pin to that available in CF.

from boost-cpp-feedstock.

As noted before, am interested in discussing other build worker designs. Automation of this build and upload while providing access to the larger community is key (thus far we have called that CI). Obviously the current free CI solution is limited and it has never been the plan to keep them indefinitely (at least not to my understanding). They are just the current constraint that we have been dealing with. Maybe this discussion of build/upload design should be a webpage repo issue and/or an item on the (soon to be created) mailing list?

from boost-cpp-feedstock.

Also agree with @isuruf's point about qt. That remains a stumbling block for us internally. Admittedly the stumbling block is updating it. However I would think that problem would have been solved by now if it were properly simplified and incentivized by tying that feedstock into some sort of build service that could handle it's requirements. That way the community could have passed this baton around and eventually solved it.

from boost-cpp-feedstock.

I see both points and I do think that the long term goal should be unifying the recipes.
At the moment we need some actionable items to seek a solution:

1. Where is the CI bottleneck, CircleCI, Travis-CI, AppVeyor ot all of them?
   If it is Travis-CI and AppVeyor we can try @isuruf's CircleCI solution for OS X and ask AppVeyor for more time. (They always granted us more time when requested.)

2. Manual uploads with personal computers will always be messy, there is no way around it.
   However, if Mike can take that burden to c3i, we may have a clean way to build those without the headaches we have with qt.

PS: I am investigating 1 for qt right now, more on that soon...

from boost-cpp-feedstock.

some build numbers have only one platform

Why is this a problem? It makes no sense to recomplie such big packages for all platforms when some changes only affect to one of them.

No build logs

Why is this needed? Qt build logs are probably on the order of 100k lines and only the first 1000 or so are useful. They could be uploaded by maintainers if required, though (e.g. to a gist).

With the mess that is qt, I'm strongly against moving boost to a manual upload approach.

It is messy, sure, but manageable. Besides, it has worked well and has given conda-forge feature parity with defaults.

I'm not advocating for maintaining the status quo, but I find the mentioned problems minimal (almost irrelevant) in comparison with the service provided to the community.

from boost-cpp-feedstock.

Why is this a problem? It makes no sense to recomplie such big packages for all platforms when some changes only affect to one of them.

It's not. Some builds have pinning updates and these are not built for example osx.

Why is this needed? Qt build logs are probably on the order of 100k lines and only the first 1000 or so are useful. They could be uploaded by maintainers if required, though (e.g. to a gist).

Why build logs are needed? That's the first time I heard that argument. One of the complaints I've heard is that conda-forge's build logs are less verbose than desired.

It is messy, sure, but manageable. Besides, it has worked well and has given conda-forge feature parity with defaults.

I agree. It's messy, but it serves a purpose in the case of qt.

from boost-cpp-feedstock.

@msarahan, can you send a PR for boost-feedstock? I think we don't need two feedstocks. There were two reasons for when this PR was split.

1. Outputs not supported at the time - this is not an issue anymore.
2. CI time - this is reduced since we are not building for multiple versions of numpy anymore and circle-ci now has increased their limits.

from boost-cpp-feedstock.

One other thing: boost copies files one-by-one, printing to stdout for each file. That's incredibly slow. We can probably dramatically cut down the build time by hacking those copies out of their build scripts and replacing them with more efficient bulk operations.

from boost-cpp-feedstock.

Why build logs are needed? That's the first time I heard that argument. One of the complaints I've heard is that conda-forge's build logs are less verbose than desired.

When the log is short and you can read it from top to bottom to spot for errors or strange behavior, I agree they are very useful.

But in a 100K log, that's hardly possible.

from boost-cpp-feedstock.

Closed by conda-forge/boost-feedstock#164

from boost-cpp-feedstock.