Best practices for making it easy for contributors to run tests about discussions HOT 18 OPEN

@gr2m & @kytrinyx : If you are planning on using fetch for both the browser and nodejs versions, https://github.com/philschatz/fetch-vcr might be useful. It is a drop-in replacement for fetch.

I use it in octokat.js here and it is pretty easy to record and playback. Just run VCR_MODE=record npm test or VCR_MODE=cache npm test rather than the default VCR_MODE=playback npm test.

Some things that I ran into while trying to use the cassette format in https://github.com/vcr/vcr were:

• having the URL in the fixture filename was useful for debugging and deleting
• the has part of the fixture file contained the request headers (sorted)
• having 2 files for each fixture made it easier to inspect binary files

Also, I was only able to record fixtures in nodejs but fetch-vcr plays back the fixtures/cassettes fine.

from discussions.

shall we move https://github.com/gr2m/octokit-fixtures to https://github.com/octokit/fixtures?

from discussions.

The problem shared fixtures is when the API responses change we need to make sure to keep it up to date. My vote would be to find a solution that keeps us as close to GitHub being the source of truth as possible.

As of right now even the GitHub API documentation sometimes becomes out of date because the payload responses are old.

We also have the problem that if people use the preview headers we'd have to save the new response and old response.

It'd be cool if we could run the test suite and generate the cassettes or whatever needed when the PR comes to fruition.

from discussions.

Hey friends, I experimented this weekend with shared fixtures for the different Octokit libraries. The project runs an easy to extend list of scenarios which send requests against the GitHub REST API. It records requests/responses against and stores them as JSON fixtures for usage as a standalone http mock server or as a Node module. Here is the repository

https://github.com/gr2m/octokit-fixtures

I also made an introductory screencast.

@tarebyte let me know if the daily cron job creating pull requests on API changes addresses your concern.

@zeke I implemented the idea for "a portable language-agnostic binary" :) It works like a charm. It’s not yet automated, but it totally can be done, so we would attach the binaries to each release on GitHub, like here: https://github.com/gr2m/octokit-fixtures/releases/tag/v1.0.0

@kytrinyx the fixtures are recorded in JSON, very close to the format that nock is using. But changing that format or even adding additional formats like VCR should be totally doable.

I would be more than happy to move the repository to the @octokit organization. I’d suggest to publish the package to npm as @octokit/fixtures then.

I’d love to hear what you think <3

Cool features:

• A scenario is an object (or array of objects) to describe requests

  {
    method: 'get',
    url: 'https://api.github.com/repos/octocat/hello-world',
    headers: {
      Accept: 'application/vnd.github.v3+json'
    }
  }
  

• For more complex scenarios, asynchronous functions can be used
• Provides a standalone mock server utilising existing fixtures that all Octokit library developers can use to test against. The server can be downloaded as a single file all-batteries-included binary, no node required to run it.
• Checks ones per day if GitHub’s APIs have changed (using a Travis cron job). If there are changes, automatically creates (or updates) a pull request that looks like this: gr2m-fixtures-cat/sandbox#18

Blockers

• An paying account that we can use to record our fixtures, some of which will require private repositories
• A test GitHub organisation which can have unlimited private repositories
• A GitHub enterprise installation that we can test against, ideally one for all versions that are still widely used today.
• Rate limits for unauthenticated requests coming from Travis. I found a workaround using a proxy running on now: octokit/fixtures#3

from discussions.

Can you elaborate? Are we running into abuse rate limits? Or are we making more than 5000 calls per hour?

I’ve explained the problem in the issue at octokit/fixtures#3 in a bit more detail. Note that this is about unauthenticated requests, which only has a limit of 60 calls per minute I think, and it seems the requests coming from the travis build are counted against all other builds from other projects, so it’s constantly out of the rate limit, or maybe GitHub blocked it entirely by now

from discussions.

The ToS cleared us for this.

I've created the following:

• octokit-fixture-user-a (unlimited private repositories)
• octokit-fixture-user-b (public only)
• octokit-fixture-org (unlimited private repositories, unlimited seats)

We are going to need to figure out shared email aliases that we can use for the user logins. Also avatars! So important!

I will give the emails and passwords to @gr2m for now.

from discussions.

perhaps we could have a bot who creates recorded calls (authenticated by the user asking for it in order to avoid spammy users or rate limit abuse).

@gr2m and I were just talking about this. It could be as simple as a shared repo full of JSON payloads, etc. Another (maybe wild) idea we had was to a make a portable language-agnostic binary that could be used by every octokit. This could be written in Go, or perhaps in Node using the new pkg tool from the Zeit folks.

from discussions.

shared repo full of JSON payloads

JSON payloads would be a good start, but there's some critically important stuff in the headers that we'd need a way to capture.

Another (maybe wild) idea we had was to a make a portable language-agnostic binary that could be used by every octokit.

Something that would spin up a server that you could hit with fixture responses? That could be pretty neat!

from discussions.

Something that would spin up a server that you could hit with fixture responses? That could be pretty neat!

That’s the idea, yeah :)

from discussions.

As of right now even the GitHub API documentation sometimes becomes out of date because the payload responses are old.

Right, this would probably be even more at risk than that.

It'd be cool if we could run the test suite and generate the cassettes or whatever needed when the PR comes to fruition.

Yeah, if we had one environment that was always set up, everyone who needed cassettes could get it from there.

from discussions.

This is very exciting!

the fixtures are recorded in JSON, very close to the format that nock is using. But changing that format or even adding additional formats like VCR should be totally doable.

I think we should just settle on some sort of consistent machine-readable format that works, and then build adapters if we need to for each library (e.g. octokit-fixtures to VCR translation adapter for the Ruby octokit)

• [A] paying account that we can use to record our fixtures, some of which will require private repositories
• A test GitHub organisation which can have unlimited private repositories

These are both fixture-data accounts, one user account and one org account. We're on the edge of breaking the ToS here, by creating a user account for a fake user, but I think that we're allowed one bot account for stuff like this.

I can make this happen. What do we want to call them, though?

We could be very explicit and say:

• octokit-fixture-user
• octokit-fixture-org

• A GitHub enterprise installation that we can test against, ideally one for all versions that are still widely used today.

I am going to need to ask for help internally on this one.

• Rate limits for unauthenticated requests coming from Travis.

Can you elaborate? Are we running into abuse rate limits? Or are we making more than 5000 calls per hour?

from discussions.

I think we should just settle on some sort of consistent machine-readable format that works, and then build adapters if we need to for each library (e.g. octokit-fixtures to VCR translation adapter for the Ruby octokit)

note that there is also the option of not using the fixtures at all, but using the mock server instead as a drop-in replacement for api.github.com or the respective APIs for GHE installations. Not sure if that is a viable option though, I’d love to here from the maintainers of the other Octokits if that would be helpful to them

from discussions.

We could be very explicit and say:

• octokit-fixture-user
• octokit-fixture-org

Yeah +1 on being very explicit. I do think that we will need more than one user down the road, as some APIs are about interactions between multiple users. Maybe GitHub could make an exception of its ToS if it would adapt the octokit-fixtures project as one of its own? I dunno how that works :) I mean it’s in GitHubs own best interest to have a good test suite for its SDKs I think?

from discussions.

This project would be for all languages, not just node.

When using the fixtures to test the JavaScript SDK, I’d use nock when testing in node as its faster, when testing in the browser I’d spawn the mock server separately and then run the tests in all browsers we support using selenium.

I wish I had talked to you first, let’s definitely chat on the fixtures you created and how! For my fixtures, I normalised things like timestamps and counts so that I can compare when recording again, to catch changes in GitHUb’s API. Looking forward to chat more on that on our call!

from discussions.

I wish I had talked to you first

better late than never!

from discussions.

I'm asking the ToS team for input on our fixture needs, by the way.

from discussions.

I just came across https://github.com/slackapi/steno and it looks really interesting. Has anyone played with it?

from discussions.

👍 yes, that'd be awesome

from discussions.