Testing about guide HOT 34 CLOSED

This is an interesting one, it's basically the Meteor Testing Manual topics :) Of course, my commercial endeavors should not in any way derail or affect this guide. I'd love to find a way to make them complimentary so there's less duplication.

What are you thinking about frameworks? Will this guide be about TinyTest, Velocity? some hybrid? My vote of course would be to go for Velocity + Jasmine since it supports all the in-app testing modes as well as the package testing approach. It's also the most popular of all the frameworks. We made a velocity-cli npm package that provides commands like test-app, test-package xxxx and test-packages which makes it much easier for users to get started. Have you seen the official Velocity docs btw?

As for the subjects, I think they are a great set to get started on, the only one I don't see is package testing.

One question I get asked about often is authentication. We created a GitHub stub (OAuth2) and a Twitter stub (OAuth1), and both of these repos have been cloned for a Facebook, Google and LinkedIn stubs by other users. These help a lot.

Another question often asked is about clearing data between tests, we have a package for that too xolvio:cleaner

Another one is email testing. Again, we created a stub

I think a section similar to point 3 that is about how to isolate Meteor framework pieces like template helpers and events would be good. You guessed it, we have a package for that too :)

from guide.

Ok I'll get started reading everything you linked to!

One question is, what parts should we cover here, and what stuff should I just be linking to from your guides?

from guide.

Package testing will probably go in the package section, although I would love to recommend something other than tinytest if it makes sense to do so.

from guide.

what parts should we cover here, and what stuff should I just be linking to from your guides?

top of my head, I'd say setup, configuration and docs like command-line switches and options should all go in the velocity website.

Guidelines and techniques can all go in here. That would create a good natural split and allows us to move some of the content from the velocity website like data setup.

@sanjo do you have some thought here?

I would love to recommend something other than tinytest if it makes sense to do so

It's really easy to do package testing with Jasmine, and to get all the benefits of a full testing framework like spies that teardown themselves amongst a ton of other goodness, and not the TinyTest tinyness! The instructions to get setup are here

In a nutshell, you have to do this:

Package.onTest(function(api) {
  api.use('sanjo:[email protected]');
  ...

And then use the velocity-cli like this:

velocity test-package package-to-test

So I'm saying I think it make sense.

from guide.

Technically, Jasmine is using TinyTest as a vehicle

from guide.

Testing is one of the driving factors for the development of the Meteor-Space project: https://github.com/meteor-space/base#3-testability

Linking for intererst

from guide.

Check also our package classy-tests which allows you to specify tests which have to interact between client and server (so you can do something on the server, observe on the client results, etc.).

from guide.

Still need to do a bunch more investigation on this one but some questions for @samhatoum et al:

1. What do people tend to use to generate test data?
   • We built factory with @dburles, (actually @timbotnik and I recently re-wrote something with the same API that works a little differently behind the scenes, can explain how if anyone's interested).
   • The biggest problem with a factory approach is it's a little slow, we've some ideas about this but I'd be interested in what others are doing.
2. What do people use to mock on the client? We've recently written (and conceptualized) two packages that seem like they solve most client side problems:
   • "stub-collections" which replaces all collections with local collections (this exists, will publish soon if people want it)
   • "stub-publications" which allows you to override subscriptions to dump data into stubbed collections via the factories above. (this doesn't actually exist, but would be trivial to implement).

from guide.

@tmeasday Answer to previous post:

1. For factories I settled on this base class for my factories. You can inherit from it for your concrete factories and overwrite getDefaultData and if needed extend build with more options. It supports simple schema data cleaning. deepmerge is a npm package for merging objects.

class BaseFactory {
  constructor(schema) {
    this.schema = schema;
  }

  getDefaultData() {
    return {};
  }

  build(customData = {}) {
    var data = deepmerge(this.getDefaultData(), customData);
    if (this.schema) {
      data = this.schema.clean(data);
    }
    return data;
  }
};

• I also use local collections for all unit / integration tests. Each test gets new empty collections. In our app we have an application class that creates the whole application. We just create a new application for each test.
• For our Cucumber end to end tests we have Meteor methods in a local debugOnly: true testing package, that allow us to create data fixtures in the database. For unit tests I didn't need to mock subscriptions so far. I just insert fixture data into the local collections to prepare the data for the test. We don't use template subscriptions right now.

from guide.

The debugOnly testing package with fixtures is reusable from integration tests.

Also, here's an approach to clearing data really quickly:
https://github.com/xolvio/cleaner/blob/master/cleaner.js

We use this code in a resetApp method

from guide.

Hey guys, thanks for the quick responses.

1. @sanjo re: your factory pattern -- but that doesn't help with relations correct? I say this because the main complexity with the factory pattern is how to make something like Factory.create('student') make sense when a student is required to be in class, that's required to have a teacher, which probably needs a school, etc. (This is also the point of difference in our new implementation).
2. I'll need to dig into this "new application" approach but it sounds conceptually the same, cool.
3. Creating fixtures via methods sounds useful for integration tests, for sure. We also built factory-client which attempted to automate this, but honestly ended up being a big pain because it means you need to wait on a bunch of stuff before each test, and of course tinytest isn't very helpful for this kind of thing. Perhaps the issue was using tinytest in the first place.
4. I'm really interested in what you guys think about the stub collections/subs approach (Possibly I'm not even using the right terminology!). I'll try to get something concrete out there early next week for you to play with.

from guide.

In addition to using factories to create data, a good approach to maintaining test-data integrity is to use your app's services to create data in the same way the app would create them at runtime. This way you won't have to worry about changing the factories every time you change your schema.

For example, in the case you described, the app would likely have services to create classes, assign teacher to classes, enroll student in classes etc, so you can use those services and if you have a local collection, this data setup would be quick.

Test data is always a pain! The trick is to make it a choice between a slap and a kick :)

Looking forward to seeing 4.

from guide.

I think testing chapter should definitely cover also CI testing. Once you move all app code to packages (which is what I think we all agree is a good structure anyway for bigger projects), then you can use Tinytest (or class based wrapper around it) to define package tests. And then you can use things like travis-ci-meteor-packages for packages and Travis CI or test-runner for whole apps and CircleCI.

from guide.

Answer to #20 (comment)

re: your factory pattern -- but that doesn't help with relations correct? I say this because the main complexity with the factory pattern is how to make something like Factory.create('student') make sense when a student is required to be in class, that's required to have a teacher, which probably needs a school, etc. (This is also the point of difference in our new implementation).

Here is an example for a user that is member of an organization:

UserFactory = class UserFactory extends BaseFactory {
  constructor() {
    super(Simian.Schemas.User);
  }

  getDefaultData() {
    return {
      organizationId: Random.id(),
      // Other default data
    };
  }

  build(customData, {organization} = {}) {
    const user = super.build(customData);
    if (organization) {
      user.organizationId = organization._id;
    }

    return user;
  }
};

OrganizationFactory = class OrganizationFactory extends BaseFactory {
  constructor() {
    super(Simian.Schemas.Organization);
  }

  getDefaultData() {
    return {
      name: 'acme'
    };
  }
};

Factory = {
  user: new UserFactory(),
  organization: new OrganizationFactory()
};

// Creating a user with organization
const organization = Factory.organization.build();
const user = Factory.user.build({}, {organization});

So you connect the models explicitly when you build your models.

The idea with the "new application" thing is basically that we treat our application instance not as singleton. Therefore it is easy to create a clean state application instance for each test.

I had to write a little hack for Meteor.methods to allow declaring already defined methods again to make this work.

Jasmine and Mocha have support for asynchronous beforeEach blocks. With this it is easy.

from guide.

For continuous integration we at Xolv.io have done it on Codeship, CircleCI and Travis. So we probably can contribute configurations for those the setup the environment for Meteor and testing (with Velocity or TinyTest).

We also have done work on parallelization for testing. We will merge this work into Velocity CLI hopefully soon. This work allows you to easily distribute all your tests between testing machines (nodes) and you can also run tests for multiple packages in parallel with one command.

from guide.

BTW, check this how to interleave client-side and server-side assertions. I think something like that should also be in the guide. (Maybe as a general pattern?)

from guide.

I have a half finished blog post that I plan to wrap up soon and would be happy to document that pattern for the guide. It just uses Jasmine, I implemented it for Ongoworks.

from guide.

@queso the client-server assertions pattern?

from guide.

No, I am running the publications and testing them, more integration style: https://github.com/reactioncommerce/reaction-core/blob/development/tests/jasmine/server/integration/publications.js#L59-L62

from guide.

Ahh, yes. I like that, I think I've done something very similar in the past.

from guide.

@stubailo I wonder what you meant by testing routes?

from guide.

I'm not sure I would see much value in testing routes, probably better to do something like cucumber to full stack testing.

from guide.

There's definitely a lot of value in testing routes e2e and the are likely to be the least setup cost, at the expense of testing speed of course. For unit / integration testing, I can see the following needs at the testing phases:

For setup, we need to control the dependencies of units, like subscriptions and injected context.

For execute, I may have a unit that only runs when a particular route is hit. I'd want to the ability to invoke that unit in isolation so I can test its execution paths within it. What this typically means is either having helper methods inside the router or monkey patching the router.

For verify, I may want to spy on a router to make sure my unit is calling it with the right params or within the right context.

from guide.

@samhatoum I guess I am wondering about this in a world where the router is no longer responsible for things like subscriptions.

In a FR-style world the router's job is pretty simple, just parse a URL and pick a template + layout to render with some params. Perhaps there are some analytics style stuff and a little redirecting going on.

Possibly that's all worth testing but
a) I'm not sure you aren't just testing the router implementation, and
b) it's tricky because the router wants to take over the URL bar, which tends to futz the test runner up.

from guide.

Again, I will just say, I think router testing will be a very low value activity in the long run. My gut says e2e testing really gives you what you are after.

I agree with @tmeasday that in a FR world where we lack subscriptions, the idea of controllers, etc. Seems like a lot less to test. I guess the point here is that Flow Router is the blessed path moving forward in the routing guide, right?

from guide.

As a guide that helps people test routing, I think it ought to how people how to test routing in different scenarios. The E2E testing approach is by far the easiest, but it's not necessarily always the best option.

Consider this code from the FR README:

FlowRouter.route('/home', {
// ... snipped extra code
function trackRouteClose(context) {
  Mixpanel.track("move-from-home", context.queryParams);
}

To test this with an E2E test, I have to write a Mixpanel stub that tracks the calls somewhere, get my browser to actually hit the site, then verify that the MixPanel stub contains what I expect. I would have to do this for every route where Mixpanel gets called. This is very wasteful as my test suite would get slower the more I shove into the UI layer (refer to the testing pyramid)

Unit testing is far more appropriate here, so to do the same, I would need to do something like this:

it('route close events tracks the context query params', function() {
  spyOn(Mixpanel, 'track');

  var route = FlowRouter.interceptor.getRoute('/home');
  route.trackRouteClose({queryParams: 'theQueryParams'});

  expect(Mixpanel.track).toHaveBeenCalledWith(jasmine.any(String), 'theQueryParams')
})

Notice the FlowRouter.interceptor. This would be a testing package that augments the FlowRouter namespace and intercept all calls to FlowRouter.route to store the routes for easy isolation. We use this pattern with the template isolator to easily isolate and test events.

Now I can test 1000 different Mixpanel events in around 100ms.

With regards to subscriptions, totally understand that they are not central to FR, however they are not the only dependency to create test doubles for. Consider this code from the FR README:

FlowRouter.route('/blog/:postId', {
    subscriptions: function(params, queryParams) {
        this.register('myPost', Meteor.subscribe('blogPost', params.postId));
    }
});

In order to test that the blog post route register the correct subscription for a specific route, we would do:

it('register the correct subscription for a specific route', function() {
  var thisContext = {
    register: jasmine.createSpy('registerSpy');
  };
  spyOn(Meteor, 'subscribe').and.returnValue('theSubscription');

  var route = FlowRouter.interceptor.getRoute('/blog/:postId');
  route.subscriptions.apply(thisContext, [null, { params: 'thePost' }]);

  expect(Meteor.subscribe).toHaveBeenCalledWith('blogPost', 'thePost')
  expect(thisContext.register).toHaveBeenCalledWith('myPost', 'theSubscription')
});

In this test the this context contains a register method that we need to create a test double for, in order to isolate the unit of code we're interested in.

a) I'm not sure you aren't just testing the router implementation

None of the above it testing the router itself, but instead we are isolating the units that we have written so that they can be tested.

b) it's tricky because the router wants to take over the URL bar, which tends to futz the test runner up.

After isolating user-written units, you can also spy on the Router.go method and bypass the window location changing at all.

I think as a guide, we need to cover both cases of how to do E2E testing for routing related tasks and also how to isolate the code and create test doubles as to exhaustively test the units being created.

from guide.

With regards to subscriptions, totally understand that they are not central to FR, however they are not the only dependency to create test doubles for. Consider this code from the FR README:

But again, I think testing things like subscriptions are a moot point for routing testing going forward.
@arunoda recently said that subscription stuff will be taken out of Flow Router: https://kadira.io/blog/meteor/flow-router-4-0-and-future-of-routing-in-meteor.

from guide.

Agreed. And subscriptions are not my focus here.

I just want to iterate that my point here is that the guide should show developers how to isolate units of code they have written, and how to control any dependency in those units.

In the context of a router, that means knowing how to isolate code that developers create inside the route definitions and how to use spies within them for typical routing use cases.

I should also add that decoupling the units from the route definitions is a good way to achieve isolation through modularity. Decoupling makes it easy to write unit tests, however, it still does not cover all cases since we need to ensure we have wired things up correctly through integration testing where again, we will need apply isolation techniques and dependency control.

In the context of router testing, here's how I see the different testing levels being applied:

End-to-end testing: Ensuring the value to the user can be attained (I can see blog posts)

Integration testing: All the wiring works (data in this route rendered template passes through subscription security rules)

Unit testing: Logic and permutations (fire analytics event A when params XYZ are passed in)

from guide.

@tmeasday: @avital and @yyx990803 came up with a synchronous fiberized wrapper for webdriver: https://github.com/meteor/e2e/blob/master/lib/wd_with_sync.js

This could be relevant for the acceptance/end to end section.

from guide.

We have done this in Chimp also. Currently cucumber only but very soon jasmine/mocha also

from guide.

FYI, I've recently used Nightwatch.js for a personal project and have to say it's pleasantly easy to use and fairly robust. I probably would've gone with it instead of writing our own e2e test runner if I were to do it again.

from guide.

@stubailo As Sam said we have done the same for WebdriverIO. It is currently used by Chimp, our end-to-end testing solution. But the it is also available as standalone npm package: https://www.npmjs.com/package/xolvio-sync-webdriverio.

@dandv has done a comparison of the available testing tools and chose Chimp as a winner. You can read more about it here: http://wiki.dandascalescu.com/howtos/from_zero_to_automated_web_app_testing_best_practices. Chimp is also available as xolvio:cucumber for Meteor. ;-) We have planned to also support other testing frameworks besides Cucumber. Mainly Jasmine and Mocha.

from guide.

In the last month, Clinical Meteor has gone live with a release-track wide QA infrastructure on Circle CI using Gagarin and Nightwatch (both of which are built on Mocha or have Mocha support). It's working perfectly, and PR requests are starting to ramp up across the entire distro.

We're currently extracting code samples across 2 dozen packages (and 3 or 4 reference apps) which are already relevant to the Guide's list of stated topics, and will be providing a QA cookbook for apps that need to go under FDA, CCHIT, or other regulatory review.

And in reference to Evan's comment above: Nightwatch has been part of the broader Meteor ecosystem since ~0.6.5 days; and has migrated apps from PHP to Meteor, managed the Spark to Blaze migration; and is gearing up to migration dozens of packages and apps from Blaze to React.

Also, there's the Starrynight utility which has commands for generating Nightwatch boilerplate; refactoring tests from Blaze components; scanning packages for Nightwatch commands and assertions; autoconfiguration of the Nightwatch config file; deployment to CI servers, and more.

We would respectfully ask the MDG to give Starrynight, Nightwatch, and Gagarin due consideration as officially recommended testing frameworks in the Guide. Particularly if MDG has any plans for the healthcare market.

from guide.

And a truly excellent tutorial from Martin on Continuous Deployment:
https://martinhbramwell.github.io/Meteor-CI-Tutorial/index.html

from guide.