A/B test is not tracking alternatives about vanity HOT 7 CLOSED

I'm having the same issue on an app we're integrating Vanity into, with avery similar setup. The X-Factor I think might play into things is Authlogic
and it's session maintenance. The vanity_id cookie token appears to remain
unchanged even after Authlogic has gone through it's session maintenance
during user creation. However, it still fails to correlate an experiment
participant with it's subsequent conversion, so something (presumably related to
the session's modification) must have been affected.

In the interim, I've found an apparent workaround (using akahn's
UsersController#create as an example):

def create
  @user = User.new(params[:user])
  if @user.save_without_session_maintenance
    track! :signup
    UserSession.create(@user)
    flash[:message] = "Thank you for signing up! You are now logged in."
    redirect_back_or_default account_url
  else  
    render :action => :new
  end
end

This defers the transformation of the session for long enough to have Vanity
successfully correllate the experiment and conversion.

from vanity.

Interesting, thanks for that workaround. I'll give that a try.

from vanity.

The problem is definitely with the vanity_identity. If you are trying to track things that start as an unregistered user and convert as a registered user, then the identity changes from the hex that is assigned to the vanity_id cookie to current_user.id. If you try to track a conversion with an identity that hasn't previously been added as an experiment participant, it will fail silently.

Honestly I'm very surprised you're having this issue, as we seem to be having the opposite issue at my office. Somehow, somewhere, vanity_identity is being called before ab_test is called in our UsersController. vanity_identity memoizes the identity, so even though we call ab_test after current_user returns a valid user object, the hex has already been memoized as the vanity_identity.

I can post some code if you like.

from vanity.

Hah actually I just discovered that our problem is because of my office's optimization to avoid hitting our DB if the user doesn't have any user_credentials cookie yet (as most of our traffic is first-time visitors).

def current_user
  return nil if cookies[:user_credentials].blank?
  @current_user ||= current_user_session && current_user_session.record
end

Anyway your problem is still a very valid one, and I think that Vanity should transparently convert a user's recorded identity in Redis from what was stored in the cookie to user.id (when it detects that the user has the cookie AND a user object).

Otherwise, vanity_identity could simply check for a cookie first, use that if it's there, then fall back to using the user's id. That seems like maybe the simplest solution.

-Steve

from vanity.

I agree that there should be some sort of handling of the cookie's vanity_id and the user.id (or any other id we wish to use). I'm not sure simply replacing the identity in the database will work. There needs to be some sort of an alias method. KISSmetrics does this well. You can use several identifiers for the same person by aliasing them together. Whether their identity comes from the vanity_id in the cookie or user.id, they are the same person. They should be treated as the same person.

I can imagine having an experiment occur on a non-logged in page, then tracking a conversion when the person is logged in. Currently, this will not be tracked as shown by the OP. They are seen as 2 different people.

This seems like something people assume Vanity will take care of, but it doesn't. Many people may see errors in their statistics and not know why as this is not posted clearly somewhere. I accidentally came across this bug and now have a trade-off to make which I wasn't aware of before.

Wesley

from vanity.

Hmmm. I'd imagine this is still a fairly big issue for people. Including me.

I may look at it shortly.

from vanity.

Closing as this is fixed in #49, will centralize comments there. Also related to #140 .

from vanity.