In the linear only case:
J = Vcm * m
where:
 J = impulse (vector)
 Vcm = change in velocity (vfvi) (vector)
 m = mass (scalar)
so the velocity change is J/m in the same direction as the impulse.
In the rotational case we can take the impulse equivalent of torque
2D case (using x and y components)  3D case (using vector equations) 
'impulse torque' = Jx * ry  Jy * rx = I * w  'impulse torque' = r × J= [I]w 
Combined Linear and Rotational
If the impulse is not in line with the centre of mass then there will be a change to both the linear and rotational velocity of the object. The aim is to derive an equation for the total change in velocity of a particle on a solid body due to an impulse on that point.
As described in the kinematics section the velocity of a point on a solid body is given by:
2D case (using x and y components)  3D case (using vector equations) 
Vpx = Vcmx  ry w Vpy = Vcmy + rx w 
Vp = Vcm + w × r 
where:
 Vp = linear velocity of a particle on a solid body (vector)
 Vcm = linear velocity of the centre of mass of solid body (vector)
 w = angular velocity of solid body around centre of mass (vector)
 × = vector cross product
 r = position of particle relative to centre of mass (vector)  Note: this is in absolute coordinates, not local body coordinates, so this will be a function of time as the body rotates.
We can write this equation in terms of changes to velocities:
2D case (using x and y components)  3D case (using vector equations) 
Vpx = Vcmx
 ry w Vpy = Vcmy + rx w 
The component of linear velocity due to rotation is: = x as derived here. so the change in linear velocity will be: Vp = Vcm + w x r 
Where:
 Vp = Vpf  Vpi
 Vcm = Vcmf  Vcmi
 w = wf  wi
This consists of a linear component and a rotational component.
For the linear component, as explained above,
linear impulse = J = Vcm * M
For the rotational component, since = x as defined here, then integrating both sides over the time of collision gives:
'impulse torque' = r × J= [I]w
where:
[I] = inertia tensor
This inertia tensor is a 3x3 matrix and since the other terms in this equation are defined in absolute coordinates the inertia tensor is defined in absolute coordinates, this means that the elements of the inertia tensor may be time dependant.
This makes the equations very complicated, so to simplify things we will start by using a single scalar value for I, this will only be valid for symmetrical objects but we can use this simple case for now, this gives:
w = (r x J)/I
where:
I = inertia scalar such that torque = I * alpha (only true when object is symmetrical)
Substituting these impulse expressions into the expression for the particle velocity gives:
2D case (using x and y components)  3D case (using vector equations) 
Vpx = Jx/m  ry (Jx ry  Jy
rx) /I Vpy = Jy/m + rx (Jx ry  Jy rx) /I 
Vp = J/M + ((r × J) × r)/I 
For the 2D case
We want to get an expression for J so we can start by grouping together the Jx and the Jy terms:
Vpx = Jx ( 1/m  ry^{2}/I)
+ Jy rx ry / I
Vpy = Jx rx ry / I + Jy ( 1/m 
rx^{2}/I)
Solving this for J gives:
Jx = (m/(1/m  rx^{2}/I  ry^{2}/I))*(Vpx
( 1/m  rx^{2}/I) Vpy rx
ry / I )
Jy = (m/(1/m  rx^{2}/I  ry^{2}/I))*(Vpx
rx ry / I + Vpy ( 1/m  ry^{2}/I))
For the 3D case
Again we want to solve for J, One option is to use the triple product identity which gives:
Vp = J/M + (r(J•r) + J(r•r))/I
Collecting the terms, this gives one part in the direction of impulse (J) vector and another part in the direction of vector r:
Vp = J( 1/M +(r•r)/I)  r(J•r)/I
As required J produces a greater velocity change if there is both linear and rotational motion. (thanks to databeaver for helping me out here)
Special cases
If J and r are parallel then the J(r•r)/I and r(J•r)/I terms cancel out and we get the one dimensional case as expected: Vp = J/M 

If J and r are at 90 degrees then J•r is zero and the equation becomes: Vp = J( 1/M +(r•r)/I) Again the change in velocity is in the direction of the impulse. 
In both the above special cases Vp will be in line with the impulse J, but for cases inbetween there will be a component in the direction of r, this means that an impulse in one direction can cause a change in velocity in a different direction (even when the object is symmetrical).
I would still like to get a more general solution for 3x3 inertia tensor,
Vp = J/M + ([I]^{1}(J × r)) × r
where:
[I]^{1} = Inverse of inertia tensor (3x3 matrix)
Alternative method using matrices
It seems like a good idea to express all terms using the same type of algebra, it is not possible to express inertia tensor purely in terms of vectors or quaternions so can we calculate purely in terms of matrices?
We can convert the vector cross product to a skew matrix as shown here.