A useful rule of thumb in maths, I think, is that "the only problems we like to solve are quadratic equations", ie if we can write an expression so it looks like X^2 + bX + c = 0 then that's now a solvable problem. So I think the first step is to check that we can even do that. Ask yourself the following questions:
1. Is it already in this shape? ie, are the powers of my variable 2,1,0?
2. If not, are they [twice something], [something], and 0?
3. If not, are they [positive thing], 0, [negative a thing]?
4. If not, then, if I write the terms in order of biggest power, middle power, smallest power, then does biggest + smallest = twice the middle?
Example:
1. x^2 + 5x + 4 is in the right shape already.
2. x^4 + 5x^2 + 4 is the same as x^[2*2]+ 5 x^[2]+4 .
3. x + 5 + 4/x is the same as x^[1] + 5 x^[0] + 4 x^[-1].
4. x + 5*sqrt(x)+4 : we can write t = t^(2* 1/2) and sqrt(t) = t^(1/2).
5. t^(11/5) + 5 t^(7/5) + 4 t^(3/5) , 11/5 + 3/5 = 14/5 = 2*7/5.
Every single one of these problems factors nicely as a result, and in more or less the same way, looking like (X + 4)(X + 1) for some clever choice of X.
If you have two variables in the problem, like the first example, then the first thing you should try is to divide everything by the highest power of one of the variables. So:
(w^2 - 3 wz - 10 z^2)/z^2 = w^2/z^2 - 3 wz/z^2 - 10 z^2/z^2
Two powers of z cancel in the last term, and one in the second term, so this is the same as w^2/z^2 - 3 w/z - 10 , which we can write as (w/z)^2 - 3 (w/z) - 10. This is now in the right shape, because we have an expression that looks like "thingy squared + thingy + constant".
This still leaves the challenge of factoring, but I'd regard that as a separate problem from knowing that you can even try to factor.
Bottom line: ask yourself if there is anything you can do to make your expression look like X^2 + bX + c. Clever choice of substitution, divide or multiply by a power of X, etc etc. I can't provide an exhaustive list of what the answers might be, but the skill is to ask the question in the first place.