Each Ability belongs to a certain branch marked by a special color. These branches are: Combat (red), Signs (blue), Alchemy (green), and General (brown). With the exception of the general branch, the first row of each (the core abilities) are unlocked at the start. Click on one of the above tabs to see that particular branch's abilities.
Note that after the second row, a certain number of points are required in that branch before they can be unlocked. However, you do not need to have the previous skill in a column to unlock the one below it. For example, if you don't have any points in Undying, but want Razor Focus, you can still put points in the latter once you've unlocked the row.
Once an ability is unlocked, they can then be put into an ability slot. You can place an ability in any of the four groups, but if all 3 in a group are the same color, you receive an additional bonus depending on the color of the connected mutagen. To get the most out of your abilities, place mutagens into the diamond-shaped slots on the Character panel that corresponds to that group's color. Red gives a bonus to attack power, blue to sign intensity, and green to vitality. This rule does not apply to the general abilities (brown).