How would I securely generate a password that contains certain characters?

When creating a password sometimes I need the password to contain uppercase, lowercase, number or a symbol. If I create the password like this it's not guaranteed that each is represented:

$input = 'a..zA..Z0..9$!_'; // generally the whole allowed alphabet
while (strlen($password) < 10) {
  $password .= securelyPickRandomCharacterFromString($input);
}

In this case there is a chance that the random process (even though it's safe and completely random) will only select lowercase letters.

I can think of naive approaches like first selecting 1 from each group and then filling the rest randomly. But that would not be cryptographically safe as say random_bytes() is. Maybe shuffle after?

What would be a good and safe algorithm to generate the password containing each "group" of characters.

Note: The password requirements are usually for external system and are out of my control.