Add support for placement anti-affinity

[manojnkumar]

To support a production cluster, the control plane nodes should be on different servers. The same goes for the compute nodes. PowerVS anti-affinity placement rules should be applied to nodes during their deployment.

[yussufsh]

There is a technical challenge to use affinity policies with PowerVS. For specifying the affinity or anti-affinity we need the id of other VMs which is not practical as the VMs are created in parallel for us.

Another option is to use ibm_pi_placement_group and use the placement group id while creating the VMs. The problem 'again' here is that we cannot use count, instances should be created one after the other. This can be done for master giving 3 configurations based on count, we cannot determine the number of workers and not feasible to write N number of configurations for workers.

@cs-zhang could you check with the PowerVS team on how to proceed on this?

[cs-zhang]

I tested with pi_replication_policy with pi_replicants, TF apply will create the number of VMs, but TF destroy only delete one of the VM. Also no count.index like attribute can be used.

[cs-zhang]

After more testing and digging, the current anti-affinity support for PowerVS could not used for OCP installation.
Here are the three ways to define anti-affinity in PowerVS:

1. pi_affinity_policy and pi_anti_affinity_instances -- it requires other pvm IDs, that is impossible to get or need to create the pvms and at a time, even in this case pvms could be be placed on same system in my test.
2. pi_placement_group_id -- it requires to create pvms one by one, the VM has to be in active status to be able to add in.
3. pi_replication_policy, pi_replicants -- it only create the VMs with the same parameters, also found the TF destroy would just remove one of the VMs, not all VMs created by replicate.

Also the TF's meta-function count is not supported by all above.