Multispeaker

[qwertyflagstop]

Hey I was curios if you have tried any methods for making multi-speaker VITS models with your encoder. Normal VITS seems to have a multi-speaker capability with this extra embedding layer for encoding speaker ID and providing that to various downstream parts (all the parts that take g)

  if self.n_speakers > 0:
    g = self.emb_g(sid).unsqueeze(-1) # [b, h, 1]
  else:
    g = None

  z, m_q, logs_q, y_mask = self.enc_q(y, y_lengths, g=g)
  z_p = self.flow(z, y_mask, g=g)

  with torch.no_grad():
    # negative cross-entropy
    s_p_sq_r = torch.exp(-2 * logs_p) # [b, d, t]
    neg_cent1 = torch.sum(-0.5 * math.log(2 * math.pi) - logs_p, [1], keepdim=True) # [b, 1, t_s]
    neg_cent2 = torch.matmul(-0.5 * (z_p ** 2).transpose(1, 2), s_p_sq_r) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
    neg_cent3 = torch.matmul(z_p.transpose(1, 2), (m_p * s_p_sq_r)) # [b, t_t, d] x [b, d, t_s] = [b, t_t, t_s]
    neg_cent4 = torch.sum(-0.5 * (m_p ** 2) * s_p_sq_r, [1], keepdim=True) # [b, 1, t_s]
    neg_cent = neg_cent1 + neg_cent2 + neg_cent3 + neg_cent4

    attn_mask = torch.unsqueeze(x_mask, 2) * torch.unsqueeze(y_mask, -1)
    attn = monotonic_align.maximum_path(neg_cent, attn_mask.squeeze(1)).unsqueeze(1).detach()

  w = attn.sum(2)
  if self.use_sdp:
    l_length = self.dp(x, x_mask, w, g=g)
    l_length = l_length / torch.sum(x_mask)
  else:
    logw_ = torch.log(w + 1e-6) * x_mask
    logw = self.dp(x, x_mask, g=g)
    l_length = torch.sum((logw - logw_)**2, [1,2]) / torch.sum(x_mask) # for averaging 

  # expand prior
  m_p = torch.matmul(attn.squeeze(1), m_p.transpose(1, 2)).transpose(1, 2)
  logs_p = torch.matmul(attn.squeeze(1), logs_p.transpose(1, 2)).transpose(1, 2)

  z_slice, ids_slice = commons.rand_slice_segments(z, y_lengths, self.segment_size)
  o = self.dec(z_slice, g=g)
  return o, l_length, attn, ids_slice, x_mask, y_mask, (z, z_p, m_p, logs_p, m_q, logs_q)

would using your XPhoneBert encoder have much of an effect on this?

[qwertyflagstop]

Update I tried this approach with Libritts multispeaker dataset (901 speakers) and it did NOT produce good comprehensible. Let me know if you want to know more about the experiment. Here is a sample after 500k steps. Maybe this is what it usually sounds like around 500k? (batch size 64)

(github doesnt suppoer wav files so i zipped them)
Samples.zip

[thelinhbkhn2014]

Sorry, I do not have enough time to test the multispeaker experiments. But I believe that it could still works well. You can double check with the experiment from this pull pull request.