A history of samples of my singing voice synthesis system.

Code: https://github.com/r9y9/nnsvs

NNSVS: DNN-based singing voice synthesis samples

NOTE: I picked 007 for demonstrating vibrato modeling capability. This sample was included in training data so it may not work well for real test data

2022/03/06 vibrato test1 svs-world-conv Nitech Jp Song070 F001 007 Baseline

2022/03/06 vibrato test1 svs-world-conv Nitech Jp Song070 F001 007 with Vibrato modeling

https://github.com/r9y9/nnsvs/pull/73

This sample is not included in the training data.

2022/03/08 NIT-SONG070 svs-world-conv sine-based vibrato modeling Nitech Jp Song070 F001 003

https://github.com/r9y9/nnsvs/pull/73 + https://github.com/r9y9/nnsvs/issues/76

This sample is not included in the training data.

2022/03/09 NIT-SONG070 sine-based vibrato modeling + skip connection Nitech Jp Song070 F001 003

https://github.com/r9y9/nnsvs/issues/76

Implemented pitch regularization loss. It seems that the pitch gets very stable.

Same model architecture as https://soundcloud.com/r9y9/20220309-nit-song070-svs-world-conv-sine-based-vibrato-modeling-skip-connection-nitech-jp-song070-f001-003 but with an additional pitch regularization loss.

2022/03/11 NIT-SONG070 pitch regularization loss Nitech Jp Song070 F001 003

https://github.com/r9y9/nnsvs/pull/80

The best sample so far:
- Clear vibrato
- Natural durations
- Stable and dynamic pitch

This sample is not included in the training data.

2022/03/12 NIT-SONG070 pitch regularization + MDN duration generation Nitech Jp Song070 F001 003

nnsvs: https://github.com/r9y9/nnsvs/pull/81

ustは https://www.nicovideo.jp/watch/sm39319737 を使わせていただきました。ust -> musicxml に変換して、nnsvsで合成しました。

モデル情報: 
Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 344,
    "out_dim": 1,
    "hidden_dim": 32,
    "num_layers": 3,
    "dropout": 0.5,
    "num_gaussians": 1
}
Duration model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 344,
    "out_dim": 1,
    "hidden_dim": 256,
    "num_layers": 3,
    "dropout": 0.5,
    "num_gaussians": 1
}
Acoustic model: {
    "_target_": "nnsvs.model.ResF0Conv1dResnet",
    "in_dim": 348,
    "out_dim": 206,
    "hidden_dim": 128,
    "num_layers": 6,
    "in_lf0_idx": 300,
    "out_lf0_idx": 180,
    "in_lf0_min": 5.162578582763672,
    "in_lf0_max": 6.491110801696777,
    "out_lf0_mean": 5.892984643557185,
    "out_lf0_scale": 0.2296959673213171
}

2022/03/21 Kiritan test SVS Roshin Yuukai

nnsvs: github.com/r9y9/nnsvs/pull/81

有声/無声誤りを修正

ustは www.nicovideo.jp/watch/sm39319737 を使わせていただきました。ust -> musicxml に変換して、nnsvsで合成しました。

モデル情報:
Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
"_target_": "nnsvs.model.MDN",
"in_dim": 344,
"out_dim": 1,
"hidden_dim": 32,
"num_layers": 3,
"dropout": 0.5,
"num_gaussians": 1
}
Duration model: {
"_target_": "nnsvs.model.MDN",
"in_dim": 344,
"out_dim": 1,
"hidden_dim": 256,
"num_layers": 3,
"dropout": 0.5,
"num_gaussians": 1
}
Acoustic model: {
"_target_": "nnsvs.model.ResF0Conv1dResnet",
"in_dim": 348,
"out_dim": 206,
"hidden_dim": 128,
"num_layers": 6,
"in_lf0_idx": 300,
"out_lf0_idx": 180,
"in_lf0_min": 5.162578582763672,
"in_lf0_max": 6.491110801696777,
"out_lf0_mean": 5.892984643557185,
"out_lf0_scale": 0.2296959673213171
}

2022/03/21 Kiritan test SVS Roshin Yuukai V/UV fixed

2022/03/23 Kiritan test SVS Hikaruno Go

https://github.com/r9y9/nnsvs

Test on a Neural source filter (NSF) model
NOTE: Training is still in progress (80k steps out of 400k)
NOTE: Haven't done any optimization yet

ust: www.nicovideo.jp/watch/sm39319737

Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 32,
    "num_layers": 3,
    "dropout": 0.5,
    "num_gaussians": 1
}
Duration model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 256,
    "num_layers": 3,
    "dropout": 0.5,
    "num_gaussians": 1
}
Acoustic model: {
    "_target_": "nnsvs.model.ResSkipF0FFConvLSTM",
    "in_dim": 335,
    "out_dim": 206,
    "ff_hidden_dim": 1024,
    "conv_hidden_dim": 512,
    "lstm_hidden_dim": 256,
    "dropout": 0.0,
    "bidirectional": true,
    "in_lf0_idx": 286,
    "out_lf0_idx": 180,
    "in_lf0_min": 5.162578582763672,
    "in_lf0_max": 6.491110801696777,
    "out_lf0_mean": 5.885598413281433,
    "out_lf0_scale": 0.23067350503371858
}
Vocoder model: {
    "generator_type": "HnSincNSF",
    "generator_params": {
        "aux_context_window": 2,
        "cin_channels": 67,
        "cnn_kernel_s": 5,
        "cnn_num_in_block": 10,
        "filter_block_num": 5,
        "harmonic_num": 7,
        "hidden_dim": 64,
        "noise_std": 0.003,
        "out_channels": 1,
        "out_lf0_idx": 60,
        "out_lf0_mean": 5.885598413281433,
        "out_lf0_scale": 0.23067350503371858,
        "out_vuv_idx": 61,
        "sample_rate": 48000,
        "sinc_order": 31,
        "sine_amp": 0.1,
        "upsample_net": "ConvInUpsampleNetwork",
        "upsample_params": {
            "upsample_scales": [
                2,
                3,
                4,
                10
            ],
            "aux_channels": 67,
            "aux_context_window": 2
        },
        "upsample_rate": 240
    }
}

2022/04/05 Kiritan test SVS w/ NSF 80k steps Roshin Yuukai

github.com/r9y9/nnsvs

Test on a Neural source filter (NSF) model
NOTE: Training is still in progress (350k steps out of 400k)
NOTE: Haven't done any optimization yet

ust: www.nicovideo.jp/watch/sm39319737

Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
"_target_": "nnsvs.model.MDN",
"in_dim": 331,
"out_dim": 1,
"hidden_dim": 32,
"num_layers": 3,
"dropout": 0.5,
"num_gaussians": 1
}
Duration model: {
"_target_": "nnsvs.model.MDN",
"in_dim": 331,
"out_dim": 1,
"hidden_dim": 256,
"num_layers": 3,
"dropout": 0.5,
"num_gaussians": 1
}
Acoustic model: {
"_target_": "nnsvs.model.ResSkipF0FFConvLSTM",
"in_dim": 335,
"out_dim": 206,
"ff_hidden_dim": 1024,
"conv_hidden_dim": 512,
"lstm_hidden_dim": 256,
"dropout": 0.0,
"bidirectional": true,
"in_lf0_idx": 286,
"out_lf0_idx": 180,
"in_lf0_min": 5.162578582763672,
"in_lf0_max": 6.491110801696777,
"out_lf0_mean": 5.885598413281433,
"out_lf0_scale": 0.23067350503371858
}
Vocoder model: {
"generator_type": "HnSincNSF",
"generator_params": {
"aux_context_window": 2,
"cin_channels": 67,
"cnn_kernel_s": 5,
"cnn_num_in_block": 10,
"filter_block_num": 5,
"harmonic_num": 7,
"hidden_dim": 64,
"noise_std": 0.003,
"out_channels": 1,
"out_lf0_idx": 60,
"out_lf0_mean": 5.885598413281433,
"out_lf0_scale": 0.23067350503371858,
"out_vuv_idx": 61,
"sample_rate": 48000,
"sinc_order": 31,
"sine_amp": 0.1,
"upsample_net": "ConvInUpsampleNetwork",
"upsample_params": {
"upsample_scales": [
2,
3,
4,
10
],
"aux_channels": 67,
"aux_context_window": 2
},
"upsample_rate": 240
}
}

2022/04/06 Kiritan test SVS w/ NSF 350k steps Roshin Yuukai

w/ GAN-based leaned post-filter (experimental)

https://github.com/r9y9/nnsvs

Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 32,
    "num_layers": 3,
    "num_gaussians": 1
}
Duration model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 256,
    "num_layers": 3,
    "num_gaussians": 1
}
Acoustic model: {
    "_target_": "nnsvs.model.ResSkipF0FFConvLSTM",
    "in_dim": 335,
    "out_dim": 206,
    "ff_hidden_dim": 1024,
    "conv_hidden_dim": 512,
    "lstm_hidden_dim": 256,
    "dropout": 0.0,
    "bidirectional": true,
    "in_lf0_idx": 286,
    "out_lf0_idx": 180,
    "in_lf0_min": 5.162578582763672,
    "in_lf0_max": 6.491110801696777,
    "out_lf0_mean": 5.885598413281433,
    "out_lf0_scale": 0.23067350503371858,
    "init_type": "kaiming_normal"
}
Post-filter model: {
    "_target_": "nnsvs.postfilters.MultistreamPostFilter",
    "stream_sizes": [
        60,
        1,
        1,
        5,
        2,
        1
    ],
    "mgc_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 58,
        "channels": 64,
        "kernel_size": [
            5,
            5
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "mgc_offset": 2,
    "bap_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 5,
        "channels": 16,
        "kernel_size": [
            5,
            3
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "lf0_postfilter": null
}
Vocoder model: WORLD

2022/05/17 NNSVS Sample1 WORLD GAN post-filter

w/ Merlin's post-filter (default of the current nnsvs)

https://github.com/r9y9/nnsvs

Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 32,
    "num_layers": 3,
    "num_gaussians": 1
}
Duration model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 256,
    "num_layers": 3,
    "num_gaussians": 1
}
Acoustic model: {
    "_target_": "nnsvs.model.ResSkipF0FFConvLSTM",
    "in_dim": 335,
    "out_dim": 206,
    "ff_hidden_dim": 1024,
    "conv_hidden_dim": 512,
    "lstm_hidden_dim": 256,
    "dropout": 0.0,
    "bidirectional": true,
    "in_lf0_idx": 286,
    "out_lf0_idx": 180,
    "in_lf0_min": 5.162578582763672,
    "in_lf0_max": 6.491110801696777,
    "out_lf0_mean": 5.885598413281433,
    "out_lf0_scale": 0.23067350503371858,
    "init_type": "kaiming_normal"
}
Post-filter model: {
    "_target_": "nnsvs.postfilters.MultistreamPostFilter",
    "stream_sizes": [
        60,
        1,
        1,
        5,
        2,
        1
    ],
    "mgc_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 58,
        "channels": 64,
        "kernel_size": [
            5,
            5
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "mgc_offset": 2,
    "bap_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 5,
        "channels": 16,
        "kernel_size": [
            5,
            3
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "lf0_postfilter": null
}
Vocoder model: WORLD

2022/05/17 NNSVS Sample1 WORLD Merlin's post-filter

w/o post-filter

https://github.com/r9y9/nnsvs

Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 32,
    "num_layers": 3,
    "num_gaussians": 1
}
Duration model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 256,
    "num_layers": 3,
    "num_gaussians": 1
}
Acoustic model: {
    "_target_": "nnsvs.model.ResSkipF0FFConvLSTM",
    "in_dim": 335,
    "out_dim": 206,
    "ff_hidden_dim": 1024,
    "conv_hidden_dim": 512,
    "lstm_hidden_dim": 256,
    "dropout": 0.0,
    "bidirectional": true,
    "in_lf0_idx": 286,
    "out_lf0_idx": 180,
    "in_lf0_min": 5.162578582763672,
    "in_lf0_max": 6.491110801696777,
    "out_lf0_mean": 5.885598413281433,
    "out_lf0_scale": 0.23067350503371858,
    "init_type": "kaiming_normal"
}
Post-filter model: {
    "_target_": "nnsvs.postfilters.MultistreamPostFilter",
    "stream_sizes": [
        60,
        1,
        1,
        5,
        2,
        1
    ],
    "mgc_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 58,
        "channels": 64,
        "kernel_size": [
            5,
            5
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "mgc_offset": 2,
    "bap_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 5,
        "channels": 16,
        "kernel_size": [
            5,
            3
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "lf0_postfilter": null
}
Vocoder model: WORLD

2022/05/17 NNSVS Sample1 WORLD no post-filter

「波音リツ」歌声データベースVer2.zip を利用しました
https://www.youtube.com/watch?v=K0NuZK0IiGA

信号処理ベースのPost-filterを利用したバージョンです。これまでのnnsvsでデフォルトで使われているものと同じです。

モデルの詳細:
Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 337,
    "out_dim": 1,
    "hidden_dim": 32,
    "num_layers": 3,
    "num_gaussians": 1
}
Duration model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 337,
    "out_dim": 1,
    "hidden_dim": 256,
    "num_layers": 3,
    "num_gaussians": 1
}
Acoustic model: {
    "_target_": "nnsvs.model.ResSkipF0FFConvLSTM",
    "in_dim": 341,
    "out_dim": 206,
    "ff_hidden_dim": 1024,
    "conv_hidden_dim": 512,
    "lstm_hidden_dim": 256,
    "dropout": 0.0,
    "num_lstm_layers": 2,
    "bidirectional": true,
    "init_type": "kaiming_normal",
    "use_mdn": false,
    "in_lf0_idx": 292,
    "out_lf0_idx": 180,
    "in_lf0_min": 4.989291667938232,
    "in_lf0_max": 6.895446300506592,
    "out_lf0_mean": 5.922958879360765,
    "out_lf0_scale": 0.2911988159887554
}
Post-filter model: {
    "_target_": "nnsvs.postfilters.MultistreamPostFilter",
    "stream_sizes": [
        60,
        1,
        1,
        5,
        2,
        1
    ],
    "mgc_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 58,
        "channels": 64,
        "kernel_size": [
            5,
            5
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "mgc_offset": 2,
    "bap_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 5,
        "channels": 32,
        "kernel_size": [
            5,
            1
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "lf0_postfilter": null
}
Vocoder model: WORLD

2022/05/21  Ritsu Sample1 Merlin-Postfilter

「波音リツ」歌声データベースVer2.zip を利用しました
https://www.youtube.com/watch?v=K0NuZK0IiGA

GAN-based post-filter ありバージョンです。過剰平滑化問題が緩和されて、クリアになっていま（一方、現状、若干ノイズっぽくなってしまいます。気になる？

モデルの詳細:
Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 337,
    "out_dim": 1,
    "hidden_dim": 32,
    "num_layers": 3,
    "num_gaussians": 1
}
Duration model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 337,
    "out_dim": 1,
    "hidden_dim": 256,
    "num_layers": 3,
    "num_gaussians": 1
}
Acoustic model: {
    "_target_": "nnsvs.model.ResSkipF0FFConvLSTM",
    "in_dim": 341,
    "out_dim": 206,
    "ff_hidden_dim": 1024,
    "conv_hidden_dim": 512,
    "lstm_hidden_dim": 256,
    "dropout": 0.0,
    "num_lstm_layers": 2,
    "bidirectional": true,
    "init_type": "kaiming_normal",
    "use_mdn": false,
    "in_lf0_idx": 292,
    "out_lf0_idx": 180,
    "in_lf0_min": 4.989291667938232,
    "in_lf0_max": 6.895446300506592,
    "out_lf0_mean": 5.922958879360765,
    "out_lf0_scale": 0.2911988159887554
}
Post-filter model: {
    "_target_": "nnsvs.postfilters.MultistreamPostFilter",
    "stream_sizes": [
        60,
        1,
        1,
        5,
        2,
        1
    ],
    "mgc_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 58,
        "channels": 64,
        "kernel_size": [
            5,
            5
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "mgc_offset": 2,
    "bap_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 5,
        "channels": 32,
        "kernel_size": [
            5,
            1
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "lf0_postfilter": null
}
Vocoder model: WORLD

2022/05/21  Ritsu Sample1 GAN-Postfilter

https://github.com/r9y9/nnsvs

vocoder: WORLD

ust: www.nicovideo.jp/watch/sm39319737

Statistical parametric SVS (sampling rate: 48000)
Time-lag model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 32,
    "num_layers": 3,
    "num_gaussians": 1
}
Duration model: {
    "_target_": "nnsvs.model.MDN",
    "in_dim": 331,
    "out_dim": 1,
    "hidden_dim": 256,
    "num_layers": 3,
    "num_gaussians": 1
}
Acoustic model: {
    "_target_": "nnsvs.model.ResSkipF0FFConvLSTM",
    "in_dim": 335,
    "out_dim": 206,
    "ff_hidden_dim": 1024,
    "conv_hidden_dim": 512,
    "lstm_hidden_dim": 256,
    "dropout": 0.0,
    "bidirectional": true,
    "in_lf0_idx": 286,
    "out_lf0_idx": 180,
    "in_lf0_min": 5.162578582763672,
    "in_lf0_max": 6.491110801696777,
    "out_lf0_mean": 5.885598413281433,
    "out_lf0_scale": 0.23067350503371858,
    "init_type": "kaiming_normal"
}
Post-filter model: {
    "_target_": "nnsvs.postfilters.MultistreamPostFilter",
    "stream_sizes": [
        60,
        1,
        1,
        5,
        2,
        1
    ],
    "mgc_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 58,
        "channels": 64,
        "kernel_size": [
            5,
            5
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "mgc_offset": 2,
    "bap_postfilter": {
        "_target_": "nnsvs.postfilters.Conv2dPostFilter",
        "in_dim": 5,
        "channels": 32,
        "kernel_size": [
            5,
            3
        ],
        "use_tadn": false,
        "init_type": "kaiming_normal",
        "scale": 1.0
    },
    "lf0_postfilter": null
}
Vocoder model: {
    "generator_type": "HnSincNSF",
    "generator_params": {
        "aux_context_window": 0,
        "cin_channels": 67,
        "cnn_kernel_s": 5,
        "cnn_num_in_block": 10,
        "filter_block_num": 5,
        "harmonic_num": 7,
        "hidden_dim": 64,
        "noise_std": 0.003,
        "out_channels": 1,
        "out_lf0_idx": 60,
        "out_lf0_mean": 5.885598413281433,
        "out_lf0_scale": 0.23067350503371858,
        "out_vuv_idx": 61,
        "sample_rate": 48000,
        "sinc_order": 31,
        "sine_amp": 0.1,
        "upsample_net": null,
        "upsample_params": {
            "upsample_scales": [
                2,
                3,
                4,
                10
            ],
            "aux_channels": 67,
            "aux_context_window": 0
        },
        "upsample_rate": 240
    }
}

2022/05/22 Kiritan test SVS Roshin Yuukai Full GAN-postfilter

本家: https://www.nicovideo.jp/watch/sm6119955
UST: https://www.youtube.com/watch?v=szFcxJwd5i0
Inst音源: https://www.youtube.com/watch?v=rxb7yCHOw7g

nnsvsのテストです https://github.com/r9y9/nnsvs
調声一切無

転載などはお控えください。音源、USTなど、研究用途として使用しています

2022/05/30 Kiritan Ruka Test

Credit:
Database:「波音リツ」歌声データベースVer2.zip
www.youtube.com/watch?v=K0NuZK0IiGA
UST: https://bowlroll.net/file/193938
原曲: https://www.nicovideo.jp/watch/sm31413058

てるみい、の最後のサビの部分だけ。てるみい最高

開発中のモデルです。以前よりもビブラートがきれいに出ているはず？
無調声、NNSVS の raw output です。IA＆OИE の2パート、どちらも波音リツです

NNSVS: https://nnsvs.github.io/

2022/08/02 NNSVS Ritsu Test Tellme

This demo shows the strong capability of the autoregressive F0 model. 

NOTE:
- Namine Ritsu's database [1] is used to train a backbone model.
- NIT-song 070 database is used for fine-tuning
- WORLD vocoder is used instead of a neural vocoder (to save my time to train a vocoder)
- Sampling rate: 48kHz

[1] Canon, [NamineRitsu] Blue (YOASOBI) [ENUNU model Ver.2, Singing DBVer.2 release], https://www.youtube.com/watch?v=pKeo9IE_L1I, Accessed: 2022.10.06.