Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add individual rate adjustment per substream for Opus #11

Merged
merged 5 commits into from
Aug 16, 2024
Merged

Add individual rate adjustment per substream for Opus #11

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8734536
Add individual rate adjustment per substram for Opus
thirv Aug 8, 2024
a047a41
Add rate adjustment per substream
thirv Aug 9, 2024
908d161
Review suggestion: switch to using a map for bitrate override.
thirv Aug 15, 2024
44e837a
Nit: Use `contains` when checking presence in a map.
jwcullen Aug 16, 2024
0461b2c
Document behavior in `codec_config.proto`.
jwcullen Aug 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 12 additions & 9 deletions iamf/cli/codec/opus_encoder.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -153,18 +153,21 @@ absl::Status OpusEncoder::InitializeEncoder() {
// `OPUS_SET_BITRATE` treats this as the bit-rate for the entire substream.
// Configure `libopus` so coupled substreams and mono substreams have equally
// effective bit-rate per channel.
float opus_rate;
if (num_channels_ > 1) {
opus_encoder_ctl(
encoder_,
OPUS_SET_BITRATE(static_cast<opus_int32>(
encoder_metadata_.target_bitrate_per_channel() * num_channels_ *
encoder_metadata_.coupling_rate_adjustment() +
0.5f)));
opus_rate =
encoder_metadata_.use_substream_rate_override()
? encoder_metadata_.substream_rate_overrides()[substream_id_]
Copy link
Collaborator

@jwcullen jwcullen Aug 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This introduces an error if encoder_metadata_.substream_rate_overrides()[substream_id_] goes out of bounds.

Before performing the lookup validate that it does not go out of bounds. At review time check against the size of the array. But if you take the "map"-based suggestion in another comment check that the substream_id is present.

I think if it would go out of bounds it is OK to fallback to the calculation based on target_bitrate_per_channels.

Same in the else.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the help, was considering it but was not sure how maps worked here.

: encoder_metadata_.target_bitrate_per_channel() * num_channels_ *
encoder_metadata_.coupling_rate_adjustment();
} else {
opus_encoder_ctl(encoder_,
OPUS_SET_BITRATE(static_cast<opus_int32>(
encoder_metadata_.target_bitrate_per_channel())));
opus_rate =
encoder_metadata_.use_substream_rate_override()
? encoder_metadata_.substream_rate_overrides()[substream_id_]
: encoder_metadata_.target_bitrate_per_channel();
}
opus_encoder_ctl(encoder_,
OPUS_SET_BITRATE(static_cast<opus_int32>(opus_rate + 0.5f)));

return absl::OkStatus();
}
Expand Down
6 changes: 4 additions & 2 deletions iamf/cli/codec/opus_encoder.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,12 @@ class OpusEncoder : public EncoderBase {
public:
OpusEncoder(
const iamf_tools_cli_proto::OpusEncoderMetadata& opus_encoder_metadata,
const CodecConfigObu& codec_config, int num_channels)
const CodecConfigObu& codec_config, int num_channels, int substream_id)
: EncoderBase(false, codec_config, num_channels),
encoder_metadata_(opus_encoder_metadata),
decoder_config_(std::get<OpusDecoderConfig>(
codec_config.GetCodecConfig().decoder_config)) {}
codec_config.GetCodecConfig().decoder_config)),
substream_id_(substream_id) {}

~OpusEncoder() override;

Expand Down Expand Up @@ -84,6 +85,7 @@ class OpusEncoder : public EncoderBase {

const iamf_tools_cli_proto::OpusEncoderMetadata encoder_metadata_;
const OpusDecoderConfig decoder_config_;
const int substream_id_;

LibOpusEncoder* encoder_ = nullptr;
};
Expand Down
1 change: 1 addition & 0 deletions iamf/cli/codec/tests/encoder_test_base.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ class EncoderTestBase {
}

int num_channels_ = 1;
int substream_id_ = 0;
uint32_t num_samples_per_frame_ = 1;
int8_t input_sample_size_ = 16;
std::unique_ptr<EncoderBase> encoder_;
Expand Down
2 changes: 1 addition & 1 deletion iamf/cli/codec/tests/opus_encoder_test.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ class OpusEncoderTest : public EncoderTestBase, public testing::Test {
EXPECT_THAT(codec_config.Initialize(), IsOk());

encoder_ = std::make_unique<OpusEncoder>(opus_encoder_metadata_,
codec_config, num_channels_);
codec_config, num_channels_, substream_id_);
}

OpusDecoderConfig opus_decoder_config_ = {
Expand Down
3 changes: 3 additions & 0 deletions iamf/cli/proto/codec_config.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ message OpusEncoderMetadata {
optional OpusApplicationFlag application = 2;
Copy link
Collaborator

@jwcullen jwcullen Aug 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you imagine any use cases where application should vary per substream (or per audio element)?

I'm thinking of a use case where one audio element is music (best encoded as APPLICATION_AUDIO) and a second audio element is voice chat (best encoded as APPLICATION_VOIP).

If you think this would be useful, we may want to consider how we would handle this - even if it does not get implemented in this PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally don't see this as a critical addition right now; I'd think separate audio elements would be used in that case. But yeah, the need may materialize suddenly at some point.

optional bool use_float_api = 3 [default = true];
optional float coupling_rate_adjustment = 4 [default = 1.0];
optional bool use_substream_rate_override = 5 [default = false];
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the proto may be better as

map<uint32, int32> substream_id_to_bitrate_override = 5;

In textproto it would look like.

# Substream ID 123 has a bitrate of 1000.
substream_id_to_bitrate_override {
key: 123
value: 1000 
}

This would be more reasonable if large substream IDs are used - which are generally supported elsewhere in the encoder.

I think in this case it makes sense to check presence of the substream in the map, and there does not need to be a use_substream_rate_override. When the substream is not present that implies it is not being overrided - using the "default" calculation would be reasonable.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the help, was considering it but was not sure how maps worked here.

// so far the below uses substream_id as array index
repeated int32 substream_rate_overrides = 6 [packed = true];
}

message OpusDecoderConfig {
Expand Down
10 changes: 5 additions & 5 deletions iamf/cli/proto_to_obu/audio_frame_generator.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ namespace {
absl::Status InitializeEncoder(
const iamf_tools_cli_proto::CodecConfig& codec_config_metadata,
const CodecConfigObu& codec_config, int num_channels,
std::unique_ptr<EncoderBase>& encoder) {
std::unique_ptr<EncoderBase>& encoder, int substream_id = NULL) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
std::unique_ptr<EncoderBase>& encoder, int substream_id = NULL) {
std::unique_ptr<EncoderBase>& encoder, int substream_id = 0) {

switch (codec_config.GetCodecConfig().codec_id) {
using enum CodecConfig::CodecId;
case kCodecIdLpcm:
Expand All @@ -68,7 +68,7 @@ absl::Status InitializeEncoder(
case kCodecIdOpus:
encoder = std::make_unique<OpusEncoder>(
codec_config_metadata.decoder_config_opus().opus_encoder_metadata(),
codec_config, num_channels);
codec_config, num_channels, substream_id);
break;
case kCodecIdAacLc:
encoder = std::make_unique<AacEncoder>(
Expand Down Expand Up @@ -109,9 +109,9 @@ absl::Status GetEncodingDataAndInitializeEncoders(
codec_config_obu.GetCodecConfigId()));
}

RETURN_IF_NOT_OK(InitializeEncoder(codec_config_metadata_iter->second,
codec_config_obu, num_channels,
substream_id_to_encoder[substream_id]));
RETURN_IF_NOT_OK(InitializeEncoder(
codec_config_metadata_iter->second, codec_config_obu, num_channels,
substream_id_to_encoder[substream_id], substream_id));
}

return absl::OkStatus();
Expand Down