Fix Messaging Pattern Performance and throughput by reducing contention and serialization points

[brianoliver]

Fix Messaging Pattern Performance and throughput by reducing contention and serialization points

[brianoliver]

Reported by narliss

[brianoliver]

@therice said:
Based upon observations in two node setup (one storage and one client) and a much larger setup (300+ storage/client nodes), this affects 11.x as well.

This UG thread indicates it has been an issue for some time => https://forums.oracle.com/thread/2180345

Any plans for a resolution that would be back-ported and available soon? Under high volumes of messages, it makes the pattern unusable as performance degrades to a point where requests time out.

I collected some timings via a test case using a auto-commit Queue for only the getMessage() method. The numbers in each line's prefix is the QUEUE_SIZE_BEFORE => QUEUE_SIZE_AFTER, with the statistics being across all invocations of getMessage() for that range. As the abstract indicates, it appears this is due to serialization of the Queue object when there are large numbers of messages being tracked for delivery.

Receiving [ 50000 ] messages from Queue [ CoherenceProviderSmokeTestQueue ]
50000 => 49000 : Snapshot

{size=1000, median=78.0, max=198.0, min=70.0, mean=88.11800000000001, stddev=23.363517071064702}

49000 => 48000 : Snapshot

{size=1000, median=76.0, max=202.0, min=51.0, mean=85.803, stddev=22.848155926228024}

48000 => 47000 : Snapshot

{size=1000, median=74.0, max=189.0, min=68.0, mean=83.49, stddev=22.176512297872684}

47000 => 46000 : Snapshot

{size=1000, median=73.0, max=186.0, min=67.0, mean=81.93100000000001, stddev=22.623174999257877}

46000 => 45000 : Snapshot

{size=1000, median=71.0, max=200.0, min=64.0, mean=80.55199999999999, stddev=22.414720960023395}

45000 => 44000 : Snapshot

{size=1000, median=70.0, max=196.0, min=62.0, mean=79.555, stddev=22.381597595143056}

44000 => 43000 : Snapshot

{size=1000, median=68.0, max=184.0, min=57.0, mean=76.64, stddev=23.42330145498386}

43000 => 42000 : Snapshot

{size=1000, median=61.0, max=172.0, min=51.0, mean=70.20599999999999, stddev=21.80112044948278}

42000 => 41000 : Snapshot

{size=1000, median=60.0, max=210.0, min=45.0, mean=67.977, stddev=21.274752315505037}

41000 => 40000 : Snapshot

{size=1000, median=59.0, max=244.0, min=50.0, mean=67.26599999999999, stddev=21.843130873717897}

40000 => 39000 : Snapshot

{size=1000, median=57.0, max=188.0, min=48.0, mean=64.43599999999999, stddev=19.975057118989817}

39000 => 38000 : Snapshot

{size=1000, median=55.0, max=175.0, min=46.0, mean=62.61600000000001, stddev=19.64389030783945}

38000 => 37000 : Snapshot

{size=1000, median=54.0, max=173.0, min=47.0, mean=60.36199999999999, stddev=18.25717065375361}

37000 => 36000 : Snapshot

{size=1000, median=53.0, max=160.0, min=46.0, mean=58.911, stddev=18.04396474347216}

36000 => 35000 : Snapshot

{size=1000, median=51.0, max=175.0, min=47.0, mean=57.267, stddev=17.92537391604513}

35000 => 34000 : Snapshot

{size=1000, median=50.0, max=160.0, min=43.0, mean=55.09499999999999, stddev=16.485683266178256}

34000 => 33000 : Snapshot

{size=1000, median=48.0, max=161.0, min=37.0, mean=54.340999999999994, stddev=18.304036624656263}

33000 => 32000 : Snapshot

{size=1000, median=47.0, max=139.0, min=41.0, mean=51.53799999999999, stddev=14.555494440877554}

32000 => 31000 : Snapshot

{size=1000, median=45.0, max=151.0, min=33.0, mean=50.489999999999995, stddev=15.691211365414489}

31000 => 30000 : Snapshot

{size=1000, median=44.0, max=150.0, min=37.0, mean=48.375, stddev=14.491804145357037}

30000 => 29000 : Snapshot

{size=1000, median=43.0, max=157.0, min=38.0, mean=46.743, stddev=13.60419145061647}

29000 => 28000 : Snapshot

{size=1000, median=41.0, max=140.0, min=31.0, mean=45.505, stddev=14.777020516170905}

28000 => 27000 : Snapshot

{size=1000, median=40.0, max=134.0, min=33.0, mean=43.60999999999999, stddev=13.331452570055136}

27000 => 26000 : Snapshot

{size=1000, median=38.0, max=135.0, min=32.0, mean=41.697, stddev=12.239076190055702}

26000 => 25000 : Snapshot

{size=1000, median=40.0, max=137.0, min=32.0, mean=43.365, stddev=11.703022504282185}

25000 => 24000 : Snapshot

{size=1000, median=39.0, max=123.0, min=33.0, mean=41.849000000000004, stddev=11.014877508466745}

24000 => 23000 : Snapshot

{size=1000, median=37.0, max=113.0, min=26.0, mean=40.132999999999996, stddev=10.882266985087918}

23000 => 22000 : Snapshot

{size=1000, median=36.0, max=152.0, min=29.0, mean=38.351, stddev=11.041272965954054}

22000 => 21000 : Snapshot

{size=1000, median=34.0, max=114.0, min=30.0, mean=36.592000000000006, stddev=9.63038318449895}

21000 => 20000 : Snapshot

{size=1000, median=33.0, max=116.0, min=22.0, mean=34.745999999999995, stddev=9.500934256785056}

20000 => 19000 : Snapshot

{size=1000, median=31.0, max=116.0, min=25.0, mean=32.769999999999996, stddev=9.081937105132441}

19000 => 18000 : Snapshot

{size=1000, median=27.0, max=111.0, min=22.0, mean=28.898000000000003, stddev=8.157336461010907}

18000 => 17000 : Snapshot

{size=1000, median=26.0, max=116.0, min=12.0, mean=27.700999999999997, stddev=9.66918259689471}

17000 => 16000 : Snapshot

{size=1000, median=24.0, max=95.0, min=20.0, mean=25.713000000000005, stddev=7.386283787412215}

16000 => 15000 : Snapshot

{size=1000, median=23.0, max=85.0, min=12.0, mean=24.453999999999994, stddev=7.6051115522207615}

15000 => 14000 : Snapshot

{size=1000, median=23.0, max=107.0, min=17.0, mean=24.328999999999997, stddev=7.220588334470801}

14000 => 13000 : Snapshot

{size=1000, median=22.0, max=96.0, min=11.0, mean=23.030999999999995, stddev=6.432061132359573}

13000 => 12000 : Snapshot

{size=1000, median=20.0, max=90.0, min=15.0, mean=21.331000000000007, stddev=5.593453827978393}

12000 => 11000 : Snapshot

{size=1000, median=19.0, max=93.0, min=10.0, mean=19.792, stddev=6.111144089455122}

11000 => 10000 : Snapshot

{size=1000, median=17.0, max=97.0, min=9.0, mean=17.995999999999995, stddev=5.046131929255055}

10000 => 9000 : Snapshot

{size=1000, median=16.0, max=69.0, min=10.0, mean=16.378000000000007, stddev=4.5254387185769644}

9000 => 8000 : Snapshot

{size=1000, median=14.0, max=87.0, min=8.0, mean=14.708999999999996, stddev=4.505620891688481}

8000 => 7000 : Snapshot

{size=1000, median=12.0, max=69.0, min=6.0, mean=12.720000000000002, stddev=4.374097933601033}

7000 => 6000 : Snapshot

{size=1000, median=10.0, max=38.0, min=6.0, mean=10.879000000000003, stddev=2.827782763112343}

6000 => 5000 : Snapshot

{size=1000, median=9.0, max=50.0, min=5.0, mean=9.451000000000006, stddev=3.08790125070965}

5000 => 4000 : Snapshot

{size=1000, median=8.0, max=45.0, min=4.0, mean=8.082000000000006, stddev=2.878812874009606}

4000 => 3000 : Snapshot

{size=1000, median=6.0, max=43.0, min=3.0, mean=6.316999999999995, stddev=2.42825192420545}

3000 => 2000 : Snapshot

{size=1000, median=5.0, max=21.0, min=2.0, mean=4.685999999999994, stddev=1.4606633895383003}

2000 => 1000 : Snapshot

{size=1000, median=3.0, max=21.0, min=1.0, mean=3.0570000000000066, stddev=1.4070290440252893}

1000 => 0 : Snapshot

{size=1000, median=1.0, max=16.0, min=1.0, mean=1.5549999999999953, stddev=0.9380057861521299}

Receiving [ 100000 ] messages from Queue [ CoherenceProviderSmokeTestQueue ]
100000 => 99000 : Snapshot

{size=1000, median=181.0, max=298.0, min=84.0, mean=191.67199999999997, stddev=39.7766086361645}

99000 => 98000 : Snapshot

{size=1000, median=162.0, max=290.0, min=120.0, mean=176.967, stddev=38.44145752513368}

98000 => 97000 : Snapshot

{size=1000, median=164.0, max=317.0, min=133.0, mean=177.844, stddev=39.89788848429831}

97000 => 96000 : Snapshot

{size=1000, median=164.0, max=272.0, min=129.0, mean=173.26799999999997, stddev=35.08080442120503}

96000 => 95000 : Snapshot

{size=1000, median=156.0, max=281.0, min=130.0, mean=169.432, stddev=34.58857825562862}

95000 => 94000 : Snapshot

{size=1000, median=158.0, max=317.0, min=125.0, mean=169.86399999999998, stddev=35.94606548182457}

94000 => 93000 : Snapshot

{size=1000, median=154.5, max=299.0, min=127.0, mean=167.779, stddev=36.31168816920659}

93000 => 92000 : Snapshot

{size=1000, median=152.0, max=293.0, min=127.0, mean=166.075, stddev=36.24178309412818}

92000 => 91000 : Snapshot

{size=1000, median=147.0, max=269.0, min=123.0, mean=161.73799999999997, stddev=34.14051743794946}

91000 => 90000 : Snapshot

{size=1000, median=144.0, max=269.0, min=121.0, mean=159.653, stddev=34.58720667914461}

90000 => 89000 : Snapshot

{size=1000, median=139.0, max=256.0, min=121.0, mean=157.068, stddev=34.110568844669025}

89000 => 88000 : Snapshot

{size=1000, median=139.0, max=278.0, min=119.0, mean=155.454, stddev=33.97620146906161}

88000 => 87000 : Snapshot

{size=1000, median=133.0, max=271.0, min=117.0, mean=150.72400000000002, stddev=32.22465286472847}

87000 => 86000 : Snapshot

{size=1000, median=134.0, max=262.0, min=115.0, mean=149.856, stddev=32.593060175006165}

86000 => 85000 : Snapshot

{size=1000, median=131.5, max=262.0, min=115.0, mean=151.403, stddev=34.86278825632005}

85000 => 84000 : Snapshot

{size=1000, median=129.0, max=258.0, min=113.0, mean=147.725, stddev=33.08616129243573}

84000 => 83000 : Snapshot

{size=1000, median=125.0, max=243.0, min=113.0, mean=142.908, stddev=30.969834785911658}

83000 => 82000 : Snapshot

{size=1000, median=122.0, max=249.0, min=112.0, mean=141.87199999999999, stddev=32.449477416201674}

82000 => 81000 : Snapshot

{size=1000, median=121.0, max=256.0, min=108.0, mean=140.38799999999998, stddev=32.11256253668761}

81000 => 80000 : Snapshot

{size=1000, median=119.0, max=259.0, min=110.0, mean=139.402, stddev=32.33580059315062}

80000 => 79000 : Snapshot

{size=1000, median=118.0, max=295.0, min=107.0, mean=136.94, stddev=32.162568784704234}

79000 => 78000 : Snapshot

{size=1000, median=116.0, max=267.0, min=105.0, mean=134.702, stddev=31.069800909058472}

78000 => 77000 : Snapshot

{size=1000, median=114.0, max=253.0, min=99.0, mean=133.02100000000002, stddev=31.245300891939397}

77000 => 76000 : Snapshot

{size=1000, median=113.0, max=254.0, min=103.0, mean=131.137, stddev=30.7912378986512}

76000 => 75000 : Snapshot

{size=1000, median=111.0, max=255.0, min=98.0, mean=129.866, stddev=34.077842410330824}

75000 => 74000 : Snapshot

{size=1000, median=109.0, max=229.0, min=97.0, mean=127.812, stddev=33.67721500971853}

74000 => 73000 : Snapshot

{size=1000, median=107.0, max=241.0, min=99.0, mean=126.298, stddev=33.72124431746724}

73000 => 72000 : Snapshot

{size=1000, median=105.0, max=236.0, min=90.0, mean=123.848, stddev=33.02143714787884}

72000 => 71000 : Snapshot

{size=1000, median=103.5, max=241.0, min=97.0, mean=121.053, stddev=31.029744375937717}

71000 => 70000 : Snapshot

{size=1000, median=102.0, max=228.0, min=93.0, mean=118.013, stddev=28.028564562450065}

70000 => 69000 : Snapshot

{size=1000, median=101.0, max=224.0, min=94.0, mean=116.29400000000001, stddev=28.19614488873583}

69000 => 68000 : Snapshot

{size=1000, median=99.0, max=231.0, min=91.0, mean=114.71499999999999, stddev=28.588478030074917}

68000 => 67000 : Snapshot

{size=1000, median=97.0, max=245.0, min=90.0, mean=113.26, stddev=29.388093848701043}

67000 => 66000 : Snapshot

{size=1000, median=96.0, max=226.0, min=89.0, mean=111.236, stddev=28.876948976340678}

66000 => 65000 : Snapshot

{size=1000, median=94.0, max=229.0, min=86.0, mean=108.71400000000003, stddev=26.725464532061125}

65000 => 64000 : Snapshot

{size=1000, median=93.0, max=222.0, min=82.0, mean=107.33099999999999, stddev=27.175060679286776}

64000 => 63000 : Snapshot

{size=1000, median=91.0, max=218.0, min=83.0, mean=104.889, stddev=26.167373013170604}

63000 => 62000 : Snapshot

{size=1000, median=90.0, max=244.0, min=81.0, mean=103.94599999999997, stddev=28.009884641633747}

62000 => 61000 : Snapshot

{size=1000, median=88.0, max=219.0, min=82.0, mean=101.543, stddev=26.433896835349454}

61000 => 60000 : Snapshot

{size=1000, median=87.0, max=205.0, min=81.0, mean=98.898, stddev=23.679026421968675}

60000 => 59000 : Snapshot

{size=1000, median=85.0, max=201.0, min=71.0, mean=98.091, stddev=25.38284869765434}

59000 => 58000 : Snapshot

{size=1000, median=84.0, max=218.0, min=77.0, mean=96.375, stddev=25.448851035341832}

58000 => 57000 : Snapshot

{size=1000, median=82.0, max=208.0, min=78.0, mean=94.471, stddev=24.001317093489334}

57000 => 56000 : Snapshot

{size=1000, median=81.0, max=205.0, min=68.0, mean=92.66199999999999, stddev=24.265872920391494}

56000 => 55000 : Snapshot

{size=1000, median=80.0, max=200.0, min=70.0, mean=92.113, stddev=26.329669243696994}

55000 => 54000 : Snapshot

{size=1000, median=79.0, max=198.0, min=71.0, mean=90.142, stddev=24.058317452362346}

54000 => 53000 : Snapshot

{size=1000, median=78.0, max=197.0, min=64.0, mean=88.59200000000001, stddev=23.68168824975024}

53000 => 52000 : Snapshot

{size=1000, median=76.0, max=209.0, min=68.0, mean=86.655, stddev=24.187910824005215}

52000 => 51000 : Snapshot

{size=1000, median=75.0, max=184.0, min=69.0, mean=84.89500000000001, stddev=22.309363116922047}

51000 => 50000 : Snapshot

{size=1000, median=73.0, max=195.0, min=63.0, mean=84.346, stddev=24.85510128846911}

50000 => 49000 : Snapshot

{size=1000, median=71.0, max=189.0, min=56.0, mean=81.30400000000002, stddev=23.581638570669217}

49000 => 48000 : Snapshot

{size=1000, median=69.0, max=184.0, min=64.0, mean=79.03999999999999, stddev=22.99075846889305}

48000 => 47000 : Snapshot

{size=1000, median=68.0, max=173.0, min=58.0, mean=77.07, stddev=21.817862386565732}

47000 => 46000 : Snapshot

{size=1000, median=67.0, max=2874.0, min=58.0, mean=79.279, stddev=91.4995810500371}

46000 => 45000 : Snapshot

{size=1000, median=65.0, max=178.0, min=59.0, mean=74.04400000000001, stddev=21.79332496460817}

45000 => 44000 : Snapshot

{size=1000, median=64.0, max=168.0, min=57.0, mean=73.62100000000001, stddev=22.886790167346838}

44000 => 43000 : Snapshot

{size=1000, median=62.0, max=174.0, min=55.0, mean=71.28500000000001, stddev=21.670657405634707}

43000 => 42000 : Snapshot

{size=1000, median=61.0, max=170.0, min=50.0, mean=69.298, stddev=21.188727148136085}

42000 => 41000 : Snapshot

{size=1000, median=59.0, max=160.0, min=54.0, mean=66.64599999999999, stddev=19.53377209481695}

41000 => 40000 : Snapshot

{size=1000, median=58.0, max=175.0, min=49.0, mean=64.916, stddev=19.840377632106353}

40000 => 39000 : Snapshot

{size=1000, median=56.0, max=173.0, min=52.0, mean=62.87, stddev=18.07910268663667}

39000 => 38000 : Snapshot

{size=1000, median=55.0, max=157.0, min=30.0, mean=61.71900000000001, stddev=18.562343591249462}

38000 => 37000 : Snapshot

{size=1000, median=54.0, max=159.0, min=50.0, mean=60.268, stddev=18.492272443919397}

37000 => 36000 : Snapshot

{size=1000, median=52.0, max=164.0, min=47.0, mean=58.221999999999994, stddev=17.100267107128285}

36000 => 35000 : Snapshot

{size=1000, median=51.0, max=163.0, min=45.0, mean=57.07500000000001, stddev=17.403915400600876}

35000 => 34000 : Snapshot

{size=1000, median=50.0, max=154.0, min=45.0, mean=55.61600000000001, stddev=17.464694659786026}

34000 => 33000 : Snapshot

{size=1000, median=48.0, max=136.0, min=41.0, mean=53.239999999999995, stddev=14.814312804307315}

33000 => 32000 : Snapshot

{size=1000, median=51.0, max=154.0, min=42.0, mean=55.32800000000002, stddev=15.519254597539888}

32000 => 31000 : Snapshot

{size=1000, median=45.0, max=151.0, min=41.0, mean=49.642, stddev=13.566277303662948}

31000 => 30000 : Snapshot

{size=1000, median=44.0, max=150.0, min=36.0, mean=48.376, stddev=14.69927524378978}

30000 => 29000 : Snapshot

{size=1000, median=42.0, max=149.0, min=34.0, mean=46.94899999999999, stddev=15.123793443714058}

29000 => 28000 : Snapshot

{size=1000, median=41.0, max=148.0, min=35.0, mean=45.23100000000001, stddev=14.006920890503757}

28000 => 27000 : Snapshot

{size=1000, median=40.0, max=148.0, min=34.0, mean=43.496999999999986, stddev=13.796896221445806}

27000 => 26000 : Snapshot

{size=1000, median=38.0, max=134.0, min=32.0, mean=41.885000000000005, stddev=12.436335370093442}

26000 => 25000 : Snapshot

{size=1000, median=37.0, max=147.0, min=32.0, mean=40.63, stddev=13.711203342416656}

25000 => 24000 : Snapshot

{size=1000, median=39.0, max=133.0, min=34.0, mean=41.864999999999995, stddev=12.172137689655683}

24000 => 23000 : Snapshot

{size=1000, median=37.0, max=132.0, min=30.0, mean=40.669999999999995, stddev=13.210511633453864}

23000 => 22000 : Snapshot

{size=1000, median=36.0, max=124.0, min=26.0, mean=38.56099999999999, stddev=11.187467918990311}

22000 => 21000 : Snapshot

{size=1000, median=34.0, max=120.0, min=24.0, mean=37.007, stddev=11.303571193287661}

21000 => 20000 : Snapshot

{size=1000, median=33.0, max=128.0, min=24.0, mean=35.30800000000001, stddev=10.732209293352232}

20000 => 19000 : Snapshot

{size=1000, median=31.0, max=111.0, min=25.0, mean=33.44399999999999, stddev=9.915199199974579}

19000 => 18000 : Snapshot

{size=1000, median=29.0, max=105.0, min=24.0, mean=31.580000000000013, stddev=9.01337011560279}

18000 => 17000 : Snapshot

{size=1000, median=28.0, max=98.0, min=20.0, mean=28.800000000000004, stddev=7.423281289639059}

17000 => 16000 : Snapshot

{size=1000, median=24.0, max=108.0, min=14.0, mean=25.906999999999993, stddev=8.290058870483135}

16000 => 15000 : Snapshot

{size=1000, median=23.0, max=88.0, min=16.0, mean=24.333999999999996, stddev=7.487490000427813}

15000 => 14000 : Snapshot

{size=1000, median=21.0, max=96.0, min=15.0, mean=22.915000000000006, stddev=7.130260557343979}

14000 => 13000 : Snapshot

{size=1000, median=21.0, max=97.0, min=11.0, mean=22.265, stddev=7.283668500613395}

13000 => 12000 : Snapshot

{size=1000, median=20.0, max=99.0, min=13.0, mean=21.244999999999997, stddev=5.764043650008142}

12000 => 11000 : Snapshot

{size=1000, median=19.0, max=87.0, min=10.0, mean=19.677000000000003, stddev=5.623548283361263}

11000 => 10000 : Snapshot

{size=1000, median=17.0, max=87.0, min=11.0, mean=17.921000000000006, stddev=4.608687340121788}

10000 => 9000 : Snapshot

{size=1000, median=16.0, max=78.0, min=9.0, mean=16.274, stddev=4.204354956542137}

9000 => 8000 : Snapshot

{size=1000, median=14.0, max=64.0, min=8.0, mean=14.613, stddev=3.9908843829605445}

8000 => 7000 : Snapshot

{size=1000, median=12.0, max=66.0, min=6.0, mean=12.519999999999994, stddev=3.7634776156852245}

7000 => 6000 : Snapshot

{size=1000, median=10.0, max=80.0, min=6.0, mean=10.869, stddev=3.4617658532637123}

6000 => 5000 : Snapshot

{size=1000, median=9.0, max=62.0, min=4.0, mean=9.381, stddev=2.861472877213261}

5000 => 4000 : Snapshot

{size=1000, median=8.0, max=39.0, min=4.0, mean=8.041000000000004, stddev=2.601939486054422}

4000 => 3000 : Snapshot

{size=1000, median=6.0, max=33.0, min=3.0, mean=6.262999999999996, stddev=1.9358663638223068}

3000 => 2000 : Snapshot

{size=1000, median=5.0, max=33.0, min=2.0, mean=4.683000000000006, stddev=1.7797973701179062}

2000 => 1000 : Snapshot

{size=1000, median=3.0, max=29.0, min=2.0, mean=3.0350000000000033, stddev=1.5113103781351653}

1000 => 0 : Snapshot

{size=1000, median=1.0, max=15.0, min=0.0, mean=1.5, stddev=0.9278258785250931}

[brianoliver]

@brianoliver said:
Thanks for the detailed information!

This is the highest priority issue we're working on next for the Incubator.

– Brian

[brianoliver]

This issue was imported from JIRA COHINC-24