Skip to content
This repository has been archived by the owner on Jan 13, 2022. It is now read-only.

Poor read hit ratio #189

Open
mozg31337 opened this issue Oct 16, 2014 · 8 comments
Open

Poor read hit ratio #189

mozg31337 opened this issue Oct 16, 2014 · 8 comments

Comments

@mozg31337
Copy link

Hello guys,

I seems to be getting a very poor read hit percentage for my flashcache setup. According to the dmsetup status I am getting between 5-8% read hits. As na example, with default EnhanceIO caching on the same cluster I was getting over 20% hit ratio.

I've got a small ceph cluster with 16 hdds and 4 ssds that I use for flashcache. I am using the Write Around caching method using the following additional sysctrl tunning options:

reclaim_policy = 1
skip_seq_thresh_kb = 4096

My hdds are formatted using XFS with default ceph-deploy settings which are:

-f -i size=2048

The XFS file system is mounted like this:

rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M

What can I do to improve the read hit ratio?

Thanks

Andrei
@mohans
Copy link
Contributor

mohans commented Oct 16, 2014

Can you paste the output of 'dmsetup table' and 'dmsetup status' here ?

On Thursday, October 16, 2014 10:24 AM, mozg31337 [email protected] wrote:

Hello guys,
I seems to be getting a very poor read hit percentage for my flashcache setup. According to the dmsetup status I am getting between 5-8% read hits. As na example, with default EnhanceIO caching on the same cluster I was getting over 20% hit ratio.
I've got a small ceph cluster with 16 hdds and 4 ssds that I use for flashcache. I am using the Write Around caching method using the following additional sysctrl tunning options:
reclaim_policy = 1
skip_seq_thresh_kb = 4096
My hdds are formatted using XFS with default ceph-deploy settings which are:
-f -i size=2048
The XFS file system is mounted like this:
rw,noatime,inode64,logbsize=256k,delaylog,allocsize=4M
What can I do to improve the read hit ratio?
Thanks
Andrei
—
Reply to this email directly or view it on GitHub.

@mozg31337
Copy link
Author

Here you go:

dmsetup table output:

2014-10-16 01:09:15.600807 osd.7 192.168.168.200:6836/32031 100 : [WRN] slow request 30.999641 seconds old, received at 2014-10-16 01:08:44.601040: osd_op(client.36035566.0:16626375 rbd_data.51da686763845e
.0000000000005a15 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704016384] 5.7b16421b snapc c4=[c4] ack+ondisk+write e61892) v4 currently waiting for subops from 9
2014-10-16 01:09:15.600811 osd.7 192.168.168.200:6836/32031 101 : [WRN] slow request 30.999581 seconds old, received at 2014-10-16 01:08:44.601100: osd_op(client.36035566.0:16626376 rbd_data.51da686763845e
.0000000000005a15 [set-alloc-hint object_size 4194304 write_size 4194304,write 203980816384] 5.7b16421b snapc c4=[c4] ack+ondisk+write e61892) v4 currently waiting for subops from 9
2014-10-16 01:09:16.185530 osd.2 192.168.168.200:6811/31891 76 : [WRN] 20 slow requests, 1 included below; oldest blocked for > 57.003961 secs
2014-10-16 01:09:16.185564 osd.2 192.168.168.200:6811/31891 77 : [WRN] slow request 30.098574 seconds old, received at 2014-10-16 01:08:46.086854: osd_op(client.38917806.0:3481697 rbd_data.251d05e3db45a54.
0000000000000304 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~8192] 5.e4683bbb ack+ondisk+write e61892) v4 currently waiting for subops from 11
2014-10-16 01:09:16.601020 osd.7 192.168.168.200:6836/32031 102 : [WRN] 16 slow requests, 2 included below; oldest blocked for > 43.531516 secs
osd_5_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC4FF69-part3), disk dev (/dev/disk/by-uuid/3df02eed-c231-48a0-92a1-a35825b656e8) cache mode(WRITE_AROUND)
capacity(145912M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(37353472), cached blocks(35772140), cache percent(95)
nr_queued(0)
Size Hist: 512:2332305 1024:1880658 1536:1853673 2048:1827770 2560:1788114 3072:1762958 3584:1738733 4096:1890815275
osd_2_CACHE: 0 5818588047 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C246D84-part3), disk dev (/dev/disk/by-uuid/1d8cdedb-bae4-43d9-b354-8ab783ab12bf) cache mode(WRITE_AROUND)
capacity(145912M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(37353472), cached blocks(35134037), cache percent(94)
nr_queued(0)
Size Hist: 512:2332305 1024:1880658 1536:1853673 2048:1827770 2560:1788114 3072:1762958 3584:1738733 4096:1890815275
osd_6_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_OCZ-REVODRIVE3_OCZ-67BQ9IG590HGJ69D-part1), disk dev (/dev/disk/by-uuid/a10c8573-1410-421f-afa9-0c703d714380) cache mode(WRITE_AROUND)
capacity(51496M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(13182976), cached blocks(12932886), cache percent(98)
nr_queued(0)
Size Hist: 512:2332305 1024:1880658 1536:1853673 2048:1827770 2560:1788114 3072:1762958 3584:1738733 4096:1890815275
osd_3_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC4FF69-part1), disk dev (/dev/disk/by-uuid/718ddbd2-4780-4ac7-8c48-ab8ac5b2599a) cache mode(WRITE_AROUND)
capacity(145910M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(37352960), cached blocks(36645912), cache percent(98)
nr_queued(0)
Size Hist: 512:2332305 1024:1880658 1536:1853673 2048:1827770 2560:1788114 3072:1762958 3584:1738733 4096:1890815275
osd_15_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC50061-part3), disk dev (/dev/disk/by-uuid/d494c2b9-5f4e-411e-9785-723730ad5e65) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27758138), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_12_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C228EA7-part4), disk dev (/dev/disk/by-uuid/3e39f94a-a546-4061-a09b-d61592340681) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27729179), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_16_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC50061-part4), disk dev (/dev/disk/by-uuid/3127f614-4026-48c8-81da-1b10c38b606b) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27727035), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_13_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC50061-part1), disk dev (/dev/disk/by-uuid/3aa80403-b675-4355-8cd2-662b202ddc0f) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27781031), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_10_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C228EA7-part2), disk dev (/dev/disk/by-uuid/7d5669b1-9d0c-4f55-8ece-d566c33a93d1) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27775253), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_14_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC50061-part2), disk dev (/dev/disk/by-uuid/84541b70-569d-444a-b7c5-0897b192d306) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27875673), cache percent(99)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_9_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C228EA7-part1), disk dev (/dev/disk/by-uuid/e879df77-f4dc-40ad-9b45-812661ea71cb) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27606923), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_11_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C228EA7-part3), disk dev (/dev/disk/by-uuid/2c90d17a-a25c-4016-ae59-159894831e7f) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27198334), cache percent(96)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653

#################

dmsetup status output:

osd_0_CACHE: 0 5818588047 flashcache stats:
reads(95864773), writes(53836173)
read hits(10216256), read hit percent(10)
replacement(38711132), write replacement(0)
invalidates(1)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(85648759), disk writes(53836281) ssd reads(10216275) ssd writes(73381597)
uncached reads(12267224), uncached writes(53836280), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(12266416), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676473), lru warm blocks(18676452)
lru promotions(1986176), lru demotions(0)
osd_7_CACHE: 0 5860531087 flashcache stats:
reads(189858408), writes(55130263)
read hits(9257098), read hit percent(4)
replacement(45754494), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(180601863), disk writes(55130346) ssd reads(9257136) ssd writes(60970851)
uncached reads(119631341), uncached writes(55130343), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(119629379), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(6591734), lru warm blocks(6591637)
lru promotions(1010641), lru demotions(0)
osd_4_CACHE: 0 5818588047 flashcache stats:
reads(149632315), writes(47642084)
read hits(13106379), read hit percent(8)
replacement(82587148), write replacement(0)
invalidates(1)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(136526226), disk writes(47642138) ssd reads(13106409) ssd writes(120721623)
uncached reads(15804678), uncached writes(47642137), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(15803885), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676733), lru warm blocks(18676701)
lru promotions(1394608), lru demotions(0)
osd_1_CACHE: 0 5860531087 flashcache stats:
reads(188146837), writes(55126618)
read hits(12814256), read hit percent(6)
replacement(107985695), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(175332881), disk writes(55126703) ssd reads(12814291) ssd writes(147270806)
uncached reads(28062137), uncached writes(55126701), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(28060182), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676732), lru warm blocks(18676687)
lru promotions(1380694), lru demotions(0)
osd_8_CACHE: 0 5860531087 flashcache stats:
reads(112933715), writes(50564099)
read hits(9510750), read hit percent(8)
replacement(53775979), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(103423164), disk writes(50564163) ssd reads(9510780) ssd writes(81165752)
uncached reads(22257467), uncached writes(50564161), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(22255511), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(13183480), lru warm blocks(13183452)
lru promotions(1562720), lru demotions(0)
osd_5_CACHE: 0 5860531087 flashcache stats:
reads(168782526), writes(56690881)
read hits(12133534), read hit percent(7)
replacement(76271507), write replacement(0)
invalidates(1)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(156649289), disk writes(56690990) ssd reads(12133589) ssd writes(114838320)
uncached reads(41811076), uncached writes(56690990), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(41810263), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676728), lru warm blocks(18676682)
lru promotions(1550950), lru demotions(0)
osd_2_CACHE: 0 5818588047 flashcache stats:
reads(136278022), writes(40067323)
read hits(12506091), read hit percent(9)
replacement(69395645), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(123772174), disk writes(40067351) ssd reads(12506140) ssd writes(106773724)
uncached reads(16998502), uncached writes(40067351), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(16996539), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676725), lru warm blocks(18676706)
lru promotions(1672532), lru demotions(0)
osd_6_CACHE: 0 5860531087 flashcache stats:
reads(214859592), writes(73920676)
read hits(8748573), read hit percent(4)
replacement(59654930), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(206111231), disk writes(73920769) ssd reads(8748596) ssd writes(74889316)
uncached reads(131222119), uncached writes(73920768), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(131220166), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(6591476), lru warm blocks(6591408)
lru promotions(925642), lru demotions(0)
osd_3_CACHE: 0 5860531087 flashcache stats:
reads(171829573), writes(43406122)
read hits(15135146), read hit percent(8)
replacement(106704196), write replacement(0)
invalidates(1)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(156694778), disk writes(43406135) ssd reads(15135193) ssd writes(145602353)
uncached reads(11092511), uncached writes(43406134), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(11091710), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676471), lru warm blocks(18676413)
lru promotions(1784672), lru demotions(0)
osd_15_CACHE: 0 5860531087 flashcache stats:
reads(175184320), writes(54273732)
read hits(10549543), read hit percent(6)
replacement(112767630), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(164635152), disk writes(54273942) ssd reads(10549573) ssd writes(142948456)
uncached reads(21686779), uncached writes(54273938), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(21684819), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038014), lru warm blocks(14037964)
lru promotions(1299939), lru demotions(0)
osd_12_CACHE: 0 5860531087 flashcache stats:
reads(193144232), writes(63252992)
read hits(11117355), read hit percent(5)
replacement(122987332), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(182027309), disk writes(63253169) ssd reads(11117407) ssd writes(153544737)
uncached reads(28482699), uncached writes(63253168), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(28480754), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038010), lru warm blocks(14037938)
lru promotions(1071529), lru demotions(0)
osd_16_CACHE: 0 5860531087 flashcache stats:
reads(177166229), writes(60110110)
read hits(10512479), read hit percent(5)
replacement(117498515), write replacement(0)
invalidates(2)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(166654151), disk writes(60110226) ssd reads(10512549) ssd writes(147703763)
uncached reads(18950472), uncached writes(60110226), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(18949682), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038007), lru warm blocks(14037968)
lru promotions(1074124), lru demotions(0)
osd_13_CACHE: 0 5860531087 flashcache stats:
reads(178746196), writes(68304081)
read hits(10329802), read hit percent(5)
replacement(113006629), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(168416832), disk writes(68304252) ssd reads(10329840) ssd writes(143360310)
uncached reads(25056628), uncached writes(68304252), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(25054680), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038008), lru warm blocks(14037936)
lru promotions(1077035), lru demotions(0)
osd_10_CACHE: 0 5860531087 flashcache stats:
reads(251580928), writes(66123353)
read hits(14032457), read hit percent(5)
replacement(174804329), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(237549126), disk writes(66123510) ssd reads(14032513) ssd writes(205422529)
uncached reads(32126811), uncached writes(66123509), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(32124868), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14037999), lru warm blocks(14037910)
lru promotions(1143940), lru demotions(0)
osd_14_CACHE: 0 5860531087 flashcache stats:
reads(180824247), writes(60381058)
read hits(11944008), read hit percent(6)
replacement(118670090), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(168880778), disk writes(60381189) ssd reads(11944031) ssd writes(149074298)
uncached reads(19806558), uncached writes(60381187), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(19804617), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038006), lru warm blocks(14037953)
lru promotions(1303845), lru demotions(0)
osd_9_CACHE: 0 5860531087 flashcache stats:
reads(149719231), writes(64327079)
read hits(8662063), read hit percent(5)
replacement(87304654), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(141057642), disk writes(64327255) ssd reads(8662097) ssd writes(117662400)
uncached reads(23395357), uncached writes(64327255), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(23393409), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038006), lru warm blocks(14037962)
lru promotions(1069380), lru demotions(0)
osd_11_CACHE: 0 5860531087 flashcache stats:
reads(122059212), writes(45266263)
read hits(8989761), read hit percent(7)
replacement(66907106), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(113069783), disk writes(45266365) ssd reads(8989790) ssd writes(96249429)
uncached reads(16820452), uncached writes(45266366), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(16818513), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038005), lru warm blocks(14037978)
lru promotions(1283892), lru demotions(0)

##################

Many thanks

@mohans
Copy link
Contributor

mohans commented Oct 16, 2014

I don't see anything obviously broken in the flashcache configuration. The vast majority of the IO requests coming into flashcache are 4K, so they should be cacheable.

How large is the disk volume that you are caching ? And what is your workload (do you expect it to be cacheable and do you expect a healthy cache hit rate) ?

You could completely turn off skip sequential and see if that has any impact on the cache hit rate, just as an experiment.

On Thursday, October 16, 2014 12:13 PM, mozg31337 [email protected] wrote:

Here you go:
dmsetup table output:
2014-10-16 01:09:15.600807 osd.7 192.168.168.200:6836/32031 100 : [WRN] slow request 30.999641 seconds old, received at 2014-10-16 01:08:44.601040: osd_op(client.36035566.0:16626375 rbd_data.51da686763845e
.0000000000005a15 [set-alloc-hint object_size 4194304 write_size 4194304,write 200704016384] 5.7b16421b snapc c4=[c4] ack+ondisk+write e61892) v4 currently waiting for subops from 9
2014-10-16 01:09:15.600811 osd.7 192.168.168.200:6836/32031 101 : [WRN] slow request 30.999581 seconds old, received at 2014-10-16 01:08:44.601100: osd_op(client.36035566.0:16626376 rbd_data.51da686763845e
.0000000000005a15 [set-alloc-hint object_size 4194304 write_size 4194304,write 203980816384] 5.7b16421b snapc c4=[c4] ack+ondisk+write e61892) v4 currently waiting for subops from 9
2014-10-16 01:09:16.185530 osd.2 192.168.168.200:6811/31891 76 : [WRN] 20 slow requests, 1 included below; oldest blocked for > 57.003961 secs
2014-10-16 01:09:16.185564 osd.2 192.168.168.200:6811/31891 77 : [WRN] slow request 30.098574 seconds old, received at 2014-10-16 01:08:46.086854: osd_op(client.38917806.0:3481697 rbd_data.251d05e3db45a54.
0000000000000304 [stat,set-alloc-hint object_size 4194304 write_size 4194304,write 2732032~8192] 5.e4683bbb ack+ondisk+write e61892) v4 currently waiting for subops from 11
2014-10-16 01:09:16.601020 osd.7 192.168.168.200:6836/32031 102 : [WRN] 16 slow requests, 2 included below; oldest blocked for > 43.531516 secs
osd_5_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC4FF69-part3), disk dev (/dev/disk/by-uuid/3df02eed-c231-48a0-92a1-a35825b656e8) cache mode(WRITE_AROUND)
capacity(145912M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(37353472), cached blocks(35772140), cache percent(95)
nr_queued(0)
Size Hist: 512:2332305 1024:1880658 1536:1853673 2048:1827770 2560:1788114 3072:1762958 3584:1738733 4096:1890815275
osd_2_CACHE: 0 5818588047 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C246D84-part3), disk dev (/dev/disk/by-uuid/1d8cdedb-bae4-43d9-b354-8ab783ab12bf) cache mode(WRITE_AROUND)
capacity(145912M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(37353472), cached blocks(35134037), cache percent(94)
nr_queued(0)
Size Hist: 512:2332305 1024:1880658 1536:1853673 2048:1827770 2560:1788114 3072:1762958 3584:1738733 4096:1890815275
osd_6_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_OCZ-REVODRIVE3_OCZ-67BQ9IG590HGJ69D-part1), disk dev (/dev/disk/by-uuid/a10c8573-1410-421f-afa9-0c703d714380) cache mode(WRITE_AROUND)
capacity(51496M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(13182976), cached blocks(12932886), cache percent(98)
nr_queued(0)
Size Hist: 512:2332305 1024:1880658 1536:1853673 2048:1827770 2560:1788114 3072:1762958 3584:1738733 4096:1890815275
osd_3_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC4FF69-part1), disk dev (/dev/disk/by-uuid/718ddbd2-4780-4ac7-8c48-ab8ac5b2599a) cache mode(WRITE_AROUND)
capacity(145910M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(37352960), cached blocks(36645912), cache percent(98)
nr_queued(0)
Size Hist: 512:2332305 1024:1880658 1536:1853673 2048:1827770 2560:1788114 3072:1762958 3584:1738733 4096:1890815275
osd_15_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC50061-part3), disk dev (/dev/disk/by-uuid/d494c2b9-5f4e-411e-9785-723730ad5e65) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27758138), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_12_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C228EA7-part4), disk dev (/dev/disk/by-uuid/3e39f94a-a546-4061-a09b-d61592340681) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27729179), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_16_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC50061-part4), disk dev (/dev/disk/by-uuid/3127f614-4026-48c8-81da-1b10c38b606b) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27727035), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_13_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC50061-part1), disk dev (/dev/disk/by-uuid/3aa80403-b675-4355-8cd2-662b202ddc0f) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27781031), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_10_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C228EA7-part2), disk dev (/dev/disk/by-uuid/7d5669b1-9d0c-4f55-8ece-d566c33a93d1) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27775253), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_14_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14300CC50061-part2), disk dev (/dev/disk/by-uuid/84541b70-569d-444a-b7c5-0897b192d306) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27875673), cache percent(99)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_9_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C228EA7-part1), disk dev (/dev/disk/by-uuid/e879df77-f4dc-40ad-9b45-812661ea71cb) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27606923), cache percent(98)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
osd_11_CACHE: 0 5860531087 flashcache conf:
ssd dev (/dev/disk/by-id/scsi-SATA_Crucial_CT512MX_14200C228EA7-part3), disk dev (/dev/disk/by-uuid/2c90d17a-a25c-4016-ae59-159894831e7f) cache mode(WRITE_AROUND)
capacity(109672M), associativity(512), data block size(4K)
disk assoc(256K)
skip sequential thresh(4096K)
total blocks(28076032), cached blocks(27198334), cache percent(96)
nr_queued(0)
Size Hist: 512:2276117 1024:1883950 1536:1870721 2048:1851248 2560:1811509 3072:1787058 3584:1745710 4096:1896659653
#################
dmsetup status output:
osd_0_CACHE: 0 5818588047 flashcache stats:
reads(95864773), writes(53836173)
read hits(10216256), read hit percent(10)
replacement(38711132), write replacement(0)
invalidates(1)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(85648759), disk writes(53836281) ssd reads(10216275) ssd writes(73381597)
uncached reads(12267224), uncached writes(53836280), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(12266416), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676473), lru warm blocks(18676452)
lru promotions(1986176), lru demotions(0)
osd_7_CACHE: 0 5860531087 flashcache stats:
reads(189858408), writes(55130263)
read hits(9257098), read hit percent(4)
replacement(45754494), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(180601863), disk writes(55130346) ssd reads(9257136) ssd writes(60970851)
uncached reads(119631341), uncached writes(55130343), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(119629379), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(6591734), lru warm blocks(6591637)
lru promotions(1010641), lru demotions(0)
osd_4_CACHE: 0 5818588047 flashcache stats:
reads(149632315), writes(47642084)
read hits(13106379), read hit percent(8)
replacement(82587148), write replacement(0)
invalidates(1)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(136526226), disk writes(47642138) ssd reads(13106409) ssd writes(120721623)
uncached reads(15804678), uncached writes(47642137), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(15803885), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676733), lru warm blocks(18676701)
lru promotions(1394608), lru demotions(0)
osd_1_CACHE: 0 5860531087 flashcache stats:
reads(188146837), writes(55126618)
read hits(12814256), read hit percent(6)
replacement(107985695), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(175332881), disk writes(55126703) ssd reads(12814291) ssd writes(147270806)
uncached reads(28062137), uncached writes(55126701), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(28060182), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676732), lru warm blocks(18676687)
lru promotions(1380694), lru demotions(0)
osd_8_CACHE: 0 5860531087 flashcache stats:
reads(112933715), writes(50564099)
read hits(9510750), read hit percent(8)
replacement(53775979), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(103423164), disk writes(50564163) ssd reads(9510780) ssd writes(81165752)
uncached reads(22257467), uncached writes(50564161), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(22255511), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(13183480), lru warm blocks(13183452)
lru promotions(1562720), lru demotions(0)
osd_5_CACHE: 0 5860531087 flashcache stats:
reads(168782526), writes(56690881)
read hits(12133534), read hit percent(7)
replacement(76271507), write replacement(0)
invalidates(1)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(156649289), disk writes(56690990) ssd reads(12133589) ssd writes(114838320)
uncached reads(41811076), uncached writes(56690990), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(41810263), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676728), lru warm blocks(18676682)
lru promotions(1550950), lru demotions(0)
osd_2_CACHE: 0 5818588047 flashcache stats:
reads(136278022), writes(40067323)
read hits(12506091), read hit percent(9)
replacement(69395645), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(123772174), disk writes(40067351) ssd reads(12506140) ssd writes(106773724)
uncached reads(16998502), uncached writes(40067351), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(16996539), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676725), lru warm blocks(18676706)
lru promotions(1672532), lru demotions(0)
osd_6_CACHE: 0 5860531087 flashcache stats:
reads(214859592), writes(73920676)
read hits(8748573), read hit percent(4)
replacement(59654930), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(206111231), disk writes(73920769) ssd reads(8748596) ssd writes(74889316)
uncached reads(131222119), uncached writes(73920768), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(131220166), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(6591476), lru warm blocks(6591408)
lru promotions(925642), lru demotions(0)
osd_3_CACHE: 0 5860531087 flashcache stats:
reads(171829573), writes(43406122)
read hits(15135146), read hit percent(8)
replacement(106704196), write replacement(0)
invalidates(1)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(156694778), disk writes(43406135) ssd reads(15135193) ssd writes(145602353)
uncached reads(11092511), uncached writes(43406134), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(11091710), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(18676471), lru warm blocks(18676413)
lru promotions(1784672), lru demotions(0)
osd_15_CACHE: 0 5860531087 flashcache stats:
reads(175184320), writes(54273732)
read hits(10549543), read hit percent(6)
replacement(112767630), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(164635152), disk writes(54273942) ssd reads(10549573) ssd writes(142948456)
uncached reads(21686779), uncached writes(54273938), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(21684819), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038014), lru warm blocks(14037964)
lru promotions(1299939), lru demotions(0)
osd_12_CACHE: 0 5860531087 flashcache stats:
reads(193144232), writes(63252992)
read hits(11117355), read hit percent(5)
replacement(122987332), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(182027309), disk writes(63253169) ssd reads(11117407) ssd writes(153544737)
uncached reads(28482699), uncached writes(63253168), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(28480754), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038010), lru warm blocks(14037938)
lru promotions(1071529), lru demotions(0)
osd_16_CACHE: 0 5860531087 flashcache stats:
reads(177166229), writes(60110110)
read hits(10512479), read hit percent(5)
replacement(117498515), write replacement(0)
invalidates(2)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(166654151), disk writes(60110226) ssd reads(10512549) ssd writes(147703763)
uncached reads(18950472), uncached writes(60110226), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(18949682), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038007), lru warm blocks(14037968)
lru promotions(1074124), lru demotions(0)
osd_13_CACHE: 0 5860531087 flashcache stats:
reads(178746196), writes(68304081)
read hits(10329802), read hit percent(5)
replacement(113006629), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(168416832), disk writes(68304252) ssd reads(10329840) ssd writes(143360310)
uncached reads(25056628), uncached writes(68304252), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(25054680), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038008), lru warm blocks(14037936)
lru promotions(1077035), lru demotions(0)
osd_10_CACHE: 0 5860531087 flashcache stats:
reads(251580928), writes(66123353)
read hits(14032457), read hit percent(5)
replacement(174804329), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(237549126), disk writes(66123510) ssd reads(14032513) ssd writes(205422529)
uncached reads(32126811), uncached writes(66123509), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(32124868), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14037999), lru warm blocks(14037910)
lru promotions(1143940), lru demotions(0)
osd_14_CACHE: 0 5860531087 flashcache stats:
reads(180824247), writes(60381058)
read hits(11944008), read hit percent(6)
replacement(118670090), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(168880778), disk writes(60381189) ssd reads(11944031) ssd writes(149074298)
uncached reads(19806558), uncached writes(60381187), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(19804617), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038006), lru warm blocks(14037953)
lru promotions(1303845), lru demotions(0)
osd_9_CACHE: 0 5860531087 flashcache stats:
reads(149719231), writes(64327079)
read hits(8662063), read hit percent(5)
replacement(87304654), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(141057642), disk writes(64327255) ssd reads(8662097) ssd writes(117662400)
uncached reads(23395357), uncached writes(64327255), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(23393409), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038006), lru warm blocks(14037962)
lru promotions(1069380), lru demotions(0)
osd_11_CACHE: 0 5860531087 flashcache stats:
reads(122059212), writes(45266263)
read hits(8989761), read hit percent(7)
replacement(66907106), write replacement(0)
invalidates(0)
pending enqueues(0), pending inval(0)
no room(0)
disk reads(113069783), disk writes(45266365) ssd reads(8989790) ssd writes(96249429)
uncached reads(16820452), uncached writes(45266366), uncached IO requeue(0)
disk read errors(0), disk write errors(0) ssd read errors(0) ssd write errors(0)
uncached sequential reads(16818513), uncached sequential writes(0)
pid_adds(0), pid_dels(0), pid_drops(0) pid_expiry(0)
lru hot blocks(14038005), lru warm blocks(14037978)
lru promotions(1283892), lru demotions(0)
##################
Many thanks
—
Reply to this email directly or view it on GitHub.

@mozg31337
Copy link
Author

My hdds are 3TB in size and each hdd is getting between 120 and 160 GB of ssd space. The hdds are used by about 40%. In terms of the usage, it's really a mixed scenario with a combination of web/sql/os data. I've got around 50-60 vms, which are not being loaded by much.

I will decrease the size of the sequential threshold and see if it helps. My guess is that it will help it as the Enhanceio doesn't have that capability and has a lot of similarities with flashcache as far as i've read on the net.

The reason for enabling the seq threshold is that ceph performs a regular scrubbing activity on the hdds, which will push the hot data out of cache. The scrubbing itself looks like a seq activity, thus I was hoping to minimise the push out effect of the hot data.

@akiradeveloper
Copy link

The hit ratio of only 20% doesn't improve performance so much (think of Amdhal's low). I think both flashcache and enhanceio aren't the best solution for your workload.

@mozg31337
Copy link
Author

What kind of workload does flashcache / enhanceio work well with?

@ghost
Copy link

ghost commented Dec 15, 2014

I just installed flashcache and I have a 6% read hit ratio so far on a very loaded box. How long does the cache take to build? What is a good read hit rate?

@ghost
Copy link

ghost commented Aug 4, 2015

Thank you for reporting this issue and appreciate your patience. We've notified the core team for an update on this issue. We're looking for a response within the next 30 days or the issue may be closed.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mohans @akiradeveloper @mozg31337