tag:blogger.com,1999:blog-379756900971116208.post766397244634519788..comments2023-09-04T01:48:39.071-07:00Comments on Pushpesh's Blog: OpenStack-Swift-VS-Ceph-RGW-READ-PerformanceUnknownnoreply@blogger.comBlogger18125tag:blogger.com,1999:blog-379756900971116208.post-17617288465687802862022-12-22T01:46:28.639-08:002022-12-22T01:46:28.639-08:00Good content. You write beautiful things.
vbet
m...Good content. You write beautiful things.
<br /><a href="https://bit.ly/vbet-vbet-vbet" title="vbet" rel="nofollow">vbet</a>
<br /><a href="https://bit.ly/mrbahis-mrbahis" title="mrbahis" rel="nofollow">mrbahis</a>
<br /><a href="https://taksikenti.com/taksi/" title="taksi" rel="nofollow">taksi</a>
<br /><a href="https://hacklink.gen.tr/" title="hacklink" rel="nofollow">hacklink</a>
<br /><a href="https://mrbahise.com/" title="mrbahis" rel="nofollow">mrbahis</a>
<br /><a href="https://taksikenti.com/" title="korsan taksi" rel="nofollow">korsan taksi</a>
<br /><a href="https://sportsbetgiris.net/" title="sportsbet" rel="nofollow">sportsbet</a>
<br /><a href="https://bit.ly/hacklink-hacklink-hacklink" title="hacklink" rel="nofollow">hacklink</a>
<br /><a href="https://vbettr.org/" title="vbet" rel="nofollow">vbet</a>sportsbethttps://bit.ly/sportsbet-sportsbet-sportsbetnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-53047036296901330482022-05-17T08:54:04.118-07:002022-05-17T08:54:04.118-07:00Fon perde modelleri
sms onay
mobil ödeme bozdurma
...<a href="https://www.perdemodelleri.org/" hreflang="tr" rel="nofollow">Fon perde modelleri</a><br /><a href="https://smsonayadresi.com/" hreflang="tr" rel="nofollow">sms onay</a><br /><a href="https://odemebozdurma.com/" title="mobil ödeme bozdurma" hreflang="tr" rel="nofollow">mobil ödeme bozdurma</a><br /><a href="https://nftnasilalinir.com/" title="nft nasıl alınır" rel="nofollow">nft nasıl alınır</a><br /><a href="https://bit.ly/ankara-evden-eve-nakliyat-ankara-evden-eve-nakliyat" hreflang="tr" rel="nofollow">ankara evden eve nakliyat</a><br /><a href="https://sigortix.com/" title="TRAFİK SİGORTASI" hreflang="tr" rel="nofollow">TRAFİK SİGORTASI</a><br /><a href="https://www.ugurelektronik.com/derin-arama-dedektorleri" hreflang="tr" rel="nofollow">dedektör</a><br /><a href="https://kurma.website/" title="web sitesi kurma" rel="nofollow">web sitesi kurma</a><br /><a href="https://www.askkitaplari.com/" rel="nofollow">Aşk kitapları</a><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-71384147034865118752022-04-28T18:36:19.377-07:002022-04-28T18:36:19.377-07:00mmorpg oyunlar
instagram takipçi satın al
Tiktok j...<a href="https://www.mmogamesturkiye.com/" title="mmorpg oyunlar" rel="nofollow">mmorpg oyunlar</a><br /><a href="https://bit.ly/instagram-takipci-satin-alz" rel="nofollow">instagram takipçi satın al</a><br /><a href="https://www.smsbankasi.net/2022/02/tiktok-jeton-hilesi.html" hreflang="tr" rel="nofollow">Tiktok jeton hilesi</a><br /><a href="https://www.hilelipc.net/" rel="nofollow">TİKTOK JETON HİLESİ</a><br /><a href="https://www.sacekimiburada.com/antalya-sac-ekim-merkezleri" title="Antalya saç ekimi" hreflang="tr" rel="nofollow">Antalya saç ekimi</a><br /><a href="https://www.smsbankasi.net/2022/01/binance-referans-kimligi-nedir-ne-ise.html" rel="nofollow">referans kimliği nedir</a><br /><a href="https://takipcialdim.com/" rel="nofollow">instagram takipçi satın al</a><br /><a href="https://www.mmogamesturkiye.com/metin-2-mt2/" rel="nofollow">metin pvp</a><br /><a href="https://takipcisatinalz.com/" title="instagram takipçi satın al" rel="nofollow">instagram takipçi satın al</a><br /><a href="https://takipcisatinalz.com/" hreflang="tr" rel="nofollow">instagram takipçi satın al</a><br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-64793619106945592482021-07-26T18:20:28.128-07:002021-07-26T18:20:28.128-07:00kayseriescortu.com - alacam.org - xescortun.com<a href="http://www.kayseriescortu.com/" rel="nofollow">kayseriescortu.com</a> - <a href="http://www.alacam.org/" rel="nofollow">alacam.org</a> - <a href="http://xescortun.com/" rel="nofollow">xescortun.com</a>SELÇUK AZİZhttps://www.blogger.com/profile/09016401847969929100noreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-91773211934606276992016-06-16T05:29:03.336-07:002016-06-16T05:29:03.336-07:00Hi,
When a rgw service is started, by default bel...Hi,<br /><br />When a rgw service is started, by default below pools are created.<br />.rgw.root<br />default.rgw.control<br />default.rgw.data.root<br />default.rgw.gc<br />default.rgw.log<br /><br />When a swift user is created, some default pools are created. But I would like to use "Pool_A" for the swift user.<br />From client when I run Cosbench the data should be placed in "Pool_A" instead of placing it in default pools. How can i achieve that.<br /><br />Also need help on how to run Cosbench with swift user. Your help is very much appreciated.<br /><br />Thanks,<br />kanchana.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-52827167469531629832015-11-12T00:42:04.827-08:002015-11-12T00:42:04.827-08:00My bad I have not checked it for a while, images s...My bad I have not checked it for a while, images should work now. Study is old and I have not worked on it for long, please read it with a pinch of salt. Ceph giant release is what i worked on things might have changed for better now. Thanks for making me aware about the disappeared images. pushpesh sharmahttps://www.blogger.com/profile/02658732148513831522noreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-22555152231364079562015-11-10T13:37:09.362-08:002015-11-10T13:37:09.362-08:00Hey Pushpesh,
The images for the configuration an...Hey Pushpesh,<br /><br />The images for the configuration and the results do not load on this page. Can you please look into it? Otherwise, do you have a pdf file of this blog that you can email it to me?Rajesh Sudarsanhttps://plus.google.com/u/0/+RajeshSudarsan/postsnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-85672334178098496892014-12-15T23:28:53.379-08:002014-12-15T23:28:53.379-08:00you can contact me @
PUSHPESH.ECK at GMAIL.COMyou can contact me @<br />PUSHPESH.ECK at GMAIL.COMpushpesh sharmahttps://www.blogger.com/profile/02658732148513831522noreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-45931801899843836292014-12-15T23:18:41.526-08:002014-12-15T23:18:41.526-08:00Hi Puspesh ,
its Anurag, I think you gave a pr...Hi Puspesh ,<br /> its Anurag, I think you gave a presentation on CEPH vs Swift Performance on OSI Days 2014 and i am interested in CEPH and trying to deploy it on simple commodity hardware just for POC purpose. but i am facing some issues and i am trying to do so inside a network with ntlm proxy using ceph-deploy . i have done it on Vm's with 2 osd and 1 monitor and its working but with physical machines i am facing problems.<br />may be you can give me a email id on which we can communicate.<br /> Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-3802002449060848642014-11-05T22:58:02.713-08:002014-11-05T22:58:02.713-08:00Hi All,
I would like to include your responses i...Hi All, <br /><br />I would like to include your responses in the blog post so it will benefit open source community in general. I will share the blog post with CEPH community mailing list and have more feedback from there. FYI, I am also giving a talk about this @ OpenStack India Mini-Conf (7-Nov-2014). I will present the Ceph-RGW results with a note that there are active discussion going on . <br /><br />I hope it make sense to all.<br /><br />PS: I am working on turntables, Yehuda suggested.pushpesh sharmahttps://www.blogger.com/profile/02658732148513831522noreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-59711681663267453792014-11-05T22:57:41.231-08:002014-11-05T22:57:41.231-08:00
Yehuda Sadeh
10:54 PM (13 hours ago)
to me, Mark...<br />Yehuda Sadeh<br />10:54 PM (13 hours ago)<br /><br />to me, Mark, Ben, Thiago, Peter, Kyle, John, Neil, Sage <br />Looking more closely into the blog post I have a couple more things:<br /><br />1. You set the following apache config:<br /><br />MaxRequestsPerChild 10000<br /><br />Which basically means that each apache process dies after getting 10k<br />connections. It really depends on how the benchmark is running, but it<br />might have severe performance impact. You should really be setting it<br />to zero.<br /><br />2. The small IO test uses relatively large IO size, beyond what the<br />gateway considers as small. It may benefit with adjusting its internal<br />IO params. You can add:<br /><br />rgw max chunk size = 1048576<br /><br />This will cut the number of rados IO operations in the small IO<br />benchmark by half.<br /><br />Thanks,<br />YehudaAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-50963810720461194052014-11-05T22:56:53.808-08:002014-11-05T22:56:53.808-08:00Hi All,
Thanks everyone for your time and valua...Hi All, <br /><br />Thanks everyone for your time and valuable feedback. <br /><br />Mark, <br /><br />There are chances of re-reading the Objects, randomization in reading is done by COSBench workload, so I just tried my best to avoid this. Yes "echo >1 " might prove good for both.<br /><br />Apache tuning was more or less based on http://httpd.apache.org/docs/2.2/mpm.html. I played around with these parameters and I believe these configs made Apache gates wide and open. I suspect FASTCGI is bigger problem here, I was also observing very high CPU usage(~95%) on RGW nodes at peak load. However I don't rule out possibility of better numbers with better tuning. <br /><br />I will try out the addition configurations recommended by you and Yahuda, will let you people know about my findings. <br /><br />I will try to attend the performance meeting. pushpesh sharmahttps://www.blogger.com/profile/02658732148513831522noreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-55894132593623651872014-11-05T22:56:26.020-08:002014-11-05T22:56:26.020-08:00Mark Nelson
Nov 5 (1 day ago)
to Yehuda, Ben, me...<br />Mark Nelson <br />Nov 5 (1 day ago)<br /><br />to Yehuda, Ben, me, Thiago, Peter, Kyle, John, Neil, Sage <br />I wanted to say too that we've traditionally not had any 40GbE or QDR IB equipment in our lab to test RGW at these speeds. Seeing it get 3GB/s+ (at least in the better cases) is actually not terrible for a first result! We just need to figure out where we are getting held up as Yehuda mentioned.<br />Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-36841805840436848332014-11-05T22:55:59.342-08:002014-11-05T22:55:59.342-08:00Yehuda Sadeh
Nov 5 (1 day ago)
to Mark, Ben, me, ...<br />Yehuda Sadeh<br />Nov 5 (1 day ago)<br /><br />to Mark, Ben, me, Thiago, Peter, Kyle, John, Neil, Sage <br />I agree with Mark that the apache config might need some tweaking.<br />Also, there are a few more configurables that will need to be updated:<br /><br />objecter_inflight_op_bytes = 0<br />objecter_inflight_ops = 0<br /><br />(I think that setting these to zero will disable them, if not then<br />will need to set them to higher values then the defaults)<br /><br />Specifically you can try looking at the radosgw admin socket, read the<br />perf counters during a run, and see if there's any throttling going<br />on.<br /><br />Another thing is that there were a bunch of performance related<br />changes going into the objecter in Giant that should help concurrency.<br />I did see that you mentioned that you saw performance regression<br />there, which might point at some other issue with the setup.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-81644775819377239772014-11-05T22:55:34.945-08:002014-11-05T22:55:34.945-08:00Mark Nelson
Nov 5 (1 day ago)
to Sage, yehuda, B...<br />Mark Nelson <br />Nov 5 (1 day ago)<br /><br />to Sage, yehuda, Ben, me, Thiago, Peter, Kyle, John, Neil <br />Hi Pushpesh,<br /><br />A couple of comments:<br /><br />I noticed that you are doing:<br /><br />sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"<br /><br />every minute which will flush dentry/inode cache. With how ceph lays objects out on the disk (and I suspect swift too) this is will hurt performance for small objects. Given that you are using SSDs I'm not sure how much effect it will have, but it's something to keep in mind. Are you re-reading objects? If so, maybe an "echo 1" here might be better to avoid cached reads while keeping dentries/inodes in cache?<br /><br />Also, with the number of total objects in the 1MB object tests, you might want to tweak these parameters:<br /><br />filestore_merge_threshold<br />filestore_split_multiple<br /><br />By default those settings restrict each PG to 300 objects before the nested directory structure where objects are stored for each PG is split into a deeper tree. With 1 million objects, 3x replication, and 4096 PGs, you likely hit those limits and caused the directory structure to split at least once. With traditional spinning disks this can have a big performance impact, though again I'm not sure how much of one you would see with SSDs.<br /><br />Having said that, it's very interesting that your civitweb results for RGW are so much faster than the apache results. I'm not an apache tuning expert, so perhaps there is more tuning we can do on that side.<br /><br />If you'd like, we have a weekly public ceph performance meeting on Wednesday mornings at 8AM PST. We'd be happy to have you join and talk about what you've been seeing! In particular, Somnath Roy from Sandisk has been doing a ton of work optimizing the read path in Ceph and RGW and might have some additional comments on your tests.<br /><br />Here's the meeting etherpad:<br /><br />http://pad.ceph.com/p/performance_weekly<br /><br />And the meeting URL:<br /><br />https://bluejeans.com/268261044/browserAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-68693947929561519812014-11-05T22:54:30.792-08:002014-11-05T22:54:30.792-08:00Ben England
Nov 5 (1 day ago)
to me, Mark, Thiag...Ben England <br />Nov 5 (1 day ago)<br /><br />to me, Mark, Thiago, Peter, Kyle, John, Neil <br />Thanks Pushpesh, this should give us a lot of ideas about how to tune Ceph and RGW in particular. I'm having some trouble understanding how to configure it so it will help there too. cc'ing Kyle Bader (RGW developer) in case this interests him.<br /><br />-benAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-45537534750163705102014-11-05T22:54:07.558-08:002014-11-05T22:54:07.558-08:00I am posting some of the mail conversation, that m...I am posting some of the mail conversation, that might be useful for everyone. pushpesh sharmahttps://www.blogger.com/profile/02658732148513831522noreply@blogger.comtag:blogger.com,1999:blog-379756900971116208.post-31313299392538043942014-11-05T21:35:33.565-08:002014-11-05T21:35:33.565-08:00Good doc Pushpesh :)Good doc Pushpesh :)Anonymousnoreply@blogger.com