Memcached Code Camp 2009

Notes:	Hide Notes

Slide 1: memcached scaling your website with memcached by: steve yen Slide 2: about me • Steve Yen • NorthScale • Escalate Software • Kiva Software Slide 3: what you’ll learn • what, where, why, when • how • especially, best practices Slide 4: “mem cache dee” • latest version1.4.1 • http://code.google.com/p/memcached Slide 5: open source Slide 6: distributed cache Slide 7: livejournal Slide 8: helps your websites run fast Slide 9: popular Slide 10: simple Slide 11: KISS Slide 12: easy Slide 13: small bite-sized steps • not a huge, forklift replacement rearchitecture / reengineering project Slide 14: fast Slide 15: “i only block for memcached” Slide 16: scalable Slide 17: many client libraries • might be TOO many • the hit list... • Java ==> spymemcached • C ==> libmemcached • Python, Ruby, etc ==> • libmemcached wrappers Slide 18: frameworks • rails • django • spring / hibernate • cakephp, symphony, etc Slide 19: applications • drupal • wordpress • mediawiki • etc Slide 20: it works it promises to solve performance problems it delivers! Slide 21: problem? Slide 22: your website is too slow Slide 23: RDBMS melting down Slide 24: urgent! emergency Slide 25: one server web app + RDBMS Slide 26: 1 + 1 servers web app RDBMS Slide 27: N + 1 servers web app, web app, web app, web app RDBMS Slide 28: RDBMS Slide 29: EXPLAIN PLAN? Slide 30: buy a bigger box Slide 31: buy better disks Slide 32: master write DB + multiple read DB? Slide 33: vertical partitioning? Slide 34: sharding? Slide 35: uh oh, big reengineering • risky! • touch every line of code, every query!! Slide 36: and, it’s 2AM Slide 37: you need a band-aid Slide 38: a simple band-aid now Slide 39: use a cache Slide 40: keep things in memory! Slide 41: don’t hit disk Slide 42: distributed cache • to avoid wasting memory Slide 43: don’t write one of these yourself Slide 44: memcached Slide 45: simple API • hash-table-ish Slide 46: your code before v = db.query( SOME SLOW QUERY ) Slide 47: your code after v = memcachedClient.get(key) if (!v) { v = db.query( SOME SLOW QUERY ) memcachedClient.set(key, v) } Slide 48: cache read-heavy stuff Slide 49: invalidate when writing • db.execute(“UPDATE foo WHERE ...”) • memcachedClient.delete(...) Slide 50: and, repeat • each day... • look for the next slowest operations • add code to cache a few more things Slide 51: your life gets better Slide 52: thank you memcached! Slide 53: no magic Slide 54: you are in control Slide 55: now for the decisions Slide 56: memcached adoption • ﬁrst, start using memcached • poorly • but you can breathe again Slide 57: memcached adoption • next, start using memcached correctly Slide 58: memcached adoption • later • queueing • persistence • replication • ... Slide 59: an early question Slide 60: where to run servers? Slide 61: answer 1 • right on your web servers • a great place to start, if you have extra memory Slide 62: servers web app web app web app web app memcached memcached memcached, memcached RDBMS Slide 63: add up your memory usage! • having memcached server swap == bad! Slide 64: answer 2 • run memcached right on your database server? • WRONG! Slide 65: answer 3 • run memcached on separate dedicated memcached servers • congratulations! • you either have enough money • or enough trafﬁc that it matters Slide 66: running a server • daemonize • don’t be root! • no security Slide 67: server lists • mc-server1:11211 • mc-server2:11211 • mc-server3:11211 Slide 68: consistent hashing source: http://www.spiteful.com/2008/03/17/programmers-toolbox-part-3-consistent-hashing/ Slide 69: client-side intelligence • no “server master” bottleneck Slide 70: libmemcached • fast C memcached client • supports consistent hashing • many wrappers to your favorite languages Slide 71: updating server lists • push out new conﬁgs and restart? • moxi • memcached + integrated proxy Slide 72: keys • no whitespace • 250 char limit • use short preﬁxes Slide 73: keys & MD5 • don’t • stats become useless Slide 74: values • any binary object • 1MB limit • change #deﬁne & recompile if you want more • and you’re probably doing something wrong if you want more Slide 75: values • query resultset • • serialized object • • page fragment pages etc Slide 76: nginx + memcached Slide 77: >1 language? • JSON • protocol buffers • XML Slide 78: memcached is lossy • memcached WILL lose data Slide 79: that’s a good thing remember, it’s a CACHE Slide 80: why is memcached lossy? Slide 81: memcached node dies Slide 82: when node restarts... • you just get a bunch of cache misses (and a short RDBMS spike) Slide 83: eviction more disappearing data! Slide 84: LRU • can conﬁg memcached to not evict • but, you’re probably doing something wrong if you do this Slide 85: remember, it forgets • it’s just a CACHE Slide 86: expiration • aka, timeouts • memcached.set(key, value, timeout) Slide 87: use expirations or not? Slide 88: 1st school of thought • expirations hide bugs • you should be doing proper invalidations • (aka, deletes) • coherency! Slide 89: school 2 • it’s 3AM and I can’t think anymore • business guy: • “sessions should auto-logout after 30 minutes due to bank security policy” Slide 90: put sessions in memcached? • just a conﬁg change • eg, Ruby on Rails Slide 91: good • can load-balance requests to any web host • don’t touch the RDBMS on every web request Slide 92: bad • could lose a user’s session Slide 93: solution • save sessions to memcached • the ﬁrst time, also save to RDBMS • ideally, asynchronously • on cache miss, restore from RDBMS Slide 94: solution • save sessions to memcached • the ﬁrst time, also save to RDBMS • ideally, asynchronously • on cache miss, restore from RDBMS Slide 95: in the background... • have a job querying the RDBMS • cron job? • the job queries for “old” looking session records in the sessions table memcached • refresh old session records from Slide 96: add vs replace vs set Slide 97: append vs prepend Slide 98: CAS • compare - and - swap Slide 99: incr and decr • no negative numbers Slide 100: queueing • “hey, with those primitives, I could build a queue!” Slide 101: don’t • memcached is lossy • protocol is incorrect for a queue • instead • gearman • beanstalkd • etc Slide 102: cache stampedes • gearman job-unique-id • encode a timestamp in your values • one app node randomly decides to refresh slightly early Slide 103: coherency Slide 104: denormalization • or copies of data Slide 105: example: changing a product price Slide 106: memcached UDF’s • another great tool in your toolbox • on a database trigger, delete stuff from memcached Slide 107: memcached UDF’s • works even if you do UPDATES with fancy WHERE clauses Slide 108: multigets • they are your friend • memcached is fast, but... • imagine 1ms for a get request • 200 serial gets ==> 200ms Slide 109: a resultset loop foreach product in resultset c = memcached.get(product.category_id) do something with c Slide 110: 2 loops for product in resultset multiget_request.append(product.category_id) multiget_response = memcachedClient.multiget( multiget_request) for c in multiget_response do something with c Slide 111: memcached slabber • allocates memory into slabs • it might “learn” the wrong slab sizes • watch eviction stats Slide 112: losing a node • means your RDBMS gets hit Slide 113: replication • simple replication in libmemcached • >= 2x memory cost • only simple verbs • set, get, delete • doesn’t handle ﬂapping nodes Slide 114: persistence Slide 115: things that speak memcached • tokyo tyrant • memcachedb • moxi Slide 116: another day • monitoring & statistics • near caching • moxi Slide 117: thanks!!! • love any feedback • your memcached war stories • your memcached wishlist • steve.yen@northscale.com Slide 118: thanks! photo credits • • • • • • http://ﬂickr.com/photos/davebluedevil/15877348/ http://www.ﬂickr.com/photos/theamarand/2874288064/ http://www.ﬂickr.com/photos/splityarn/3469596708/ http://www.ﬂickr.com/photos/heisnofool/3241930754/ http://www.ﬂickr.com/photos/onourminds/2885704630/ http://www.ﬂickr.com/photos/lunaspin/990825818/