Membase Meetup - San Diego

Notes:	Hide Notes

Slide 1: San Diego Thursday, January 6, 2011 Slide 2: What is Membase? Thursday, January 6, 2011 Slide 3: Membase is a distributed database Application user Web application server Web application server Web application server Membase Servers In the data center On the administrator console 3 Thursday, January 6, 2011 Slide 4: Membase is Simple, Fast, Elastic Five minutes or less to a working cluster • Downloads for Linux and Windows • Start with a single node • One button press joins nodes to a cluster Easy to develop against • Just SET and GET – no schema required • Drop it in. 10,000+ existing applications already “speak membase” (via memcached) • Practically every language and application framework is supported, out of the box Easy to manage • One-click failover and cluster rebalancing • Graphical and programmatic interfaces • Conﬁgurable alerting 4 Thursday, January 6, 2011 Slide 5: Membase is Simple, Fast, Elastic Predictable • “Never keep an application waiting” • Quasi-deterministic latency and throughput Low latency • Built-in Memcached technology High throughput • • • • Multi-threaded Low lock contention Asynchronous wherever possible Automatic write de-duplication 5 Thursday, January 6, 2011 Slide 6: Membase is Simple, Fast, Elastic Zero-downtime elasticity • Spread I/O and data across commodity servers (or VMs) • Consistent performance with linear cost • Dynamic rebalancing of a live cluster All nodes are created equal • No special case nodes • Any node can replace any other node, online • Clone to grow Extensible • Filtered TAP interface provides hook points for external systems (e.g. full-text search, backup, warehouse) • Data bucket – engine API for specialized container types 6 Thursday, January 6, 2011 Slide 7: Built-in Memcached Caching Layer Memcached Membase Database Membase Cache Membase Database Memcached Mode Membase Mode Fact: Membase development team has also contributed over half of the code to the Memcached project. Thursday, January 6, 2011 7 Slide 8: Use Cases Thursday, January 6, 2011 Slide 9: Ad targeting 40 milliseconds to come up with an answer. 3 profiles, real time campaign statistics 2 1 events 9 Thursday, January 6, 2011 profiles, campaigns Slide 10: Search and Gaming Portal Database 10 Thursday, January 6, 2011 Slide 11: Membase Caching at The Knot Jason Sirota Thursday, January 6, 2011 Slide 12: Membase Architecture Thursday, January 6, 2011 Slide 13: Clustering • Underlying cluster functionality based on erlang OTP • Have a custom, vector clock based way of storing and propagating... – Cluster topology – vBucket mapping • Collect statistics from many nodes of the cluster – Identify hot keys, resource utilization 13 Thursday, January 6, 2011 Slide 14: Thursday, January 6, 2011 Slide 15: Thursday, January 6, 2011 Slide 16: Thursday, January 6, 2011 Slide 17: Thursday, January 6, 2011 Slide 18: TAP • A generic, scalable method of streaming mutations from a given server – As data operations arrive, they can be sent to arbitrary TAP receivers • Leverages the existing memcached engine interface, and the non-blocking IO interfaces to send data • Three modes of operation Data Mutations Working set Data Mutations Working set Working set 15 Thursday, January 6, 2011 Slide 19: Membase data ﬂow – under the hood SET request arrives at KEY’s master server 12 SET acknowledgement returned to application 3 Listener-Sender membase storage engine 3 4 RAM Disk Disk Disk Disk Disk Disk Replica Server 1 for KEY Thursday, January 6, 2011 Master server for KEY Replica Server 2 for KEY 16 Slide 20: Clients, nodes and other nodes Client moxi + Client port 11210 memcached operations port 11211 memcached operations REST/comet cluster topology and vbucket map memcached operations moxi ns_server membase ns_server (memcached + membase engine) TAP memcached operations with tap commands vbucketmigrator 17 Thursday, January 6, 2011 Slide 21: Data buckets are secure membase “slices” Application user Web application server Bucket 1 Bucket 2 Aggregate Cluster Memory and Disk Capacity Membase data servers In the data center Thursday, January 6, 2011 On the administrator console 18 Slide 22: vBucket mapping 19 Thursday, January 6, 2011 Slide 23: Disk > Memory Bucket Conﬁguration Dataset may have many items infrequently accessed. However, memcached has different behavior (LRU) than wanted with membase. Still, traditional (most) RDBMS implementations are not 100% correct for us either. The speed of a miss is very, very important. memory quota mem_high_wat mem_low_wat 20 Thursday, January 6, 2011 Slide 24: Membase Demo Thursday, January 6, 2011 Slide 25: Key-Value Patterns Thursday, January 6, 2011 Slide 26: Key-Value 23 Thursday, January 6, 2011 Slide 27: Key-Value Items have: Operations include: Key Get/Set Value Increment/Decrement Expiration Append/Prepend Flags CAS (more on this later) 23 Thursday, January 6, 2011 Slide 28: Key-Value 23 Thursday, January 6, 2011 Slide 29: Key-Value (with a replica Image courtesy http://www.flickr.com/photos/brenda-starr/3509344100/sizes/m/in/photostream/ ) 23 Thursday, January 6, 2011 Slide 30: Membase Datatypes 24 Thursday, January 6, 2011 Slide 31: Membase Datatypes • byte[] – Does your data have 1s and 0s? 24 Thursday, January 6, 2011 Slide 32: Membase Datatypes • byte[] – Does your data have 1s and 0s? “Any customer can have a car painted any colour that he wants so long as it is black.” 24 Thursday, January 6, 2011 Slide 33: Membase Datatypes • byte[] – Does your data have 1s and 0s? • Items do have flags – Many clients use flags – Data type options • Google protobuf • Thrift • Avro Thursday, January 6, 2011 “Any customer can have a car painted any colour that he wants so long as it is black.” 24 Slide 34: Transactions • Lock == slow me down • CAS operations – Optimistic locking User 1 Su User 2 • Very useful with complex datatypes – Imagine two clients trying to update a complex item • You’re likely using CAS already... if you use a CPU 25 Thursday, January 6, 2011 Fa il! cc es s Slide 35: Common Use: Sessions • Web user sessions – Highly read, less writes in many case – Protocol advantage of memcached • Options already for PHP, Ruby and Java • Application state – Not necessarily “entity” style things – May be appropriate for a “cache” pool 26 Thursday, January 6, 2011 Slide 36: Common Use (cache): Rate Limiting • Want to provide API calls into the system – Twitter search – Google search services Your Users h uc ¡O ! • Use the atomic increment – Set an item with a unique ID – Upon API request, increment and check • HTTP 420: go away and come back later Your App 27 Thursday, January 6, 2011 Slide 37: Looking Ahead: NodeCode Thursday, January 6, 2011 Slide 38: NodeCode – Motivation Beyond key-value • Indexing/Range Queries • Advanced Data Structures • Sub-object direct manipulation Validation and In-ﬂight transformation • Block mutations failing validation • Enrich or transform objects Connectors (Integrate easily with other systems) • Solr • Hadoop • MySQL 29 Thursday, January 6, 2011 Slide 39: Thursday, January 6, 2011 Slide 40: Q&A 31 Thursday, January 6, 2011 Slide 41: Attributions • • • http://commons.wikimedia.org/wiki/File:Flag_of_China.png http://commons.wikimedia.org/wiki/File:Flag_of_South_Korea.svg http://commons.wikimedia.org/wiki/File:Flag_of_Japan.svg 32 Thursday, January 6, 2011 Slide 42: Thursday, January 6, 2011