Cloud Computing Primer: Using cloud computing tools in your museum

Notes:	Hide Notes

Slide 1: Cloud Computing Primer: Steps for using the Cloud in Your Museum Ari Davidow – jwa.org Charles Moad – imamuseum.org Robert Stein - imamuseum.org Slide 2: wikipedia on cloud computing via wordle.net Slide 3: CLOUD COMPUTING UTILITY COMPUTING CLOUD APPLICATIONS Slide 4: Cloud Applications ... eliminate the need to install and run the application on the customer's own computer, thus alleviating the burden of software maintenance, ongoing operation, and support. Slide 5: Cloud Applications Slide 6: Utility Computing … a style of computing where scalable and elastic IT-related capabilities are provided as a service to external customers using Internet technologies. Slide 7: Utility Computing Slide 8: Buzz Worthy Search Trends for “cloud computing” Via Google Search Trends Slide 9: 69% of Americans use cloud computing services Source “Cloud Computing Gains in Currency”, Pew Research, May 2008 http://pewresearch.org/pubs/948/cloud-computing-gains-in-currency Slide 10: Gartner’s Hype Cycle for 2009 image courtesy of gartner.com Slide 11: 21% of companies are piloting SaaS applications up from 18% last year – Forrester, Feb 2009 Slide 12: In Forrester’s List of the Top 15 Technology Trends Slide 13: State of Cloud Computing “ Forrester feels that cloud computing is one of the Top 15 Technology Trends and that it warrants investment now so you can gain the experience necessary to take advantage of it in its many forms to transform your organization into a more efficient and responsive service provider to the business -Forrester, October 13, 2009, http://blogs.forrester.com/it_infrastructure/2009/10/cloud-computing-belongs-on-your-3year-roadmap.html Slide 14: Gartner’s #1 Strategic Technology Area for 2010 Slide 15: State of Cloud Computing “ Cloud computing isn't going to be vapor much longer… It's complicated, poses security risks, and computing technology companies are latching onto the buzzword in droves, but the phenomenon should be taken seriously… -Gartner - October 20,2009 http://news.cnet.com/8301-30685_3-10378782-264.html Slide 16: Concerns about SaaS Slide 17: Pros of Cloud Computing Fast Deployment Lower cost / No Capital Expense Reduced IT maintenance Elastic and Unlimited Scalability Energy Efficiency Reliability (Service & Data) Better Resource Utilization Slide 18: Cons of Cloud Computing Information Security Physical Security Long Term Offline Storage Bandwidth Bottleneck Potential Vendor Lock-in Lack of control during downtime Slide 19: Amazon Web Services (AWS) Overview Virtual Private Cloud CloudFront Content Delivery Network Relational Database Service (RDS) Elastic Block Store (EBS) Elastic MapReduce Elastic Compute Cloud (EC2) Simple Storage Service (S3) SimpleDB Simple Queue Service (SQS) Amazon Web Services (AWS) Infrastructure Services Slide 20: How to make choices about Cloud Computing - What sort of security requirements fit your data? - How granular is the information you’re working with? (documents, images, video?) - Where are your likely performance bottlenecks? (compute, bandwith, latency) - What is your IT staff like? (small but flexible, large) Slide 21: Jungle Disk Slide 22: Jungle Disk - $20 / www.jungledisk.com Slide 23: Requires - Amazon S3 account, and the requisite keys: - JungleDisk software installed Slide 24: Usage - Backs up at scheduled times - Can back up more than one machine, or to more than one backup set - The first backup may take days – or longer. No problem. The software gracefully goes to sleep when you shut down or hibernate; resumes upon waking until done - Can retrieve files using drag/drop interface using pull-down to set the date of the view from which you wish to retrieve (i.e., let me see the files as they were on July 7, 2008). - Retrieves files gracefully and quickly Slide 25: Converse Example Slide 26: IMA’s SAN - IMA Purchased 32TB of EMC SAN in 2006 - 16TB local and 16TB at an offsite colocation facility - Due to growth in Collection Photography, Video, and Conservation Imagery that space is all but full! Slide 27: IMA’s SAN - Benchmark Growth Rate - Total Current Size – Initial Size / 36 months - Ballpark Rate of 142 GB/month - Yielding 13.9TB estimated in the next 4 years Slide 28: IMA’s SAN - 16TB Onsite + 14TB AWS = $164,544 - 16TB Onsite + 16TB Colo = $94,200 $140,000 $120,000 $100,000 $80,000 $60,000 $40,000 $20,000 $0 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 Slide 29: Hidden Costs - DS3+Colo = $96,000 (3 yr commitment) - Maintenance and administration of servers (2 FTE’s) - As replicated backup for super large filesystems time for restore would be huge! Slide 30: Tools - S3Fox - AWS Console - Elastic Fox Slide 31: Moving Drupal to the Cloud: Step by step Slide 32: Introducing the EC2 Console Slide 33: Creating a Key Pair Slide 34: Creating a Security Group Slide 35: Selecting a Starter AMI Slide 36: Launching an AMI Slide 37: Connecting to Your EC2 Instance Slide 38: Creating an EBS Volume Slide 39: Configuring Apache and MySQL Slide 40: Setting up Drupal Slide 41: Bundling an AMI Slide 42: ami-764bab1f Slide 43: Fedora for DAM Slide 44: Fedora as a testbed on AWS Project Goals: External vendor to create least-possible Fedora instance to enable preservation work Slide 45: Summary - Create on AWS and hand over instance when done - When dev site is completed, create “Amazon Machine Instance” and check into Subversion - Document installation and everything else in wiki - We create new instance from checked-out AMI - This ensures that we have maintainable code that we can get up and running, before developer moves on Slide 46: What we did - Original server created using developer’s favorite Linux - We use CentOS, so when we checked out the AMI, we recreated running under CentOS, bundled new AMI to S3 - AMIs can be independent of the underlying OS Slide 47: Bugs - Our repository, which consists of lots of very large files, uses a unix filesystem called XFS - XFS supports very large volumes better the usual filesystem, and supports real-time snapshotting of huge file-systems - AWS updated CentOS and broke XFS - We (actually, our webmaster) rebuilt kernel to work around AWS CentOS bug Slide 48: Other Gotchas - An EC2 instance doesn’t preserve state - When you restart, it restarts from scratch - All config changes, anything else that was done and saved to the previous instance is gone - So, you use EBS, which acts something like a network drive (think NetApps) - You purchase blocks of EBS space at a time, but it is cheaper than S3 per GB/Month - This is different from S3 storage where you pay only for what you consume Slide 49: AMIs - Amazon Machine Instance - Sort of like a “ghost”ed server image - Amazon (and others) provide lots of AMIs to work with - AMIs can be public or private - You can use different AMIs on different servers in your AWS setup - “bundling” is the AWS term for saving that AMI with your modifications for future use. - We store AMIs on S3; could also use EBS Slide 51: Lessons Learned - We liked AWS so much, and saved so much money, that we have now moved all of our web services to AWS. - Our website used to cost us $1200/ mo. It has added about $450/mo to what we already pay for the Fedora instance – about $900/mo total. Slide 54: Rightscale – basic services free www.rightscale.com Slide 57: AWS Infrastructure Virtual Private Cloud CloudFront Content Delivery Network Relational Database Service (RDS) Elastic Block Store (EBS) Elastic MapReduce Elastic Compute Cloud (EC2) Simple Storage Service (S3) SimpleDB Simple Queue Service (SQS) Amazon Web Services (AWS) Infrastructure Services Slide 58: 4. Videos download from S3/CDN ArtBabble Videos Wowza Streaming Server 3. Video stream WWW Images ArtBabble.org Apache/MySQL EBS S3 CloudFront EC2 1. Web request 2. Images served directly from S3/CDN Slide 59: Scalabble Slide 60: Video Processing Slide 61: Total Monthly AWS Slide 62: Monthly Bill for ArtBabble.org Web Server Slide 63: Monthly Bill for Wowza Video Server(s) Slide 64: AWS Bill - CloudFront Slide 65: AWS Bill – EC2 Slide 66: AWS Bill – S3 Slide 67: AWS Bill - Wowza Slide 68: The Numbers (so far) - 150,000 video views (168k visits / 576k pages) - 81,000 note clicks - 1:3 of the notes expanded - 22,400 views of “Behind the Babble” - 25,015 views of most popular YouTube video posted Feb. 1st, 2008 - 5,000 registered users - 44% signed up using OpenID (but didn’t realize it) Slide 69: The Numbers (so far) for geeks - 112 hours of video processed - 525 videos - 1700 instance hours - At a cost of ~$0.65 per video - April 1st – October 30th - 167,000 visitors - April 1st – September 31st - 1.1TB of web data transferred out - 1.25TB of video streamed - 11 Mbps average transfer on embedded videos - IMA just upgraded to 5Mb pipe Fall ‘08 - At a transfer cost of $250 - At a transfer cost of $200 - From 166 countries Slide 70: QUESTIONS?