Slide 1: Moving Pictures
Implementing Video on Flickr
Cal Henderson
Slide 2: Hello
Web 2.0 Expo NYC, 17th September 2008
2
Slide 3: Flickr
• Large scale kitten sharing website • Started 2003, launched 2004
– 5 years old this december
• Almost 3 billion photos
Web 2.0 Expo NYC, 17th September 2008
3
Slide 4: Enter: Video
• Video was added this year
– Launched April 2008
• Many hundreds of thousands of videos uploaded • Many millions of playbacks
Web 2.0 Expo NYC, 17th September 2008 4
Slide 5: Web 2.0 Expo NYC, 17th September 2008
5
Slide 6: Web 2.0 Expo NYC, 17th September 2008
6
Slide 7: Web 2.0 Expo NYC, 17th September 2008
7
Slide 8: Web 2.0 Expo NYC, 17th September 2008
8
Slide 9: Web 2.0 Expo NYC, 17th September 2008
9
Slide 10: “Video? That’s just like photos!”
-Me, before Flickr Video
Web 2.0 Expo NYC, 17th September 2008
10
Slide 11: 12 4 Steps
• 4 main tasks
– Uploading – Transcoding – Storage – Serving & Playback
Web 2.0 Expo NYC, 17th September 2008
11
Slide 12: 1. Upload
Web 2.0 Expo NYC, 17th September 2008 12
Slide 13: Simple upload
• Web forms
– Just like any other file
<form action="/uploadify/" method="post" enctype="multipart/form-data"> <input type="file" name="fred" /> <input type="submit" value="Go!" /> </form>
• But files are large / huge
Web 2.0 Expo NYC, 17th September 2008 13
Slide 14: Issues
• Two components for uploading:
– Sending from the client – Receiving on the server
• Both of these present problems
Web 2.0 Expo NYC, 17th September 2008
14
Slide 15: Sending from the client
• Multiple options
– Simple form – Flash – Desktop app
Web 2.0 Expo NYC, 17th September 2008
15
Slide 16: Simple Forms
• Pros
– Very easy to implement – Works on every browser out of the box
• Cons
– Upload progress is harder – ‘Slow’ – Select a single file at once
Web 2.0 Expo NYC, 17th September 2008 16
Slide 17: Flash
• Pros
– Upload progress is easy – ‘Fast’ – Multi select of files
• Cons
– Harder to implement – Flash isn’t quite ubiquitous
Web 2.0 Expo NYC, 17th September 2008 17
Slide 18: Desktop App
• Pros
– Upload progress is easy – ‘Very fast’ – Multi select of files – Drag and drop
• Cons
– Hard to develop – Hard to deploy (relative to the web)
Web 2.0 Expo NYC, 17th September 2008 18
Slide 19: Making Progress
• Upload progress
– Not impossible with plain forms – Just need to be able to query the upload progress via AJAX
• Multiple machines
– The VIP issue – Enter Perlbal
Web 2.0 Expo NYC, 17th September 2008
19
Slide 20: Making Progress
1. Browser starts upload
Web 1
Browser
Load balancer
2. Web server broadcasts progress via UDP
3. Browser queries progress via AJX call
Web 2
Web 2.0 Expo NYC, 17th September 2008
20
Slide 21: Receiving on the server
• File uploads are slow
– Much slower than serving pages
• Apache processes are heavy
– Waste of resources
• Use a poll based server (like jetty)
Web 2.0 Expo NYC, 17th September 2008
21
Slide 22: Receiving on the server
• Or, use a buffering layer
– Perlbal is great for this
Slow
Browser Buffer
Fast
Server
• Or a lightweight Apache
– E.g. w/ mod_proxy
Web 2.0 Expo NYC, 17th September 2008 22
Slide 23: But wait
• It’s not just the first step that’s slow • Moving files around between servers is slow
– Do it out of band
• Asynchronous jobs are in order anyway
– Do it!
Web 2.0 Expo NYC, 17th September 2008 23
Slide 24: 2. Transcode
Web 2.0 Expo NYC, 17th September 2008 24
Slide 25: Transcode?
• Why transcode at all? • Input comes from many sources
– Point & shoots – DV Cams – Mobile devices – Video editing software
• All in different formats
Web 2.0 Expo NYC, 17th September 2008 25
Slide 26: So many formats
• But very few of these formats can be played back cross platform
– Without special software or hardware
• Formats are designed to do one thing well
– They don’t always manage even that
• Transcoding puts all videos on an equal footing
Web 2.0 Expo NYC, 17th September 2008 26
Slide 27: Video file basics
• Most file types are really just containers
– MOV, FLV, AVI
• The data inside can be in multiple formats
– We call these codecs (encoder + decoder)
• Files contains multiple ‘streams’
– Both audio and video
Web 2.0 Expo NYC, 17th September 2008 27
Slide 28: Interleave
• Audio and video are often interleaved
– Hence AVI
• A video file looks like this:
– Headers – Video chunk – Audio chunk – Video chunk – Audio chunk – Etc
Web 2.0 Expo NYC, 17th September 2008 28
Slide 29: Compress
• Raw video files are huge
– A bitmap for every frame – Rarely used, even in post production
• At 30 fps, that gets crazy pretty quickly • We don’t need to store every frame
– Static backgrounds don’t change (much) between frames
Web 2.0 Expo NYC, 17th September 2008 29
Slide 30: Compresssss
• Intraframe
– Treat each frame as a picture – Compress it (just like JPEG) – DCT (Discrete Cosine Transform)
• Interframe
– Store the differences between frames – Treat the pixels as a 3D array to be compressed
Web 2.0 Expo NYC, 17th September 2008 30
Slide 31: The IPB
• Three frame types: I, P & B • Intra coded pictures
– A full raw frame
• Predicted pictures
– Based on a single reference frame
• Bi-predictive pictures
– Based on two or more reference frames
Web 2.0 Expo NYC, 17th September 2008 31
Slide 32: IPBIPBIPBIPB
• Reference frames may be I, P or B • P & B frames may contain a mix of image data and motion vector displacements • I frames require the most bits
– Then P frames – Then B frames
Web 2.0 Expo NYC, 17th September 2008 32
Slide 33: Bad terminology
• We should really say picture
– (Not frame) – Because of interlacing
• Really, we encode fields not frames
– Picture is the general term
• And H.264 contains ‘slices’
– Sub-regions of the field – ‘Macroblocks’ & ‘Artifacts’
Web 2.0 Expo NYC, 17th September 2008 33
Slide 34: I-Frames
• Also called Key Frames • Allow easy random seeking • Twice a second for Digital TV & DVDs • More widely spaced online
Web 2.0 Expo NYC, 17th September 2008
34
Slide 35: Seekable
Web 2.0 Expo NYC, 17th September 2008
35
Slide 36: Seekable
Web 2.0 Expo NYC, 17th September 2008
36
Slide 37: Oh, and audio too
• We can worry less about this
– Older problem, well solved
• MP3 is pretty good
– Who cares how it works?
• Syncing is the only issue
– Presentation Time Stamps (PTS) and Decode Time Stamps (DTS) in MPEG-2
Web 2.0 Expo NYC, 17th September 2008 37
Slide 38: Flash! Woah-oh!
The big question:
Flash?
Web 2.0 Expo NYC, 17th September 2008 38
Slide 39: Non-flash sites
• QuickTime • Windows Media • This is gradually disappearing • Flash player is ubiquitous • Compression is good enough • Interactive too • But no 3D/VR as with QuickTime :(
Web 2.0 Expo NYC, 17th September 2008 39
Slide 40: The Flash Player
• Flash Player 6
– March 2002
• Video: Sorenson Spark (H.263) • Audio: MP3
– Or ADPCM / Uncompressed – Or Nellymoser Asao
Web 2.0 Expo NYC, 17th September 2008 40
Slide 41: Second Generation
• Flash Player 7
– August 2005
• Video: On2 TrueMotion VP6 • Audio: MP3
Web 2.0 Expo NYC, 17th September 2008
41
Slide 42: The hot shit
• Flash Player 9 (update 3)
– December 2007
• Video: H.264 (MPEG-4 Part 10)
– w/ container formats from MPEG-4 Part 14
• Audio: AAC (MPEG-4 Part 3) • Plus 3GPP Timed Text (MPEG-4 Part 17)
Web 2.0 Expo NYC, 17th September 2008 42
Slide 43: TrueMotion VP6
• Proprietary • Reasonable compression • Created by On2
– Patented – Probably illegal for GPL code
• YouTube uses it for lower quality and old streams
Web 2.0 Expo NYC, 17th September 2008 43
Slide 44: H.264
• Not proprietary • Good compression • MPEG Standard
– – – – Open, but patented Patent licenses from the MPEG LA Unclear how this applies to (L)GPL code But probably badly
• YouTube using it for higher quality streams • iPhones and AppleTV
Web 2.0 Expo NYC, 17th September 2008 44
Slide 45: Software
• Open source transcode tools • FFmepg
– libavcodec for VP6 – Probably illegal – dubious – Also pretty shoddy – Can only decode H.264
Web 2.0 Expo NYC, 17th September 2008
45
Slide 46: More software
• MEncoder
– libmpcodecs uses libavcodec
• VLC
– libvlc uses libavcodec
• So basically the same
– Different muxing, same codecs
Web 2.0 Expo NYC, 17th September 2008 46
Slide 47: Free H.264?
• Unfortunately, not really • x264 is the only usable one
– It’s pretty good – MEncoder can use it – Still limited in options at this point – Again, dubiously legal
Web 2.0 Expo NYC, 17th September 2008 47
Slide 48: Non-free tools
• Flash encoder
– Not automatable
• On2 FlixEngine
– Creators of VP6 – Windows or Linux – Some support for H.264
• Rhozet Carbon Coder
– The new hot shit – Good H.264 support – Windows
Web 2.0 Expo NYC, 17th September 2008 48
Slide 49: Choices
• • • • • Video codec Resolution Bitrate (VBR, CBR) Keyframes Audio codec
– Channels – Bit depth – Sampling rate
Web 2.0 Expo NYC, 17th September 2008 49
Slide 50: Doing it at scale
• Not really a problem • Very easily parallelizable • Amazon EC2 is awesome here
– Exactly what it was design for – Grow/shrink as needed – But, per-CPU software licensing
Web 2.0 Expo NYC, 17th September 2008
50
Slide 51: 3. Store
Web 2.0 Expo NYC, 17th September 2008 51
Slide 52: Easy!
• Really, just like photos
– But with bigger files
• Same disk layout as any other serving • But the serving part is slower
Web 2.0 Expo NYC, 17th September 2008
52
Slide 53: But..
• Remember the files are huge • Operations take time
– More likely to fail halfway through – Checksums are your friend
• Do it all asynchronously
Web 2.0 Expo NYC, 17th September 2008 53
Slide 54: 4.
Serve & Playback
Web 2.0 Expo NYC, 17th September 2008 54
Slide 55: The choice
Streaming
vs
Progressive
Web 2.0 Expo NYC, 17th September 2008 55
Slide 56: Streaming
• Pros
– Easily seekable – Live feeds
• Cons
– Special server software – Slower to start – Firewall troubles
Web 2.0 Expo NYC, 17th September 2008 56
Slide 57: Progressive download
• Pros
– Just use a web server – Play offline – Firewall/proxy friendly
• Cons
– Harder to seek ahead (but not impossible)
Web 2.0 Expo NYC, 17th September 2008
57
Slide 58: Streaming tech
• Non flash stuff
– We’ll ignore that – You’re using flash, right?
• RTMP
– Real-time Messaging Protocol – Proprietary (thanks Adobe!)
Web 2.0 Expo NYC, 17th September 2008
58
Slide 59: RTMP
• RTMP - Raw TCP socket stuffs • RTMPT – RTMP tunneled over HHTP
– For firewalls, etc
• Flash Media Server
– previously Flash Communication Server
• Wowza Pro
– $1000/server
Web 2.0 Expo NYC, 17th September 2008 59
Slide 60: Open source
• It’s not all bad • Red5
– Java implementation of RTMP server – Mostly feature complete – Beta quality, but usable in production – Facebook
Web 2.0 Expo NYC, 17th September 2008 60
Slide 61: Progressive
• Used by the majority of large sites • Very simple! • Seekable with server support
Web 2.0 Expo NYC, 17th September 2008
61
Slide 62: Seeking
• Serve the FLV starting at a different point • Just add a simple FLV preamble before seeking into the file • Simple to do in PHP, Perl, etc • mod_flvx for Apache • mod_secdownload for lighttpd
Web 2.0 Expo NYC, 17th September 2008 62
Slide 63: 5.
Other considerations
Web 2.0 Expo NYC, 17th September 2008 63
Slide 64: Review
• Videos are slow • Expensive to review • Review grids
– Doesn’t cover audio
Web 2.0 Expo NYC, 17th September 2008
64
Slide 65: Web 2.0 Expo NYC, 17th September 2008
65
Slide 66: Not enough?
• Social tools are useful here
Web 2.0 Expo NYC, 17th September 2008
66
Slide 67: Summary
Web 2.0 Expo NYC, 17th September 2008 67
Slide 68: Summing up
• Flash makes sense
– For uploading too
• H.264 is probably your best bet today • Transcoding software still costs money
– Unless you’re willing to take on the risk
• Progressive download is basically awesome
Web 2.0 Expo NYC, 17th September 2008 68
Slide 69: The end!
Web 2.0 Expo NYC, 17th September 2008
69
Slide 70: Awesome!
These slides are available online: iamcal.com/talks/
Web 2.0 Expo NYC, 17th September 2008
70
