Hide Notes 

Slide 1: Software Measurement Software Economics 2010 lecture 3: metrics in organizations give your feedback: twitter with #softmetrics
Slide 2: Mark Kofman  Co-founder at PROGRAMETER − Metrics tracking kit for software development  Interests: software quality, metrics, startups, entrepreneurship, distributed and multi-culture teams Contact details: − − −  Email: mark.kofman@programeter.com Skype: kofman Twitter: @markkofman give your feedback: twitter with #softmetrics
Slide 3: Agenda • Applications of software metrics • Why introducing metrics is so hard? • Lesson from the battlefield • Common mistakes • Steps to introduce metrics in organization give your feedback: twitter with #softmetrics
Slide 4: Agenda • Applications of software metrics • Why introducing metrics is so hard? • Lesson from the battlefield • Common mistakes • Steps to introduce metrics in organization give your feedback: twitter with #softmetrics
Slide 5: Purpose of Metrics in Software Development  Quality assurance − Code quality, defects, application performance How efficient is the organization, team or individual member faster react to changes in the context  Performance measurement −  Process improvement − give your feedback: twitter with #softmetrics
Slide 6: Quality Assurance  Quality cannot be measured directly → derived from other metrics give your feedback: twitter with #softmetrics
Slide 7: Quality Assurance Funtionality Time  Cost One can only fix any two → cannot put lots of different qualities without affecting both budget and speed give your feedback: twitter with #softmetrics
Slide 8: Quality Assurance  What happens if we overemphasize − − − Quality of code Performance of the software User satisfaction  Focus on one ultimately important quality and minimize drawbacks for others − − − Start-ups: low-cost or development speed Airplanes: bug free Netbooks: small and light give your feedback: twitter with #softmetrics
Slide 9: QA: User Satisfaction  Surveys − − − User is able to operate software without assistance User understands what software does User likes software Number of defects per iteration Number of delivered features  Software works and does the right thing − − give your feedback: twitter with #softmetrics
Slide 10: QA: Application Performance  Application average response time under particular load Throughput – number of concurrent requests Memory and CPU usage under particular load Scalability − −    How metrics change if hardware is improved How metrics change if software is distributed over several systems give your feedback: twitter with #softmetrics
Slide 11: QA: Code Quality  Well documented − Comment density – ratio of comment lines to all lines Distance from Main Sequence → abstractness and instability Depth of inheritance tree, number of children, coupling, complexity, ... number of very similar code blocks give your feedback: twitter with #softmetrics  Well designed − −  Code duplication −
Slide 12: QA: Defect Statistics       How many defects we have found? How many defects escaped QA? How many are open, assigned? What is the ratio of defects to tasks? Defect fixing effort estimation Toolkit: − Jira, Mantis, Trac, Bugzilla, Programeter, ... give your feedback: twitter with #softmetrics
Slide 13: QA: Code and Design Improvement   Code compliance to standards and conventions Identify design smells − Rigidity, fragility, immobility, viscosity Open close principle, dependency inversion principle, ... Checkstyle, FindBugs, PMD, ckjm, Eclipse Metrics, ... give your feedback: twitter with #softmetrics  Design compliance to principles −  Toolkit: −
Slide 14: QA: Performance Tuning   Memory and CPU usage Performance − Average response time, number of concurrent users Most used methods, execution time YourKit, JProfiler, JMeter, JAMon, ...  Analyze user behavioral patterns −  Toolkit: − give your feedback: twitter with #softmetrics
Slide 15: QA: Value Estimation    How much value does particular feature bring? Is it worth developing/buying? Which features bring the most value? give your feedback: twitter with #softmetrics
Slide 16: Performance Measurement  Performance management is a set of activities to ensure that goals are consistently being met in an effective and efficient manner. Focus employees' attention on what matters most to success “if you don’t measure results, you can’t tell success from failure and thus you can’t claim or reward success or avoid unintentionally rewarding failure,” give your feedback: twitter with #softmetrics  
Slide 17: PM: Focusing on Important Things  Use metrics to find out components that − − Change the most (e.g. churned files) Are used the most (e.g. logs) Design for easier change Improve performance where it is crucial Programeter, JAMon, FishEye, SvnPlot  Drive optimization effort there − −  Toolkit: − give your feedback: twitter with #softmetrics
Slide 18: PM: Time Control  How do I (or my personnel) spend days? − − − How much time is spent on coding, communication and procrastination? How much is “billable” time? Are there any trends?  Toolkit: − RescueTime, Toggl give your feedback: twitter with #softmetrics
Slide 19: PM: Burn-down Chart   How much work remains Will we hit the deadline? 10 8 Ac tu al Re ma ini ng Points 6 4 2 0 1.Sept 3.Sept Pl an ne d 5.Sept 7.Sept 9.Sept give your feedback: twitter with #softmetrics
Slide 20: PM: Burn-up Chart  How much work is done 15 12 aining Rem  Scope  Deadline estimation Points 9 6 3 0 1.Sept d lete p om C give your feedback: twitter with #softmetrics 3.Sept 5.Sept 7.Sept
Slide 21: PM:Team Performance  Developer's performance − − − − Velocity – number of completed features Churned lines of code – number of added, modified or deleted lines of code Coding efficiency – ratio of developer's code to churned lines of code Code maturity – how much of developer's code gets rewritten by peers Programeter, Scrum Sprint Monitor give your feedback: twitter with #softmetrics  Toolkit: −
Slide 22: PM: Task and Feature Tracking  Project estimation − Size → number of specialists, time and budget estimates How much is completed, how much is left Are we on time? When we will deliver? MS Excel, MS Project, Gantt charts, Jira, Pivotal Tracker, Trac, Mingle, ... give your feedback: twitter with #softmetrics  Project tracking − − −  Toolkit: −
Slide 23: PM: Developer Benchmarking give your feedback: twitter with #softmetrics
Slide 24: Developer Benchmarking VS give your feedback: twitter with #softmetrics
Slide 25: Developer Benchmarking VS give your feedback: twitter with #softmetrics
Slide 26: PM: Expertise Management  What tools, technologies and libraries developers use − − − Programming languages Library dependencies (Ant+Ivy, Maven, ...) Code analysis: which external components developer uses in his code   Better team formation and expertise sharing Toolkit: − ... give your feedback: twitter with #softmetrics
Slide 27: Process Improvement  Series of actions taken to identify, analyze and improve existing processes within an organization to meet new goals and objectives. Metrics are used to identify bottlenecks  give your feedback: twitter with #softmetrics
Slide 28: PI: Velocity  How much work can a team complete per iteration Completed Points 15 10 5 0 1 2 3 4 5 6 7 8 9 Iterations give your feedback: twitter with #softmetrics
Slide 29: PI: Knowledge Sharing  Risky knowledge − − Percentage of components where developers posses only unique knowledge Pair programming, developer rotation Ratio of shared knowledge to total knowledge   Team friendliness of developer − − 1 → all knowledge is shared, 0 → only unique Indicator of a culture: “my code – don't touch it” Programeter give your feedback: twitter with #softmetrics  Toolkit: −
Slide 30: PI: Knowledge Sharing give your feedback: twitter with #softmetrics
Slide 31: PI: Return on Investment (ROI)  Attractiveness of an investment calculated by financial performance of invested money (Vfinal – Vinit) / Vinit − − Vinit – initial value of an investment Vfinal – final value of an investment   Example: bank savings account → 4% Payback Period – length of time when investment is returned give your feedback: twitter with #softmetrics
Slide 32: PI: Total Cost of Ownership  Total cost of procuring, using, managing and disposing of an asset over its useful life Example: TCO of a car much higher than its price! − − − − −  Price Gasoline Technical reviews Insurance … give your feedback: twitter with #softmetrics
Slide 33: Example: E-Print  Feature A: support of doc, rtf, docx, txt, odt, pdf documents − − Development cost: 2.35K EUR Infrastructure cost: 40 EUR per month Development cost: 350 EUR Infrastructure cost: No additional costs  Feature B: support of pdf documents − −  Application is not validated yet → cut the costs → implement feature B give your feedback: twitter with #softmetrics
Slide 34: Example: E-Print   Initial application 3 year revenue: 10K EUR Additional development costing 3K EUR would increase 3 year revenue by 4K EUR ROI = (4K – 3K) / 3K = 33%  give your feedback: twitter with #softmetrics
Slide 35: TCO vs ROI  ROI → Making go or not go decisions − − Project drives increase in revenue “Automation of customer relations” Build or buy, in-house or outsource, ... Swapping one system with another “Migrating from one CRM system to another”  TCO → Evaluating possible decisions − − − give your feedback: twitter with #softmetrics
Slide 36: One Metric – Different Benefits   Code complexity Developer ← code quality − Complex code is not comprehensible Complex code requires more testing effort Complex code leads to higher maintenance costs  Project manager ← testing effort −  Executive ← maintenance costs − give your feedback: twitter with #softmetrics
Slide 37: Metrics at Specialist Level LOC WMC DIT LCOM DIT Ce Ca NOC RFC Abstractnes s Instability Friendliness Churned LOC Churn Count # Requirements # Bugs Scalabilit y # Defects Points Knowledg e Velocity Response Time give your feedback: twitter with #softmetrics
Slide 38: Specialist Level → Project Level Project Size Design Quality LOC Points LCOM RFC DIT DIT # Requirements Abstractnes s give your feedback: twitter with #softmetrics
Slide 39: Metrics at Project Level Effort Productivity Design Quality Project Size Architecture Quality Knowledge Sharing Team Efficiency Customer Satisfaction Application Performance Correctness give your feedback: twitter with #softmetrics
Slide 40: Project Level → Organization Level ROI TCO Personnel Qualities Effort Productivity Productivity Design Quality Team Efficiency Project Size give your feedback: twitter with #softmetrics
Slide 41: Metrics at Organization Level ROI Personnel Qualities TCO Information Productivity Knowledge Capital give your feedback: twitter with #softmetrics
Slide 42: Agenda • Applications of software metrics • Why introducing metrics is so hard? • Lesson from the battlefield • Common mistakes • Steps to introduce metrics in organization give your feedback: twitter with #softmetrics
Slide 43: We don't like to be measured  People are afraid that data will be used against them Software developers are not different  give your feedback: twitter with #softmetrics
Slide 44: Prima donna effect   Great developers is a scarce resource In most organizations developers are treated as prima donnas allowing them to dictate development processes give your feedback: twitter with #softmetrics
Slide 45: Too Much Attention to Code Metrics  In 19XXs code metrics got too much attention of researchers and university teachers Code metrics are cool, but not enough to give you a full picture about your software project or organization Lines of Code most popular metric When manager hears “software metrics” she often feels that this is a developers playground    give your feedback: twitter with #softmetrics
Slide 46: There are no industry standards for software measurement    Software industry is still young Historically metrics had “bad smell” New trendy process appears every 10 years − It was RUP in 90s, then extreme programming, now Scrum and agile processes  If process changes every 10 years it is hard to talk about standard metrics Open Initiative: −  sdlcmetrics.org, you are welcome to contribute give your feedback: twitter with #softmetrics
Slide 47: Agenda • Applications of software metrics • Why introducing metrics is so hard? • Lesson from the battlefield • Common mistakes • Steps to introduce metrics in organization give your feedback: twitter with #softmetrics
Slide 48: How we helped introducing metrics in “Brown Solutions”  Brown Solutions – not a real name − International company, about 1,000 employees, young company, young managers, success highly depends on software product quality. Introduce metrics into the organization, automate collection of the metrics, assure needed data is gathered and is accurate  Metrics team role − give your feedback: twitter with #softmetrics
Slide 49: “Brown Solutions”, part 1  I got a call “Mark, we need metrics. CEO has setup us a goal to have it ready by the end of this year. Let's meet tomorrow” We meet. −  “Do you know which metrics you want to collect?  NO. But we have a list of 30 metrics somebody has gathered” Mmmm...you know, metrics are important. Mmm...I guess everybody give your feedback: twitter with #softmetrics − − “Why do you need metrics”  Who will be using the metrics? 
Slide 50: “Brown Solutions”, part 2  Now I am also in the “Mmmm” state. What should we do? Where should we start? We start building the prototype based on the list of metrics provided to gather feedback Prototype implementing 5 metrics is ready. We go for feedback. − −   “Mmm....I have no time this week. Let's talk next month” “Wow. It looks cool. Can you also implement this metric for me.” give your feedback: twitter with #softmetrics
Slide 51: “Brown Solutions”, part 3  We are still in “Mmmm” state. What should we do? How should we proceed? We decide that we should limit number of metrics to only high priority ones. For that purpose we need clear metrics framework that would allow to prioritize metrics prior to implementing them. After few workshops, we have a metrics framework that groups metrics into different categories and asks metric author to provide context information to each metric. give your feedback: twitter with #softmetrics  
Slide 52: “Brown Solutions”, part 4   It's getting better. Or at least I thought so... The person responsible for metrics implementation in Brown Solutions resigns. New person has new ideas about what metrics are needed, how they should be introduced. We have to iterate over metrics framework again. Again changes of scope.  give your feedback: twitter with #softmetrics
Slide 53: “Brown Solutions”, part 5  Finally we start thinking about automation of metrics gathering. We are digging into company development tools: bug tracking, development planning, code repositories. We need to change fields in defect tracking system. We need to introduce new field “found in version” and make few other fields mandatory. Again resistance. Luckily we have strong support in management team and we get things done fast. give your feedback: twitter with #softmetrics  
Slide 54: “Brown Solutions”, part 6  Development teams starts building scripts to collect data from different tools. We face some issues, due to the fact that different teams use development tools in a different way. Another round of discussions and changes into the development tool usage guidelines. Finally, automation solution has been rolled out and employees can use metrics in their daily work. give your feedback: twitter with #softmetrics   
Slide 55: “Brown Solutions”, part 7   How to spread the metrics to all employees? We build a wiki page, describing each metric in detail so users can anytime find metric definitions and examples of usage. We also setup skype chat where users can ask any metric related question from the metric specialist. From first days of solution usage we again receive new set of metrics people would like to use... so we start iterating. give your feedback: twitter with #softmetrics  
Slide 56: What did we do wrong? give your feedback: twitter with #softmetrics
Slide 57: Agenda • Applications of software metrics • Why introducing metrics is so hard? • Lesson from the battlefield • Common mistakes • Steps to introduce metrics in organization give your feedback: twitter with #softmetrics
Slide 58: “Great! Let's collect data on everything and then we'll find correlations, meaning and information!” give your feedback: twitter with #softmetrics
Slide 59: Start Small!  Define a concrete goal and start moving towards it using metrics as indicators Use metrics frameworks to assist you − − −  Goal – Question – Metric (GQM) Practical Software Measurement (PSM) ... give your feedback: twitter with #softmetrics
Slide 60: Measurement Culture is Missing   Don't use metrics to punish or reward Don't ignore the data − Make decisions and follow through with actions give your feedback: twitter with #softmetrics
Slide 61: Hope that there is a single super metric for all  I don't know one give your feedback: twitter with #softmetrics
Slide 62: give your feedback: twitter with #softmetrics
Slide 63: give your feedback: twitter with #softmetrics
Slide 64: give your feedback: twitter with #softmetrics
Slide 65: Forgetting about context  There is a difference in what “1 bug” means in different organizations. − − Not a huge issue for 10 developers team of web service? But what about manager of failed NASA project because of this 1 bug? give your feedback: twitter with #softmetrics
Slide 66: Factors Influencing People Productivity  People factors: size and expertise of the development organization Problem factors: complexity and number of changes in design constraint and requirement Process factors: analysis and design techniques, languages and CASE tools Product factors: product requirements – reliability, performance etc Resource factors: availability of hardware and software resources give your feedback: twitter with #softmetrics    
Slide 67: “A good guideline is that measures of individual productivity give you questions to ask but they don’t give you the answers” Steve McConnell give your feedback: twitter with #softmetrics
Slide 68: Agenda • Applications of software metrics • Why introducing metrics is so hard? • Lesson from the battlefield • Common mistakes • Steps to introduce metrics in organization give your feedback: twitter with #softmetrics
Slide 69: 9 steps • Identify user roles for metrics • Target goals • Define questions • Select metrics • Standardize definitions • Requirements for Life-Cycle Tools • Automate metrics collection • Decision criteria after collected metrics • Communication plan give your feedback: twitter with #softmetrics
Slide 70: 1. Identify user roles for metrics  Don't start if you don't know who will be using metrics Users will be evaluating your work, when you deliver the results  give your feedback: twitter with #softmetrics
Slide 71: 2:Target Goals, 3: Define Questions, 4: Select Metrics  Goal-Question-Metrics, Victor Basili Conceptual Level: Goals Operational Level: Questions Quantitative Level: Metrics Q1 Goal 1 Goal 2 ANALYSIS DEFINITION Q2 Q3 Q4 M1 M2 M3 M4 M5 give your feedback: twitter with #softmetrics
Slide 72: 5. Standardize definitions • You are dealing with numbers → be precise in definitions Misinterpretation can lead to • • • Wrong strategic decisions Somebody fired give your feedback: twitter with #softmetrics
Slide 73: 6. Requirements for Life-Cycle Management Tools  Collecting metrics in most cases influences the practices how your company uses application life cycle management tools − bug tracking, project management, test management, requirements management, build and release management, code repositories   Be ready to do significant changes Try to secure support of development tools team give your feedback: twitter with #softmetrics
Slide 74: 7. Automate collection of metrics  People don't like routine work – manual measurement is a burden Measurement should become a habit −  Use tools that simplify or automate data collection       Programeter, Cast Software, ... Rescue Time, Toggl, ... Pivotal Tracker, Jira, ... Hackystat, Sonar, FishEye, … Memory and CPU usage profilers Automatic load testing tools give your feedback: twitter with #softmetrics
Slide 75: 8. Decision criteria  Answer “what if..” questions − − − … escaped defects per week increased 20% … 0 lines of code was developed … velocity increased 2 times  Integrate this into you knowledge base give your feedback: twitter with #softmetrics
Slide 76: 9. Communication Plan  Numbers if used incorrectly they can damage your organization Be sure to clearly and loudly communicate of metrics in your organization. Be honest and straight-forward Share the information −   Show the data, tell how it is used, interpret data together, encourage to use data Feeling of ownership give your feedback: twitter with #softmetrics  Involve people in metrics definition −
Slide 77: 9. Communication Plan   Don't use metrics to punish or reward Articulate organizational goals − Let people know your intentions give your feedback: twitter with #softmetrics
Slide 78: 9. Communication Plan  Don't ignore the data − Make decisions and follow through with actions Same understanding what do metrics express, when and how they are collected and reported Trends are more important than single data points give your feedback: twitter with #softmetrics  Define data items and procedures −  Understand trends −
Slide 79: Go Back To Step 1   Don't expect to get it right in the first try Start small, but be ready to iterate give your feedback: twitter with #softmetrics
Slide 80: Additional Reading  Encyclopedia of Software Development Life-Cycle Metrics http://www.sdlcmetrics.org L. Westfall, 12 Steps to Useful Software Metrics − −  http://www.westfallteam.com/Papers/12_steps_paper.pdf  SDLC Metrics blog by Programeter − http://blog.programeter.com give your feedback: twitter with #softmetrics
Slide 81: Call to Action  Encyclopedia of Software Development Life-Cycle Metrics www.sdlcmetrics.org We are looking for contributors Internship position −  give your feedback: twitter with #softmetrics
Slide 82: E-mail: mark.kofman@programeter.com Skype: kofman give your feedback: twitter with #softmetrics
Slide 83: Thank you for your time and attention! give your feedback: twitter with #softmetrics