Notes from 51st International STC Conference
Baltimore, Maryland, May 9-12, 2004

My Favorite Metrics

Kathryn Auten
Texas Instruments

Kathryn Auten is Product Information Manager for ASIC Division of Texas Instruments. See the STC conference website for handouts.

Session Description: This session focused on showing attendees how to make metrics work for them, rather than them working for metrics. It also explained how to translate metrics to meaningful information and how to keep metrics "friendly."

• Key questions
   
• What are metrics?
• Why use metrics?
• How to set "good metrics"
• Types of metrics
• How to keep metrics friendly
• Examples
• What are metrics?
   
• Metrics are tools for analysis, decision-making, problem-solving, public relations, and education.
• Metrics show value: how well you are doing (against standards, goals, competition).
• Metrics facilitate process improvement, show progress over time.
• Metrics help identify and assess the extent of problems, analyze causes, and develop solutions.
• Metrics can be a teaching tool.
• Poor metrics:
   
• Waste time
• Waste money
• Can hurt you (at least make you not look good)
• Can misdirect your attention
• Can lead to failure
• To develop good metrics
   
• Identify audience
• Have a goal; ensure possible actions
• Focus on essentials
• Ensure consistency
• Compare apples to apples
• Link them to your process
• Have a plan – consider the metrics needed at the beginning of an effort
• Plan for follow-up
• Use them in conjunction with anecdotes and other material
• Establish a baseline
• Involve all key players
• Seek feedback, test ideas
• Look at other disciplines for ideas (SEI, Six Sigma, ISO)
• Develop complementary metrics
• Avoid
   
• Metrics without practical application
• Metrics that are too complex or expensive to collect
• Misleading, unclear, or inconsistent data
• Straining at a gnat (minutia) and swallowing a camel
• "So what" metrics
• Types of metrics: management
   
• Financial (tangible): cost, savings, impact on revenue (ROI), productivity
• Value (less tangible): customer satisfaction, service, flexibility, competitive edge
• Types of metrics: information development team (documentation)
   
• Effectiveness: goal attainment, process, problem-solving
• Value: quality, usability
• Types of metrics: customer audience
   
• Value: survey results and actions, support, competitive analysis
• Financial: cost avoidance/savings
• To make metrics "friendly"
   
• Make sure they are easy to interpret them (should not require a book to understand)
• Clearly document any limitations, disclaimers
• Identify what needs follow-up
• Automate what you can
• Build metrics into processes, systems
• Be willing to let them go
• Think of visuals
• Financial: cost
   
• Overall cost over time
• Average cost per team member
• Average cost per deliverable
• Cost per page
• Cost/benefit analysis for tool selection
• Productivity cost example: pinned to page count
• Financial savings
   
• Savings due to process improvement
• Savings due to costs avoided
• Savings due to time saved
• Revenue impact
• Example of print costs avoided by going strictly online
• Financial – Volume
   
• Volume per team
• Volume over time
• Average volume per team member
• Productivity factors
   
• New vs revised vs update
• Text vs graphics
• Source creation method (team, SME)
• Source provider
• Source provider experience level
• Source provide experience level
• Communicator experience level
• Source quality
• Product maturity
• Technology maturity
• Content technical complexity
• Production process
• Automation
• Minutes it takes to perform given tasks can be a good metric.
• Value: customer satisfaction
   
• Surveys
• Unsolicited feedback
• Usage data
• Service metrics
• Value: customer surveys
   
• Use to determine general satisfaction, areas for improvement, reaction to proposed changes
• Use combination of closed- and open-ended questions
• Be specific
• Tell respondents findings and planned actions!
• Follow up.
• Examples:
   
• Which design and specification menu page to you prefer? Why?
• If you are looking for information about [topic], which of these headings would you select?
• Value: usage data
   
• Software that automatically collects data and generates charts and tables is available.
• Can get daily, monthly reports
• Includes info such as total hits, total unique sites, total unique hits, etc. (daily and monthly reports)
• Typical tool: Webalzyer
• Value: service
   
• Estimate incurred costs and define capability level
• Problem tickets and customer questions to Contact Us mail address resolved
• Customer questions answered successfully, in timely fashion
• All responses to customer requests were categorized, tracked, and tallied.
• Value:
   
• Flexibility
• Demonstrate variety of deliverables
• Advertise skills of team
• Should you give your management metrics if they haven't asked for them? Yes. Work proactively to keep/get yourself in the best position possible. Quantifying your value improves your professional standing, helps your job security, and advances your career.
• Metrics itself can be a service you can provide; if they like your own measurement of internal client satisfaction, management might get you involved in measuring external client satisfaction.
• Value: competitive edge
   
• Benchmarks
• Industry studies/comparisons
• Feedback from customers
• Maintain a database of anecdotal evidence to back up your statistical data.
• Effectiveness: goal attainment
   
• Key result areas/management by objective
• Demonstrate how team goals support overall organization's goals
• Percentage of planned tasks completed
• Percentage of projects delivered on time
• Success of quality-related goals
• Success of customer satisfaction goals
• Effectiveness: process
   
• Staffing
• Schedule-related
• Rework, iterations
• Predictability
• Complexity
• Risk assessment
• Lessons learned
• Effectiveness: staffing
   
• Comparative statistics with other organizations (e.g., how many developers per tech writer)
• Estimate workflow and labor
• Value: quality
   
• Requirements satisfied
• Adherence to standards, templates
• Inspections/peer reviews (log defects)
• Problem reports
• Value: usability
   
• Time to complete task
• Time to find topic
• Customer satisfaction
• Readability
• Number of errors/problem reports
• Effectiveness: problem-solving
   
• In many cases, should be tied to process
• Should help to identify and analyze root causes
• Can help to assess possible solutions
• Should measure effectiveness of solutions applied
• QUANTIFY!
• Effectiveness: problem analysis
   
• Many documentation requests
• Late source code from SMEs: created metric showing late inputs, by department/manager
• Could take the next step and show correlation between late inputs and higher level of customer problem calls (costing money)