Skip to main content

Product Processes and Best Practices

Top Innovation and Product Development Resources of 2012

Revisiting Wisdom from Analysts, Thought Leaders, and Practitioners

Happy New Year! As we kick off 2013, I thought it would be helpful to share some of the most popular innovation and portfolio management resources of 2012. These whitepapers and Webcasts feature analysts, thought leaders, and practitioners sharing their wisdom and experience to help us make better, more informed decisions and optimize our limited resources.

Here are five most widely read (and viewed) resources:

  1. PDF: Issue in Focus: Meeting Fixed Product Launch Windows, Managing Portfolios When Time to Market is Non-Negotiable
    In this research report, Jim Brown, president of Tech-Clarity, addresses the critical nature of hitting launch windows and provides insight from successful companies as well as tips for improving the odds of success.
  2. Video / webcast: On-Demand Webcast: Building and Managing an Innovation Portfolio
    Hosted by best-selling business author Jerry Manas, featuring Chip Gliedman, vice president and principal analyst, Forrester Research, Inc. and Carrie Nauyalis, NPD solution market manager, Planview, this webcast provides insights to help you build and manage your innovation portfolio.
  3. PDF: Ten Proven Military Strategies For Better Resource Planning: Avoiding Custer's Last Stand
    This latest whitepaper from Jerry Manas explores 10 timeless military strategies ‒‒ tried and tested over thousands of years ‒‒ that can be effectively applied toward modern day resource planning.
  4. Video / webcast: On-Demand Webcast: How Technology-Enabled Visibility Lets You Prioritize Products and Optimize Resources
    Find out how product portfolio management (PPM) technology enables organizations to optimize their limited people and financial resources to achieve their product delivery objectives. Featuring Planview, Isabel SA, and Frost & Sullivan.
  5. PDF: The 3rd Product Portfolio Management Benchmark Study
    With input from more than 1000 product development executives and managers over the last three surveys, this report provides insightful statistics and informative data on the state of product portfolio management along with thought-provoking recommendations for all product development organizations.

Continue the Conversation Online

We're actively engaged in social media connecting product developers and practitioners to the latest news and information in innovation and product portfolio management. Join the conversation by:

Are there other materials that provided you important guidance in 2012? Please use the comment section below to share your favorite white papers, Webcasts, blogs, etc.

How Does Your Product Organization Compare?

Written by Maureen Carlson, Partner at Appleseed

Maureen Carlson

Participate in The Third Annual Product Portfolio Management Benchmark Survey

It's hard to believe that we're now conducting the third annual benchmark survey on the State of Product Portfolio Management. More and more product development professionals are looking to this study to learn about how their peers manage complex product portfolios and what trends are affecting their processes and automation plans.

The last survey was conducted in late 2010 and the results were posted in 2011. We have learned a lot about the key risks and pain points that companies face including risk adversity, how companies recognize good projects and kill bad projects, the challenges around allocating, and we've added new questions this year to glean even more valuable information. Our goal is to share the results with peers in the product development community to continue to evolve this strategically important area as a discipline.

Here are a few takeaways from the last survey:

  • 57% of participants said that one of their top three pain points was having "too many projects for their resources"
  • 42% find their forecasted schedules "mostly to highly inaccurate"
  • The 3rd greatest risk was "Not being able to drive innovation fast enough"
  • 60% indicated that their organizations are "risk averse to highly risk averse on new product innovation"

We invite you to take 10 minutes to participate in The Third Annual Product Portfolio Management Benchmark Survey and automatically enter to win an Apple® iPad as a way to say thank you for your time. We will provide the full report and share the results on May 10th at PIPELINE 2012.

Top Five Most Popular Product Development Resources of the Year

It has been an amazing year in the product portfolio management space and it is time to reflect on accomplishments and new goals on the horizon. As the year comes to a close, I'd like to share the most popular product development resources of 2011. The list covers a variety of information on how PPM tools coupled with best practices can achieve a greater ROI while effectively managing resources to maximize productivity. It's a good set of reference material as you begin planning initiatives for 2012.

Five most widely read product development resources of 2011:

  1. PDF: Navigating the New Normal: How Companies Handle the Portfolio Management Question -- The article addresses how companies are navigating the potential pitfalls and unknowns facing product development professionals today.
  2. PDF: Improving Portfolio Decision Making -- Author Jim Brown of Tech-Clarity reveals how PPM plus best practice processes are a great enabler of ROI for product development firms the world over.
  3. Video / webcast: Right Resources, Right Management, Right Time -- The video highlights how to effectively allocate and manage resources to maximize productivity.
  4. Video / webcast: Play to Win with Your Product Portfolio -- The webcast shares proven strategies on how to confidently assess the value of your product portfolio and improve the odds of success.
  5. Webpage: Innovation Tools: Enablers or Derailers? 4 Easy Steps to Keeping It on the Rails -- The post provides answers on how to drive measurable business results through innovation processes and tools.

We want to hear from you. What resources did you find beneficial and why? Did you find a particular subject valuable that did not make the list? Tell us why by posting a comment below.

Happy New Year and I look forward to connecting with you in 2012!

5 Game Changers in Product Management - Part 2

Hopefully, you've had the opportunity to read Part 1 of this 2-part blog series. Carrie Nauyalis began our series with 3 of the top 5 game-changing tools that you can take advantage of to improve product management in your organization and make your life much easier. The last 2 tools are equally important.

Social Product Management -- "Are Product Managers going social?"

You bet we are. I follow a large number of blogs on a daily or weekly basis and post my own Tweets and blogs as well. Why? I can post questions and get great responses that help me better deliver products that I know customers want. Blogs also allow me to follow competitors, analysts and industry experts so I am first to know if there is an issue with our customers. It's amazing what you can learn about your company when you follow Twitter! Industry experts reveal predictions and the next big trend, while competitors use blogs to make big announcements. I'm not waiting for a formal press release… I want to know what's going on NOW.

Another key aspect for us at Planview in social product management is collaboration. If you're in the middle of a product development or sprint, you need to communicate with your global team at a moment's notice. What if you could click on a team member's name and instantly video chat with them to ask a question or get a live update? If you could do that from your product management dashboard, that's powerful. With Planview Enterprise, you can.

Product Management Analytics -- "What data do I need?"

The most important question a product manager can ask themselves is 'what data can I use to show we're doing our job and the product is performing as expected?' Real-time access to centralized data is crucial to answering this question and performing critical analysis.

Key data that we closely track at Planview:

  1. Analysis of product backlog
  2. Mix of Innovation vs. Enhancement vs. Customer Requirements of the product backlog or next release
  3. Cost release that includes capacity planning scenarios
  4. Revenue forecast by product and market
  5. Roadmap by release, product, technology and market

So what is the bottom line? You need to be able to automate everything from ideation through launch -- and be able to see it and manipulate it in real-time. Product development is a key part of this automation. To be an effective product manager, you must have the analytics that back up each one of these stages. The impact of these 5 Game Changers in Product Management is the ability to take out the tactical, manual, day-to-day slogging we had to go through before Planview Enterprise and automate the process. Now it's your turn to be a Game Changer in Product Management!

Related post: 5 Game Changers in Product Management - Part 1

Evaluating Product Portfolios -- Saying NO To Good Ideas

As a product line manager in charge of several software products, my job is challenging. It's a constant balancing act around delighting the customer, minimizing risk, maximizing revenue, leveraging resource availability, and about 12 other considerations. While I quite often think the pain of having to make difficult tradeoffs between good ideas, I have never had the opportunity to "measure" the portfolio that I didn't choose. That is… not until I got to participate and help facilitate a Discovery Lab called Reinventing Portfolio Management at this year's PDMA Global Conference in Phoenix.

The session was led by SmartOrg's Co-Founder, President, and CEO David Matheson who is a regular lecturer at Stanford on the topic of Strategic Portfolio Management. In his brief introduction to the workshop, David introduced the dice game. While the portfolio simulation sounded interesting and fun, I had no idea how enlightening it would be. While I don't think we "reinvented" portfolio management, I certainly gained a perspective on portfolio management that was new. And after 16 years in the portfolio business, new is uncommon, and delicious.

Through the dice simulation, we were tasked to pick a portfolio of R&D projects to invest in. But that left a collection of projects not included in the selected portfolio but that still clearly had some sort of value. Before evaluating the selected portfolio, we were told to roll the dice to calculate the success and value of the unselected projects. For my Type-A personality, this was a little unsettling on two fronts:

  1. It's very disturbing to me to think that a "roll of the dice" could be considered an accurate metaphor representing my chances of success with my portfolio. SURELY, I was a much better product manager (with better assessment criteria) and way too controlling to leave things up to chance, right? Statistically speaking, given the number of project and product failures, it can be a less than certain game, especially with new technologies.
  2. The second thing that was so disturbing to me was the fact that there was so much value in the projects that didn't get selected. When you think of that value going unrealized, it can be a bitter pill to swallow. I've never actually considered calculating the potential value associated with my product backlog. Ouch.

I think that the dice simulation really opened my eyes about the real risk and uncertainty we deal with daily in portfolio management. It seems that this dice game gave everyone in the class (see picture below -- such a good looking group!) a different insight or nugget to take home, based on the dynamics of their particular organization. Check out the post-class review video for some of those takeaways.

PDMA Global Conference

Are There Best Practices for Product Portfolio Management?

Written by Jim Brown, President & Founder, Tech-Clarity

Jim Brown

I recently did some research on how companies can take a practical path to improve PPM decision-making. I won't keep you guessing (if you were), there are some VERY good best practices that have been developed by the new product development (NPD) community. The research consists of interviews with three distinguished manufacturers in different industries, and they all concurred that the best approach is to start small with PPM and grow over time. The report, Improving Portfolio Decision-Making: Marrying PPM Best Practice Processes and Technology to Drive ROI, provides an overview on the value of PPM in addition to laying out a practical plan to leverage best practices tools and techniques to driver better portfolio decision-making and drive up company profitability.

The Research

During the research I had the pleasure of talking to several very knowledgeable PPM practitioners:

  • Don Kingsberry, Enterprise PMO, Green Mountain Coffee Roasters (the man should really write a book, I told him I thought so)
  • Ian McKenna, IT Business Partner, Infineum (a joint venture between ExxonMobil and Shell)
  • The Manager of Quality Management, R&D, and Legal Applications for a medical device company (who unfortunately was not permitted to share his company name in the report, but I promise you I didn't make him up!)

The Value of Improving PPM

The first thing I discussed with them was why they invested in PPM in the first place. I had done some past research in this area, including Issue in Focus: The ROI of Product Portfolio Management, and thought it was important to start with the business in mind. I was very impressed with a statement from the participant from the medical device company. He saw PPM as his responsibility to be a steward for his company. How many companies would love to have that form of dedication, work ethic, and contribution from their employees? He offered that "PPM processes and tools help us be good stewards of our business. It costs a lot of money to develop products and we should do what we can to select them right and do them right -- because then we have more money to invest."

Mr. Kingsberry of Green Mountain Coffee Roasters (GMCR) has implemented PPM multiple times, and shared experiences from GMCR as well as past experiences where he says PPM has a "profound effect" and "each time I have found the return to exceed our expectations." He shared the history from one of his past experiences in a very large, well respected company saying "we cut millions of dollars of projects that were wasting time and we shouldn't have been working on. We got clarity on that immediately and had a huge multi-million dollar return in 6 months. PPM software and process help bring visibility to those things." What more can I say when people that have "been there and done that" are so positive on the value they received?

Taking the Practical Approach

One of the other key findings of the report is that too many companies overthink their PPM implementation. For some of you that know me, you might be falling out of your chairs! I am always a proponent of a well-planned, well thought out implementation. And my PPM benchmark research at Aberdeen Group showed that business processes and metrics were even more crucial to best in class PPM performance than in any other enterprise technology I have researched. So what gives? People need to understand that the goal of PPM is to provide better information, in a standard way, so people can make better decisions about product investments. Too often, even in my own experience when I ran product management for a software company, I have had people want a scoring algorithm to magically spit out an answer. Don't get me wrong, the metrics and analysis are very important. But they are just one input into a decision making process. As such, it's important to find out what information and metrics the decision-makers will actually use and trust (think simple versus black box voodoo) to make decisions.

In fact, the companies interviewed used some really straight-forward metrics that helped them make decisions. "We implemented fairly standard calculations, NPV (net present value) being an example," explained Mr. McKenna of Infineum. Don Kingsberry also suggested that based on his experience companies need to strive for simplicity, particularly as organizations get bigger.

The other aspect of simplification was to start small and grow. I heard this from every company I spoke with, they all agreed that you shouldn't do too much at once and go "big bang" with your PPM implementation. This is particularly true because companies are now integrating and extending PPM processes further into the front end of innovation and trying to develop a more integrated, streamlined innovation process. Now this is where I get back up on my soapbox about planning ahead. Just like with my "PLM Program Approach," you can start small and build -- but you had better have a plan for the bigger picture you are trying to achieve. You should also partner with a software company that can help provide the path to your larger objectives as you mature and improve. Don't paint yourself into a corner, you want to make sure your initial investments and learnings serve as a foundation for even move value over time.

Implications for Manufacturers

So what does this mean to manufacturers? First, there is a lot of value to be had from PPM. Second, you don't have to reinvent the wheel. There are very good methodologies and metrics available, and software solutions that encompass and enable them. Third, don't spend months trying to invent a process or algorithm that automatically makes portfolio decisions. That is not realistic and likely won't be used. Take the time to provide good, trusted, simple information in a consistent way so decision-makers can compare "apples to apples" when reviewing portfolios.

So that was a quick peek into some recent research on product portfolio management, I hope you found it interesting.

Note from Planview: If you would like to read Jim's full report, it is available here: Improving Portfolio Decision-Making: Marrying PPM Best Practice Processes and Technology to Drive ROI

5 Game Changers in Product Management - Part 1

If you've been in product management for a while, you know there are a lot of things you could be doing with your time to manage your product portfolio. But which things are the most important and effective at helping your company reach its goals.

This 2-part blog series will disclose the top 5 game-changing tools that are at your disposal and can make you a hero at your organization, not to mention make your job easier.

Portfolio Prioritization -- "Are we working on the right things?"

The discipline of Portfolio Management isn't new, but applying it in a product organization may be. Your competition is prioritizing their product portfolio so their hottest products get to market. Are you? Here are the things you need to consider when evaluating your product portfolio:

  1. Will it drive revenue?
  2. Does it align with our corporate and product strategic goals?
  3. Do we have the resources to deliver it on time?
  4. What is the risk involved?
  5. How will this impact the brand?
  6. Will this help us leapfrog the competition?
  7. What will the sustainability impact be?
  8. Will our customers be delighted?

The key is to run multiple 'what if' scenario on products, projects, programs, or features to evaluate how those align with set targets, and whether you have the capacity to deliver. There are powerful tools available to product and portfolio managers that pull all of this information together onto one central page to give executives the information they need to make the decisions that drive the best, most profitable products to market.

Resource Management -- "How do I best utilize my people?"

It's simple math. If you don't have the resources to execute, you can't deliver desirable products on time and on budget. According to our annual benchmark survey on product development, this has actually been found to be the #1 pain point for product development organizations for the past two years. Effective resource management breaks down into 3 key components:

  1. Organizational capacity planning
  2. Development resource planning
  3. Ongoing change management to schedule and resource assignments

If you take this three prong approach to Resource Management, you can expect to achieve these benefits:

  • Clear visibility of demand and capacity to avoid resource bottlenecks, reduce risk, and hit launch targets
  • Forward and backward-looking resource planning to eliminate resource cost overruns
  • Resource management processes that lead to consistent, proactive responses to change

Agile Product Management -- "Why should we care?"

Making the switch from Waterfall to Agile was difficult for me. I felt like I was losing control of the schedule and what was being delivered to our customers. But I soon discovered that Agile actually gave me ultimate control and flexibility as a product manager. In fact, through our Inner Circle program, we regularly show customers what's going on and get feedback from them, involving them in the decisions that need to be made along the way, so I can ensure we're "delivering to delight." But what, specifically, does Agile do for us?

  • Iterative sprint cycle ensures we're delivering what customers want
  • Central repository for all product ideas and backlogs reduces risk and saves time
  • Ultimate flexibility with stories and priorities allows us to quickly respond to change
  • Testing done throughout development often leads to higher quality and met deadlines
  • Embrace the change to Agile. You'll come to love it, like I did.

Be sure to check back for our next blog post to learn about the remaining two game-changing tools. Louise Allen, Vice President of Product Management at Planview, will finish our series. I encourage you to evaluate your processes and tools to see if you are maximizing your product development potential.

Innovation Tools: Enablers or Derailers? 4 Easy Steps to Keeping It on the Rails

"Innovate or Die." It's the current battle cry for businesses trying to get or stay ahead of the competition and remain viable in the market. But how do you avoid innovation for innovation's sake, and instead drive measurable business results through innovation processes and tools?

Birds of a Feather Innovation Leaders Workshop
Birds of a Feather Innovation Leaders Workshop

I learned the answer to that question (and more) at the Birds of a Feather Innovation Leaders Workshop in Chicago a few weeks ago. The event was sponsored by Brightidea and hosted on the Kraft campus. The speakers were great and I especially appreciated their willingness to share what did and did not work in their innovation endeavors. Here's the Cliff's Notes version in four easy steps:

  1. Prepare to Innovate!
    Often companies jump right into idea collection with their employees and customers, desperately searching for the next big thing without giving any consideration to organizational preparedness. I think it was Yoda who said: "There is great responsibility in receiving the idea." Okay, maybe it wasn't Yoda, but it sure sounds like him because it is so profoundly true. For successful innovation, it's necessary to ensure the organization:
    1. is in the right mindset (that of problem-solving)
    2. has supporting processes and tools in place
    3. is properly structured/staffed to execute the processes and administer the tools
    4. is culturally prepared to respond.
    Whatever you do, don't skip these steps! It will sink the innovative spirit of the organization and backfire in harmful ways. I'm not saying you need to spend months and years getting ready to innovate; I'm just recommending that you have answers to the above questions to ensure you are equipped to handle and implement the changes innovation programs can bring.

  2. Focus! And Capture Needs BEFORE Ideas!
    There is a time and a place for the electronic employee suggestion box and open innovation. But having a short-term (2-4 week), focused innovation campaign that gives people a specific problem to solve often leads to more valuable results. And by narrowing idea capture down to a specific problem to solve, it's much easier to define metrics (hard or soft) to determine success and benefit.

  3. Brand It, Baby!
    One way to get people excited about driving innovative results is by branding and marketing your innovation tool and specific campaigns. With today's technology, there is no excuse for rolling out a vendor-branded software tool. Presenters at the conference had full-blown advertising campaigns, including videos to explain the objective and generate hype, and logos with associated color schemes to skin the software. Re-branded programs and tools had names like "Think Tank" and "Idea Kitchen" to really capture the attention of idea submitters. Some companies even sponsored specific "Innovation Days" around a particular campaign to encourage involvement and ownership. Don't fall victim to assuming that "if you build it, they will come." (Field of Dreams; not Yoda.) By branding your innovation program, you're sure to drive quality ideas that deliver results… because it is, after all, about the results.

  4. Show Me the Money!
    Sometimes solving a problem and the feeling of accomplishment in contributing to the success of the company is a reward in itself. But for most of us, a little incentive or some recognition for the great, problem-solving ideas we submit goes a long way. You know how it feels when you put thought and effort into something and it seems as if your idea has dropped into a black hole where there is never so much as a reply or simple, "thank you." It sucks. And why would you waste your precious time on something to which management is only giving lip service. Whether feedback comes in the form of public recognition for being a top innovator (that's FREE for all of you penny pinchers keeping track at home) or in the form of a big fat check (option B, please!), it is critical for the success and sustainability of your innovation program to reward good ideas.

So there you have it, Product Pulse Peeps…innovation in four simple steps. Implementing an innovation tool and program isn't hard, but you can't skip any of the steps if you want to drive real results. As it relates to innovation, Yoda would say (and this time, you can look it up!): "Do or do not. There is no try."

Chief Customer Advocate Reporting for Duty

There were many great takeaways from my recent 2-day session with Pragmatic Marketing. Their expert team, including John Milburn, a friendly upbeat guy with 20 years of product management experience in the tech industry including long stints at Tivoli and VTEL, has more than 10 years of experience working with over 70,000 professionals globally in product management training.

Let's start with the most important thing: The FOOD at the AT&T Conference center on The University of Texas campus (where the class was held) is excellent! This information is neither here nor there but if you get the chance, check out the conference center or hotel as an option next time you're in Austin.

From a more business-oriented standpoint we learned:

  • Product management's primary job is to know and speak for the customer.
    • The following quote (with a few modifications) by Peter Drucker was used multiple times: "The aim of product management is to know and understand the customer so well that the product or service fits him and sells itself."
  • Product management's main responsibility is finding their market's problem. To do this, product managers should:
    • Contact customers (both their own and their competitors'), evaluators, and prospects.
    • Use various research methods including discovery (onsite interviews, focus groups, secondary research) and validation (surveys, choice models, experiments).
  • Too often product management gets overwhelmed with tactical instead of strategic activities.
    • As Milburn noted in class, if product management doesn't do its job then the other departments will fill the void. Other groups have their own goals and ideas so if the product management team is not there to speak for the customer, an inside out product could result (one that solves a space problem but not a strategic one).
  • By focusing on my customers' needs and wants, product managers can help their company build solutions that the market will buy, and what is profit if not the ultimate goal of any company?

After spending two full days getting my head stuffed with information (and my belly stuffed with yummy food), I was raring to go out and apply all these new tips and energy to my job. I was eager to see how much of what I learned in class was applicable to real life.

So far, most of the points ring true. I've spoken with multiple customers about their needs and concerns and gathered a lot of useful information. I have also met a good portion of the executives, sales, marketing, and development teams and worked with them on a hodgepodge of projects. It will be interesting to see how well I remember my role as Chief Customer Advocate once I get inundated with a lot of tactical stuff. I will try my best to focus on strategy, Milburn and Drucker!

I am new to product management, but there were people in my session with years of experience, and we each felt that Pragmatic Marketing taught us something useful. Just goes to show that no matter where you are on your career path, there are always opportunities to better understand the marketplace and your role in it. And, that information is almost as satisfying as the AT&T Conference Center's dessert bar!

Put a Spike In It

'Spikes' are used in Agile Software Development environments to investigate a concept and/or create a simple prototype. Basically, a spike is a quick (typically less than a sprint of effort) exploration by coding of an area in which the development team lacks confidence. The spike is concluded when you learn what you needed to learn. So-called because a spike is "end to end, but very narrow", like driving a spike through a log.

The term "Spike" was introduced by Extreme Programming advocate, Kent Beck, who occasionally consults with us at Planview.

"Create spike solutions to figure out answers to tough technical or design problems. A spike solution is a very simple program to explore potential solutions. Build the spike to only addresses the problem under examination and ignore all other concerns. Most spikes are not good enough to keep, so expect to throw it away. The goal is reducing the risk of a technical problem or increasing the reliability of a user story's estimate."

Mankind always pays attention to the thin interface between the trivial and the impossible, and that boundary is where spikes live. If your development staff knows for sure how to accomplish something, then the process of estimating and tracking progress against estimates via velocity produces a predictable, repeatable process that confines risks. Although not trivial to the people working on it, to an observer outside the team the work is essentially trivial, and therefore a snore.

Spikes are used to expand the realm of the trivial by establishing beachheads into the realm of the impossible. Build a hypercube out of one million facts in 12 seconds? Impossible -- until you prove that it can be done with a simple spike. Like all good magic, the trick is simple once you see how it is done.

I cannot improve upon this maxim, but perhaps it can be enriched with some real-world experience.

We have used spikes for a number of purposes at Planview. Here are just three that stand out:

  1. A spike was written to create a Silverlight grid that could support 50,000 cells to test the claim that this was possible with good performance (it is). This was done by one engineer in about 5 days of effort.
  2. The performance of a custom hypercube implementation was tested using a spike. This took about 7 days of effort.
  3. We tested the inclusion of very powerful dynamic graphic elements and an abstract machinery for utilizing.

None of these were velocity-generating. Each was done outside the bounds of our formal agile process which tracks progress against estimates of effort, yet each was critical in its own way to the success of a project. In keeping with the quote above, we use spikes to learn something -- generally if a given approach will work, and often to eliminate the risk that a software company we don't control is up to snuff. Of course, in a well-run development effort, you want to be doing that at beginning of the development cycle rather than at the middle or the end, but sometimes a problem and its solutions are not presented until late within a project.

If an organization cares too much about velocity, it can fall into a subtle trap of dysfunctionality: missing great opportunities because it is unwilling to invest a small amount in broadening the realm of the trivial into the impossible. It is relatively easy to run a software project to do something that everyone agrees can be done with enough effort. But if it is easy for you, it is easy for your competitors as well. To be truly excellent, a software development organization must utilize spikes in a disciplined manner to produce the best possible overall result for a given amount of effort.