Productivity, Performance, & Earned Value

 View Only
Expand all | Collapse all

The Power (and Pitfalls) of Metrics in Project Management

  • 1.  The Power (and Pitfalls) of Metrics in Project Management

    Posted 09-04-2024 04:31 AM

    Metrics are critical in project management. They help measure progress, inform decisions, and keep teams on track. However, relying on metrics alone can sometimes be misleading if not interpreted correctly. Today's quote reminds us of the importance of using metrics wisely:

    Quote of the Day

    "Without Metrics, You're Just Another Guy with an Opinion."
    ~ Stephen Leschka, Hewlett Packard

    To what extent do you think metrics are essential in project management and control? Have you found them to be more helpful or misleading in your experience? Share the key metrics you have used in your projects and any insights on how they have influenced your outcomes, both positively and negatively. Let's discuss!

    Thanks.



    ------------------------------
    Moj Kesheh, PSP, FCIArb
    Senior Director
    Forensic & Litigation Consulting – Construction, Projects & Assets
    FTI Consulting
    London | United Kingdom
    Moj.Kesheh@fticonsulting.com
    ------------------------------


  • 2.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-25-2024 11:00 AM

    Hi Moj, I would very much agree with your quote above by Stephen Lesckha. In my experience as a project manager, earned value metrics are a necessary component of project success. At a basic level, earned value (EV), planned value (PV), actual cost (AC), cost variance (CV), and schedule variance (SV) can adequately summarize project progress. In my opinion, these metrics are simple enough where they can provide meaningful information to the project team regardless of the team's experience or background in Earned Value Management.

    One of the more complex metrics that I've had great experience with is TCPI (To-Complete Performance Index). TCPI will tell you what CPI the project must run at to hit the current EAC. By comparing the TCPI vs past CPI performance, you can gain insight into how realistic your EAC is. For example, if the TCPI to hit your EAC is 1.1 and the project is performing at a 0.9, that project would need thorough corrective actions to improve performance and hit the EAC. If no corrective actions are in place, then the project may need to up their EAC to a more realistic target.

    I completely agree with your statement above that the metrics can be misleading if interpreted incorrectly. I've experienced this in past projects and have seen the negative impact it can have. For this reason, with project teams and stakeholders that are less familiar with Earned Value Management, I always try to keep the metrics as simple as possible using EV, PV, AC, CV, and SV mentioned above. If I'm using more complex metrics, such as TCPI, I'll always give a quick disclaimer as to what the metric is and try to gauge the audience to verify whether they're understanding the interpretation.



    ------------------------------
    Kyle Palmer
    Consultant
    Dallas
    kpalmer@k2consulting.com
    ------------------------------



  • 3.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-26-2024 12:41 PM
    Hi Kyle,

    totally agree that EVM indices TSPI and TCPI have been underused. Main reason is that they are answering a question most PM's don't ask. Instead of - what do I have to do to finish on time. They generally ask - what do I have to do to get back on schedule in 3, 6, 9 12 months time, can we do it, what do I have to changed, what productivity increase is necessary, at what acceleration cost and do we want to do it.

    It's easy to calculate the budget volume of planned work on the data date current schedule for data dates 3, 6, 9 , 12 months ahead and compare this to the planned budget value curve (maybe original baseline early start, late start or 50/50) you are using to monitor the project. I call these value points Intermediate Forecast CPI/SPI (IFCPI/IFPSI) indices. This gives the percentage increase required (by budget value) to get back on to schedule. This can then be feed back into the current schedule (e.g. Primavera) and what-if's performed by limiting the budget value points as a resource limit. The new what-if changes (and variations) for 3, 6, 9 12 month points can then be priced and a decision made how to proceed. Currently this is done generally, by the team on an ad-hoc basis by the PM doing a risk review with the Work Package Managers (WPM) and challenging them to "do better than the schedule".

    I have also been pushing for an update in the index nomenclature which hasn't changed for 60 years. We currently just show for example a CPI of 0.8 or a SPI of 0.7. What is missing is WHEN they were produced, against WHICH schedule and WHAT measure you are using to monitor progress. So if you add a SUPERSCRIPT of say P9 of P30 - produced at period 9 of contract period 30, and a SUBSCRIPT showing that the Planned Value curve being used to monitor the project is early start, 50/50, late start (ES=0%, ES=50/50%, ES=100%). We also need to know which schedule is being used (b0 - original baseline, b1 - 1st approved update of etc) and what value measure you are using (budget, quantity, commitment, benefit, activity days etc) . 
                                                                                    P9 of 30
    so instead of CPI=0.8 we have f(bg)(b0,PV) = 0.8
                                                                                    ES=50/50%

    We can now see that the index was produced using budget values against the baseline curve at Period 9 of a contract period 30 using the median planned value line.

    Good luck in your feedback

    Peter Holroyd







  • 4.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-27-2024 02:45 PM

    Hi Kyle,

    Thank you for your thoughtful response, It's great to hear that you're utilizing TCPI effectively; it's indeed a powerful metric when used wisely. I appreciate how you emphasize the importance of simplifying metrics for broader understanding, especially when working with teams less familiar with Earned Value Management (EVM). That's a practical approach that can help prevent misinterpretation and ensure everyone stays aligned.

    For the benefit of other AACE International members who might be less familiar with EVM, it's important to note that the primary motivation for Earned Value Management is to actively participate in the decision-making processes of Program Management. EV is just one of several indicators of planned performance, and while it is cost-centric, it can also contribute to early schedule assessment within a program. However, as the program progresses, EV's Schedule Performance Index (SPI) tends to lose sensitivity, particularly when more precise insights are needed. This is why it's crucial to complement EVM metrics with other tools and methods (like Earned Schedule , probabilistic risk assessment, etc) to maintain accurate oversight throughout the project lifecycle.    



    ------------------------------
    Moj Kesheh, PSP, FCIArb
    Senior Director
    Forensic & Litigation Consulting – Construction, Projects & Assets
    FTI Consulting
    London | United Kingdom
    Moj.Kesheh@fticonsulting.com
    ------------------------------



  • 5.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-29-2024 11:35 AM

    Kyle:

    Are your contracts for which you are using earned value mostly cost reimbursable or fixed price?  



    ------------------------------
    Christopher (Chris) Carson FAACE
    Director of Program & Project Controls, Vice President
    Virginia Beach
    Chris.Carson@Arcadis.com
    ------------------------------



  • 6.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-29-2024 12:06 PM
    Hi Chris,
    In my experience, Earned Value can benefit project success regardless of the contract type. I've used Earned Value for cost reimbursable contracts and many fixed price contracts as well. What primarily differs between the two is the interest and engagement between the buyer and seller.
    The buyer of a fixed price contract may not be interested in the cost side of Earned Value since their payment structure for the project has already been negotiated. However, they very well may have an interest in schedule performance (SPI) to gauge whether their project will be completed on time. Alternatively, the buyer of a cost reimbursable contract will often require both cost performance (CPI) and schedule performance (SPI) information since they are ultimately responsible for funding the project. 
    From the seller's perspective, I believe it is best practice to implement Earned Value Management regardless of the contract type. It is proven to improve project success when implemented correctly, and that's exactly what I've experienced in my past projects. 



    ------------------------------
    Kyle Palmer
    Consultant
    Dallas
    kpalmer@k2consulting.com
    ------------------------------



  • 7.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-29-2024 01:28 PM
      |   view attached

    Kyle:

    It sounds like we are in agreement, I advocate for the use of EVM on all projects, whether cost reimbursable or fixed price.   I do think the difference is more related to fixed price contracts loading the price to the owner, so as you note, cost performance is less useful when Actual Costs are only a percentage of the BAC instead of a true assessment of current contractor costs.

    I think you mentioned it earlier but EVM on fixed price contracts carry a risk that, if the project is running late, the schedule metrics during the final third or so of the project rapidly become unreliable so use of Earned Schedule is vital for accurate assessments.  Since SV=0 and SPI = 1 at project completion, regardless of the completion date, we just can't rely on EVM in that situation.

    But for me, that's a major reason why it matters if the contract is fixed price.  I like using the schedule for invoicing so it forces more attention on the status. 

    My fellow Arcadian, Andrew Dick, and I wrote a paper using our version of EVM (loading duration-days, as we call it) that works very well on fixed price contracts and provides accurate metrics all the way to completion even if late.  I've attached it, it's particularly useful if the schedule is not loaded with costs or other resources and it doesn't require a contractor to do anything in the way of loading.  In our testing of the process, Andrew assigned arbitrary costs to the resources, and it seems to offer value on the cost side as well as the time side.  Since the AC input to the EVMS is actual durations, the process allows for resource overruns, which means the BAC can be greater than the planned value, which emulates cost reimbursable results quite well.  I've attached the paper.   

    Good discussion!



    ------------------------------
    Christopher (Chris) Carson FAACE
    Director of Program & Project Controls, Vice President
    Virginia Beach
    Chris.Carson@Arcadis.com
    ------------------------------



  • 8.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-30-2024 05:28 PM

    Chris,

    Really appreciate you sharing that paper. I'm very interested in the duration-day concept, especially for projects that are nearing completion. As you mentioned, SPI becomes less and less reliable when nearing project completion and this appears to be a suitable alternative.

    I'm envisioning a scenario of using standard SPI on projects up until a pre-defined project % complete is reached, at which time you can phase in the duration-day analysis. Will need to think more on that application, but appreciate you sharing this information.

    Kyle



    ------------------------------
    Kyle Palmer
    Consultant
    Dallas
    kpalmer@k2consulting.com
    ------------------------------



  • 9.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-31-2024 08:58 AM

    Kyle:

    Glad to share, it's always good to get wider use of techniques - that's often how improvements are made.  

    We run EV and ES together but typically only report on EV until it starts getting unreliable.  Part of that is that I think there is some analysis that would be insightful if we can start to understand how the two systems vary throughout the project.  I think the ES levels out any disparity from a large early delivery of equipment that would allow EV to show better performance that actually warranted.  

    We have a Recommended Practice, No. 55R-09, "Analyzing S-Curves" which mentions ES, but I'd like to collect data and discuss what insight examining both EV and ES curves yields and then get the RP updated so it fully discusses ES and the differences.

    I've been kicking around the idea of submitting the Duration-Day EVM for an RP - it would get more attention that way.  



    ------------------------------
    Christopher (Chris) Carson FAACE
    Director of Program & Project Controls, Vice President
    Virginia Beach
    Chris.Carson@Arcadis.com
    ------------------------------



  • 10.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 10-31-2024 08:45 PM

    I find EV discussion very interesting, and I have some comments based on my experience of EV which is being a contributor to "Australian Standard 4817 Project performance measurement using Earned Value" and setting up EV systems for 4 medium sized projects between 2000 and 2010.

    Planned Value, I see five Planned value curves, Early, Early Levelled, Late, Late levelled and the Performance Measurement Baseline. I usually set the PMB halfway between the Early and Late which in turn give the project manager some leeway before being behind. But unfortunately, most planning and scheduling software does not give the flexibility to display all these curves at the same time.

    Earned Value, I have found that most software will give this reasonably well when the user know what they are doing.

    Actual Costs, I have found on large complex projects which have many subcontractors, POs and a day labour force etc. that collecting the Actual Costs accurately is a mammoth task and requires all the senior management on board and very slick systems including calculating accurate accruals for all unpaid but complete work. If this is not done well accurately then the data is almost useless. I have found that getting everyone to comply when there are many organisations involved in a project is very difficult. 

    ETC, this is often not completed, and I remember that the old PMI EV Standard did not even mention the ETC curve. Getting accurate actuals and ETCs into a software package like P6 or any other commercial planning and scheduling software is also very difficult.

    If you are complementing using EV on a large complex project, then make sure you setup your systems early, including procedures etc, review the structure of your contracts, POs and day labour time booking so all costs may be mapped to an EV activity in a timely fashion, train your staff and be prepared to lose the people who will not comply who will in turn make running the EV systems very hard. 

    Also, do not underestimate the number of people require to run an EV system. One project I was involved in auditing the scheduling systems, which was started without EV and had five schedulers. It was decided to implement EV and then the number of schedulers and people involved in running the EV system went up to thirty people. This project was built about 10 years ago and was worth about AUD4billion.

    I have found EV systems either give great results or are a big waste of money when they are inaccurate. Usually the Planned and Earned are OK but then the wheels fall off when trying to get the Actuals corrrect and align with the Earned.



    ------------------------------
    Paul Harris
    President and Managing Director
    Eastwood Harris Pty Ltd
    Templestowe, Australia
    ------------------------------



  • 11.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 11-02-2024 07:12 AM
    Paul,

    totally agree about ACTUAL COSTS. We (the planners)  just used the QS monthly CVR Report as in the end they have to align.

    Peter





  • 12.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 11-03-2024 09:36 PM

    On the mining and infrastructure projects I was working on we did not have a QS.

    To get the EV Actual Costs I put into all contracts that the contractors had to submit their monthly claim with an updated program as at the end of the month which I used as the basis of updating the master program and calculating the EV Actual Costs.



    ------------------------------
    Paul Harris
    Retired
    Templestowe, Australia
    ------------------------------



  • 13.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 11-04-2024 08:25 AM

    AACE has a definition called estimated actual cost. Is that the same

    ESTIMATED ACTUAL COSTS – In earned value management according to the ANSI EIA 748 standard, these are cost
    added to cost from the accounting system to create the appropriate actual cost of work performed (ACWP).
    Estimated actuals are sometimes necessary to ensure the ANSI – EIA 748 requirement that budgeted cost of work
    performed (BCWP) is on the same basis as the reported ACWP. The basis for estimated actuals is documented and
    reversed which in the cost is accrued in the accounting books of record. Example of records may include invoices
    received, material purchase orders, submitted journal vouchers. (October 2013)



    ------------------------------
    Michael Persaud
    Columbus
    taradep@gmail.com
    ------------------------------



  • 14.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 11-05-2024 11:55 AM
    Not sure why someone has to estimate for actual cost.  Estimate is for future and actual is for past.





  • 15.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 11-05-2024 01:04 PM

    Hi Michael,

    Estimated Actual Costs (also known as estimated ACWP) have a slightly different application. They are often used in material planning to keep Budgeted Cost of Work Performed (BCWP) and Actual Costs (AC) aligned. One of the foundations of material planning is that BCWP must be credited in the same period that the work is completed and that ACWP is applied. 

    BCWP for material is often planned according to the point of receipt, so what happens when material has been received but the project has outstanding invoices that haven't been paid at the end of the month? Well, the project will have earned the BCWP but will show zero AC until they pay the invoice. This is going to result in a false variance in the earned value, claiming that the project earned value without actually paying any cost for it.

    This is where Estimated Actual Costs come into play. The project can use an estimated actual cost to keep BCWP and AC aligned in that reporting period. Once the invoice is paid, and the cost is captured by the project, the estimated actual cost that was applied is removed. This keeps BCWP and AC aligned and improves the accuracy of your earned value metrics.

    Kyle



    ------------------------------
    Kyle Palmer
    Consultant
    Dallas
    kpalmer@k2consulting.com
    ------------------------------



  • 16.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 11-05-2024 02:13 PM
    Materials are always and has to be delivered in advance of construction.  Will it not better to actualized delivered material based up on receipt and delivery confirmation rather than estimate for actual delivered material.  It will easier for the wholistic reference when it is actualized.   If one use estimated for delivered material, it may confuse other.

    Current market condition demand that certain percent of advance be paid without material delivery.  In those conditions, estimate will be OK.  





  • 17.  RE: The Power (and Pitfalls) of Metrics in Project Management

    Posted 11-05-2024 06:58 PM

    Part of the long process of setting up an EVPM system is agreeing how materials are managed. There are many types of materials such as bulks (like concrete) and major equipment. The important issue is that the Planned, Earned and Actual Costs are calculated on the same basis so the variance are calculated on the same basis and actually mean something. One major materials decision is which materials are included with the installation activities and which are included with the procurement activities. I like to include bulks with installation and handle major equipment outside installation and try to accrue the cost of major equipment when they are in the store and have passed their inspection. I also do two EV evaluations on equipment, one on the value of Purchase Orders placed, to ensure the POs are being placed fast enough and one on their inspection to ensure the equipment is being delivered and checked fast enough for installation. I also often do two EV evaluations on design and installation activities, one on hours and one on costs, again these decisions are all part of setting up an EV system. In my experience the organisations existing systems often restrict what you can do with the EV system. One underground mine project I was involved in we conducted EV only on mandays with no cost at all and ran EV curves on Planned, Earned and Actual; the Actual was calculated by security by counting the number of contractor people who went through the gate. This was the cheapest and most successful EV system I has ever set up. Another decision is how many curves you are going to produce, some projects I have been involved in only produce Planned and Earned, some have added the Actuals and only a couple attempted to include the ETC.

    Do not underestimate the time and commitment required to set up EV systems.



    ------------------------------
    Paul Harris
    Retired
    Templestowe, Australia
    ------------------------------