2.-0 LCA consultants


Page last updated

May 22, 2013
by Bo Weidema


Guidelines for critical review of product LCA

By Bo P. Weidema, 2.-0 LCA consultants, www.lca-net.com

Originally published 1997 by SPOLD (www.spold.org), Brussels.

Traditionally, critical reviews - or peer reviews - are known from the international scientific journals, where submitted articles are subjected to critical scrutiny by anonymous colleagues (peers) before being accepted for publication - often after considerable adjustments. Since it is difficult to determine objective criteria for scientific quality, the subjective - but professional - judgement of peers becomes the ultimate quality assurance for scientific work.

Life cycle assessments have in common with scientific work the difficulty of establishing objective quality criteria. Many of the judgements a practitioner will have to make in the course of a life cycle assessment cannot be said to be true or false, but only more or less justifiable. Therefore, the ultimate quality judgement can only be subjective - although based on professional experience.

Until now, critical reviews of life cycle assessments have not been generally applied - not even for life cycle assessments supporting a comparative assertion disclosed to the public. The main argument against critical reviews has been their costs. However, if the critical reviews are used as interactive reviews (where the review is performed both after the scope definition, after the data collection and after the conclusion), the reviews are likely to increase the efficiency of data collection to the extent that the benefit of the reviews will exceed their cost. Besides this, critical review should improve reliability of the results to the extent that this in itself would justify the review costs.

With the requirement in the ISO 14040 standards for critical review of all life cycle assessments supporting a comparative assertion disclosed to the public, the use of critical reviews is likely to become more widespread. In this paper, the procedural aspects of critical reviews are discussed in more detail.

1. Types of critical review: integrated 3-step review or post-study review

The scientific peer review, as known from international scientific journals, is typically performed after the research has ended and the article has been written. In this respect, the peer review does not give any input during the research process, i.e. the peer review is not part of a quality management system. Nevertheless, quality management is not impossible in certain aspects of research work and has lately become more widespread, especially in corporate research. This means that specific parts of the research are subjected to a peer-review-like assessment during the research work, e.g. after the research plan has been made but before the experiments are performed, again after an experiment but before calculating results, etc.

In the same way, critical review of a life cycle assessment may be a simple peer review of the final report, or it may be a more integrated quality assurance involving typically three review steps:

This interactive procedure has especially been advocated by SETAC (Consoli et al. 1993). The advantage of the 3-step procedure is that problems can be corrected at an early stage, before resources are expended on work which later turns out to be inadequate. A 3-step review should not be more expensive than a post-study review - on the contrary: It is less time consuming to guide a study onto the right track from its beginning, than to figure out how a complicated result has been influenced by dubious assumptions, or to reconstruct a missing calculation, once the study is finished. The only disadvantage of the 3-step procedure is that it may take a little more time to perform the study, since the work has to rest while the review is made. Typically one should calculate one month additional time to allow for communication and adjustment of schedules between the practitioner(s) and the reviewer(s).

2. Selecting the reviewer

The reviewer may be found within the same organisation as the practitioner or externally. This should not affect the quality of the review, since this depends on the qualifications of the reviewer and not on his or her affiliation. Nevertheless, an external reviewer is less likely to be biased by relations to the practitioner or the culture of the organisation. Therefore, external reviews tend to be regarded as more credible.

It is also important to consider who selects the reviewer: the practitioner himself, the commissioner or a third, completely independent party. The payment for the review ultimately comes from the commissioner and it may therefore be seen as the commissioner's privilege to choose the reviewer as his trusted representative and guarantee for the quality of the study. Nevertheless, if neither the practitioner nor the commissioner has any influence on the selection of the reviewer, it gives more credibility to a claim of independence from influence from the commissioner. It may therefore be advised to let the choice be made by an independent, third party.

For comparative studies disclosed to the public, an intermediate option has been chosen for the ISO standards, namely to require the commissioner to make a choice of a reviewer who then on his part selects further members of a panel. However, this still allows a third party to make the choice of reviewer on behalf of the commissioner, thus improving the credibility. The review panel may include persons representing interested parties, such as suppliers, employees, competitors, customers, government agencies or non-governmental groups, which is why this type of review in the ISO standards has been entitled review by interested parties. It is obvious that such involvement of external interests will increase the credibility of the result.

3. Qualifications of the reviewer

A qualified reviewer is a person who has adequate professional knowledge of both the life cycle assessment technique itself and the specific product type which is investigated by the life cycle assessment to be reviewed.

Besides this professional knowledge, a reviewer must also possess certain psychological qualifications. A reviewer judged by some practitioners to be too strict and by others to be too lax, is likely to have found the right balance between being too demanding an idealist, and being a naive or "understanding" colleague. The idealist who refuses to recognise the difference between theoretical principles and their practical application can give rise to unnecessary conflicts. The naive reviewer may be deceived by empty declarations which have no foundation in the actual work performed, thus possibly overlooking misleading assumptions or serious mistakes.

The reviewer should be independent, which is, first of all, a state of mind. However, some formal requirements should be fulfilled: A reviewer should have no business ties with the practitioner and should not have any commercial interests in the immediate topic of the study under review. Payment must not depend on the result of the review.

4. Confidentiality and access to information

As a life cycle assessment nearly always includes data for company internal processes and considerations on future plans, it is important that the reviewer maintains the strictest confidentiality about these issues. Confidentiality agreements should specify how, when, where and by whom information is obtained, transferred, stored and deleted.

However, it is important that the confidentiality is not extended further than what is necessary to protect the interests of the involved parties. The confidentiality should not extend to the techniques applied by the practitioner. The progress of the life cycle assessment technique comes from a free exchange of information on methodology and a reflection on the nature of quality and the causes of mistakes, and not from complacency or arrogance hidden behind pretended professional secrecy.

A thorough and satisfactory review can only be performed if adequate secrecy agreements are in place which allows the reviewer unlimited access to all relevant background material, including computerised data.

5. Budget, time requirements and contract

A thorough review will take approximately 10% of the time and budget allocated for the study. As already mentioned, interactive reviews may take a little less time, because mistakes and divergences will be caught in their infancy. This is especially true if the practitioner has a good internal quality management procedure. Critical review should not be an alternative to normal internal quality assurance or an excuse for sloppy work on behalf of the practitioner. If the life cycle assessment is concerned with a new product area or involves many controversial methodological choices, the review budget may have to be considerably larger than specified above.

Obviously, the time requirement is dependent on the size of the study to be reviewed and the budget allocated for the review. Besides this, time should be reserved for the physical communication between practitioner and reviewer as well as the adjustment of schedules when more people are involved. For these reasons, integrated 3-step reviews will be more time consuming as will reviews involving more than one reviewer and other interested parties. When making the schedule for the review, it is important to ensure that the practitioner is available for questioning during the period when the study is reviewed.

It is important that a contract is made, specifying the contents and the background of the review, to avoid any misunderstandings between the commissioner, the practitioner and the reviewer. The contract should specify:

6. Accordance with ISO standards

The most obvious objective of the critical review procedure is to ensure that the life cycle assessment is consistent with the standard to which the study refers. This will in most cases mean the ISO 14040-series, although other national, product specific or case-specific standards may also apply.

The general requirements of the ISO standards are:

Additional requirements apply to the individual phases of the life cycle assessment and to the final report. These requirements are listed in the following sections:

7. Critical review of the goal and scope definition

The requirements of the ISO standards are:

For comparative life cycle assessments disclosed to the public an additional requirement is that the choice of environmental categories shall be as complete as possible as well as appropriate and reasonable in relation to the goal of the study so that the comparison is fair and equivalent for the product alternatives.

In as complex and versatile technique as life cycle assessment, it is impossible to pin down every aspect in an explicit requirement. There will be assumptions and procedures which are obviously unacceptable from a professional point of view, even though they do not conflict with any of the explicit requirements of the standard.

Therefore, besides ensuring accordance with the explicit requirements of the standard, the critical review shall, also according to the ISO standards, ensure that the methods used to carry out the life cycle assessment are scientifically and technically valid. This asks for the scientific and technical judgement of the reviewer, and allows a large degree of freedom to state opinions on aspects not explicitly covered in the ISO standards. This may, for example, be relevant for issues such as the choice of product alternatives to be compared, which is not very explicitly described in the ISO standards.

8. Critical review of the inventory analysis

The ISO standards state explicit requirements on how to record data (see section 8.1), on validation (section 8.2), and on calculations (see section 8.3).

Besides ensuring accordance with the explicit requirements of the standard, the ISO standards require the critical review to ensure that the methods used to carry out the life cycle assessment are scientifically and technically valid. For the inventory phase, the most important issue in this context is the way data are aggregated. The scientific justification for aggregating data should be thoroughly reviewed. Also, the validity of the methods used for calculations should be reviewed.

8.1 Adequacy of data

The ISO standards state the following requiremen

Besides ensuring accordance with the above requirements, the ISO standards require the critical review to ensure that the data used are appropriate and reasonable in relation to the goal of the study. Special attention should be given to ensure that the choice of data does not unduly favour companies participating in or financing the study. Also, it should be ensured that data quality information is handled appropriately (e.g. that it is not aggregated).

The critical review should check that the above requirements are fulfilled for the most important data. These data should be selected by the reviewer according to his or her previous experience supported by any sensitivity analyses performed by the practitioner during data collection. Especially the review on missing data requires the reviewer to use prior experience from life cycle assessments of similar products, performed either by the reviewer or by other practitioners. To ensure that the data are generally appropriate and that the data quality is adequate, a minimum of 10% of the data should be reviewed. A smaller part of the reviewed data should be chosen at random (i.e. not according to their importance to the result).

8.2 Factual validation of data

The ISO standards mention validation as a requirement for all data finally included in a study.

Validation is a general concept covering many different specific procedures, such as mass balances, comparing with earlier measurements or other data, and review by another person than the one responsible for collection of data.

The critical review should check that adequate validation has been made. This can be combined with the general check on adequacy of data and data quality mentioned in section 8.1. Although it is not the purpose of the critical review to re-do the validation, it may be necessary for the reviewer to perform factual validation of those data, which are most important for the conclusion.

8.3 Checks on calculations

Calculation is a general term covering several discrete operations, including:

The ISO standards require all such data calculation procedures to be explicitly documented. Furthermore, the following requirements apply to co-product allocation:

The critical review should ensure that the calculation procedures used are adequate, scientifically and technically valid, adequately documented and justified when necessary.

9. Critical review of the impact assessment

The ISO standards state the following requirements:

Besides ensuring accordance with these explicit requirements of the standard, the critical review shall also - according to the ISO standards - ensure that the methods used to carry out the life cycle assessment are scientifically and technically valid. Thus, the reviewer should state his or her professional judgement regarding the appropriateness of the methods used.

10. Critical review of the interpretation

The ISO standards state the following requirements:

Additional requirements apply to comparative life cycle assessments disclosed to the public. These additional requirements are:

The critical review shall - according to the ISO standards - ensure accordance with the above requirements of the standard, and especially ensure that the interpretations are in accordance with the goal of the study and the identified limitations.

11. Critical review of the final report

The ISO standards require the results, data, methods, assumptions and limitations to be clearly, fairly, and accurately reported in sufficient detail to allow the intended audience to comprehend the complexities and trade-offs inherent in the study. For life cycle assessments, which are to be communicated to a third party, the report shall cover:

Reports of comparative life cycle assessments should furthermore cover:

The critical review shall ensure that the report is in accordance with the above requirements. Especially, it should be ensured that the report is adequate, transparent and consistent, and that all conclusions have a factual basis.

12. Publication of critical reviews

Disregarding the type of critical review, the review statement shall be included in the report of the life cycle assessment study. For external reviews, the response of the practitioner to the recommendations of the reviewer or panel, shall be included in the report from the study.

References

Consoli F, Allen D, Boustead I, Fava J, Franklin W, Jensen A A, de Oude N, Parrish R, Perriman R, Postlethwaite D, Quay B, Sequin J, Vigon B. (1993). Guidelines for life-cycle assessment: A "Code of Practice". Brussels & Pensacola: Society for Environmental Toxicology and Chemistry, (Report from a workshop in Sesimbra, 1993. 03.31-04.03).