International Journal of Project Management Vol. 15, No. 2, pp. 67-72, 1997
Copyright © 1997 Elsevier Science Ltd and IPMA Printed in Great Britain. All rights reserved 0263-7863/97 $17.00 + 0.00
Evaluation of proposals for BOT projects Robert L K Tiong School of Civil and Structural Engineering, Nanyang Technological University, Singapore 639798
Jahidul Alum School of Civil and Structural Engineering, Nanyang Technological University, Singapore
This paper is primarily concerned with the evaluation of tender proposals for build, operate and transfer (BOT) projects particularly in the Asia-Pacific region. It presents an overview of the current practices and techniques in the selection process, namely the NPV method, the score system and the Kepnoe-Tregoe decision-making technique. A review of the findings from the survey of BOT practitioners is also presented. It covers the major criteria that are commonly used by governments in evaluating BOT proposals. The survey also covers the practicality of the Kepnoe-Tregoe decision-making technique for evaluation of BOT proposals. Copyright © 1997 Elsevier Science Ltd and IPMA Keywords: BOT, RFP, evaluation, criteria, Kepnoe-Tregoe
Evaluation of tender proposals for privatized infrastructure projects that are let under the build, operate and transfer (BOT) concept involves decision making by different government officials from different departments and by their consultants. Choosing the best proposal would depend on three elements: the quality of the definition of specific criteria, the quality of evaluation of the available alternatives, and the quality of the understanding of what these alternatives can produce. The purpose of the evaluation and selection process is to identify what needs to be done, develop the specific criteria for its accomplishment, and identify the advantages, disadvantages and risks involved in each of the proposals. This paper describes the selection process and examines the current evaluation practices and techniques in a few countries. Responses were solicited from BOT practitioners to establish the major criteria that are commonly used by governments in evaluating BOT proposals. The practical application of the Kepnoe-Tregoe decision-making technique in evaluating BOT proposals is also discussed.
Research methodology The study on evaluation practices and techniques as used by governments or their agencies in selecting BOT proposals is based on documented experiences and lessons as presented by project promoters, government officials and their advisors in expert seminars and conferences on BOT projects. Information was collected from multiple sources. These are: • request for proposals (RFPs) issued by governments; • interviews with professionals involved in BOT projects
such as bankers, developers, contractors and government agencies; newspaper articles, articles in business magazines, journal papers and conference articles on BOT projects; direct correspondence with governments and promoters for specific views on their actual tendering and negotiation experiences.
The competitive selection process The typical evaluation and selection process in a competitive BOT tender is shown in Figure 1. The process consists of pre-qualification of interested promoters, evaluation of tender proposals, pre-award negotiations and concession award. This process is commonly used by governments or their agencies to serve 3 purposes: 1. to obtain the best deal for the public; 2. to be fair to all competing promoters; and 3. to allow alternative proposals to be considered. Competitive proposals should offer solutions to the many requirements of government's policies, as solicited proposals have to comply with the tender guidelines and criteria as specified in the RFP. Based on the study of documents on BOT projects, the broad requirements that are commonly stated by the government in the RFPs can be grouped under macro and micro requirements. The macro requirements are: (a) the project must be economically feasible with benefts to the community; (b) the environmental impact study must be sound; (c) public safety; 67
Evaluation of proposals for BOT projects: R L K Tiong
\ Figure 1
Selection process in a competitive tender of BOT project
(d) the project must be socially and politically acceptable, and (e) the proposed development must solve the identified needs for the project, represent the best solution to that need and be consistent with a co-ordinated development strategy of government. The micro requirements are: (a) the winning consortium must be technically strong and financially sound; (b) the technical proposal must provide the solution to meet the macro objectives and the demand for the services adequately; and (c) the financial proposal must be attractive and competitive to support the level of tolls or tariff specified or proposed.
Pre-qualification Like the traditional public sector construction contracts, some form of pre-qualification is adopted for most BOT projects in countries such as Australia, Canada, the Philippines, the UK and the US. The main aim of the request of qualification (RFQ) is to shortlist a number of competitive proposals by consortia which consist of reputable and experienced contractors, operators and bankers. In a competitive tender for BOT concession, the number of promoters for the BOT contract may be limited due to the technical, financial, and/or sometimes political constraints imposed upon the promoters. Bruce Nicholls, Executive Director, Major Projects and Infrastructure for New South Wales State Government, Australia, in correspondence with the author on this topic stated: "As a general practice, detailed submissions are not sought in initial responses to an invitation to develop a major infrastructure project. The preferred option is to identify the characteristics of the companies or consortia submitting and from this develop a 'shortlist' of perhaps three or four bidders. This procedure ensures that unsuccessful bidders are not required to incur unnecessary costs". The pre-qualification process will therefore ascertain the financial, technical and managerial ability of each consortium in undertaking the project. For governments in some countries such as Malaysia, 68
Hong Kong and Thailand, it is not the practice to carry out pre-qualification of BOT promoters. In these countries, the invitation to tender for a BOT project is announced in the newspapers and interested promoters are generally given a period of 3--4 months to prepare detailed submissions. The reasons for these governments following this approach is that they believe that the scale of investments required for BOT projects and the keen competition will deter small companies from submitting proposals. Instead, it will attract only serious promoters who are financially strong.
Evaluation of proposals The term 'evaluation' describes the procedure for the assessment of tender proposals submitted by pre-qualified promoters. For BOT projects, the selected promoters would prepare schemes and financial schedules of financing and operating revenues based on the government requirements for the physical structure, the methods of payment and their sources of funds for the project debt and equity. In evaluating the proposals, governments usually follow the Chadwick/Demsetz principle 1 which states that a concession contract must be awarded to the promoter offering the lowest consumer price, or alternatively, to the one offering the price/quality combination which is judged superior to all others. For the selection of the best proposal, the offer price on tolls or tariff would therefore be the predominant, though not necessarily the only, criterion. Other criteria would include cost-related factors (e.g. the construction cost and the operation and maintenance costs), price-related factors (e.g. the extent of future toll increases), technical factors (e.g. the reliability and long-term maintainability of the plant facility) and environmental factors. For most projects, the promoters, after submission of proposals, undergo several rounds of negotiations before the decision is made and the concession is awarded. This is the process whereby some will be eliminated, others will drop out on their own while two to three are shortlisted for final negotiations. This happened for BOT projects in Australia, Hong Kong, Indonesia, the Philippines, the UK and the US. For countries which do not practise pre-qualification such as Malaysia and Thailand, the process is different. In Malaysia, a promoter is selected, based on evaluation of the
Evaluation of proposals for BOT projects: R L K Tiong proposals which should contain detailed submissions. The promoter is then issued with a letter of intent which invites the promoter to enter into an exclusive period of negotiations with the government. The concession is only awarded after the government is satisfied with the best and final offer from the promoter. Successful negotiation therefore plays an important role in the selection process. In Thailand, the government agency will hold pre-award negotiations with the shortlisted promoters and postaward negotiation with the selected promoter. If the postaward negotiations are successful, the government agency will recommend to the Cabinet for the final approval. In both countries, the selected promoters are informed that if the negotiations are not satisfactory, the government will choose the second ranked promoter to continue the selection process. The government always keeps such options open.
Alternative proposals A high degree of flexibility is usually contained within the BOT system of tendering as governments generally permit alternative proposals. Alternative proposals are usually required to be accompanied by a conforming proposal in order that the government may assess the costs and benefits of the alternatives.
Evaluation criteria and techniques The current techniques of evaluation of BOT proposals are based on the net present value (NPV) method, the scoring system and the Kepnoe-Tregoe decision-making technique.
The NPV method Some governments evaluate the commercial and financial package by performing an NPV calculation to discount the project cash flows due to the promoters. The lower the NPV, the cheaper the offer. For utility projects, the comparison is straightforward as it is generally based on government's offtake agreement. For a tolled road, it is more complicated as traffic is not normally guaranteed. Nevertheless, as long as there are adequate traffic studies and conservative traffic forecasts, the government will compare the NPV of the cash flows based on the toll revenues, operation and maintenance costs, financing charges and loan repayments. The advantage of using the NPV method is that the proposals could be compared based on calculated numbers. The method, however, ignores the relative advantages and disadvantages of the technical solution in different proposals. Score system In this system, points are given to the selection criteria and the proposals containing the financial package, technical designs and others are evaluated based on the scores obtained. The proposal with the highest score is considered to be the best overall proposal. The advantage of this method is that several criteria are used in comparing the proposals. The NPV technique could also be used first in evaluating the cash flows. The disadvantage of the score system is that it assumes that all the criteria are of equal importance. The NPV and the score system are the two most common methods used by governments. Some governments use different methods of evaluation and some may choose a single criterion based on their objectives and priorities to
make their final decision. For example, in Hungary, the government will fix the level of tolls and the competition will be based on the lowest concession period offered. For the Canadian Northumberland Strait Crossing, the annual subsidy to be paid by the government to the successful winner was the final and most important criterion.
The Kepnoe-Tregoe decision-making technique Several governments, such as the Hong Kong government and the New South Wales State government, have used more sophisticated evaluation techniques, particularly the Kepnoe-Tregoe decision-making technique in the analysis and evaluation of BOT proposals. The Kepnoe-Tregoe decision analysis method 2 is used by some commercial companies in evaluating alternative proposals for a new investment. The major elements in this technique consist of the evaluation statement, the MUST criteria, the WANT criteria and the evaluation matrix. The modified procedures of this technique for evaluation of BOT proposals are shown in Figure 2 and are explained in the following sections. The validity of the modified procedures for BOT projects was tested by seeking responses of the government officials on the practicality of its major elements through an opinion survey. In order to improve the quality of responses to the questions, the survey was targeted at government officials and their advisors. These persons were asked to respond to the questions with reference to a specific BOT project that they were involved in evaluating. Out of 75 questionnaires mailed, 30 government officials and their advisors responded, giving a response rate of 40%. Respondents who declined to participate cited undertakings of confidentiality of tender information. Several cited sensitivity of the information as their BOT projects are still under negotiation and disclosure of their previous experiences in the projects may undermine the success in the final selection. The evaluation statement. In this technique, the evaluation and selection process must first begin with the adoption of an evaluation statement. The evaluation statement, once agreed by the selection committee, provides the focus for all the evaluation and negotiations that follow and sets the limits of the choice. Government responses. In order to establish the validity and usefulness of the evaluation statement, the government respondents were asked whether they agreed or disagreed that the following statements would have been useful during the evaluation of the BOT projects that they were involved in. These statements are commonly stated in the RFPs as governments' objective in inviting tenders. These statements are: (i)
"Select the proposal that offers the best overall value for money". (ii) "Select the proposal that offers the most attractive financial package and most cost-effective technical solution". (iii) "Select the proposal that is best researched overall in the technical and financial aspects of the project". The responses are shown in Table 1. The most favoured statement is statement no. 2. The positive responses show that the respondents agreed on the practicality of the statements. Respondents perceived that the governments want a package that is financially attractive and technically cost-effective. A few other evaluation statements were 69
Evaluation of proposals for BOT projects: R L K Tiong Criteria for evaluation. Once the evaluation statement is stated for the areas in which the decision is to be made, the criteria are to be established. For the New South Wales government, the criteria adopted for determining a shortlist is flexible and may vary according to the particular needs of a project. Criteria that could be adopted include the bidder's previous experience - 'track record' - in projects relevant to the one under consideration; the range of expertise and professional competence that can be found within the organization concerned; financial and technical strength, including access to lines of credit, within the project, and any uniquely attractive features in the respective bids. For the Kepnoe-Tregoe technique, the criteria can be divided into two categories: MUSTs and WANTs 2.
I Evaluation Statement
Evaluation Matrix & Analysis
The MUST criteria. The MUST criteria are mandatory: they must be achieved to guarantee a successful decision and any proposal that cannot fulfil a MUST criterion will be discarded. Government responses. Through a study of the RFPs, the author found that there are three common criteria that fit the description of MUST criteria. In the survey, the government respondents were asked whether these three criteria could be accepted as MUST criteria for selection of proposals in their BOT projects. They were also asked what other criteria should be included. The three MUST criteria postulated are:
* criteria * weights * SCOreS
* sensitivity analysis
i Aust 1 Scores
Clarification & levelling of proposals
Proposals must be complete and must comply with the tender guidelines. (ii) Promoters must have proven capacity (financial and technical) and experience in construction and operation of similar project. (iii) Promoters must have a local company in its team.
Negotiations with promoters proposal
Figure 2 Selection of BOT proposals on the Kepnoe-Tregoe technique Table 1 Responses on evaluation statement Evaluation statement
Statement 1 Statement 2 Statement 3
Responses by governments Yes
20 22 17
10 8 13
added in by the respondents for specific BOT projects. This shows that the evaluation statement is acceptable to the respondents and is therefore useful in providing the focus in the evaluation process of a BOT project. Some of these statements are as follows: (i) Sydney water treatment plants • proposed plants must be capable of being expanded or upgraded in time at an affordable price (ii) Sydney airport rail link • the evaluation statement must also include externalities such as road congestion relief (iii) Light Rail Transit system, Philippines • proposed project must be financially viable (iv) California transportation projects • projects must make good transportation, environmental and business sense. 70
Table 2 shows that while the majority of the respondents agreed with criteria 1 and 2, most did not agree with criterion 3. The common reason given was that the inclusion of a local company is not essential at this stage of evaluation, but it will be an important requirement prior to the implementation of the project. Other evaluation criteria that the respondents felt are relevant and could be included as MUST criteria in evaluating their BOT projects are: (i) NSW toll roads, Australia • proposals must address distribution of windfall profits arising from unforeseen development or other cause (ii) Canada's Northumberland Straits Crossing project • promoters must comply with environmental requirements • promoters must meet requirements of the financial package • promoters must meet other requirements regarding local involvement such as labour, materials purchases etc. Table 2 Responses on MUST criteria
Responses by governments
Evaluation of proposals for BOT projects: R L K Tiong (iii) Malaysia's power plants • proposals must meet environmental requirements (iv) California transportation projects • project proposals must make good transportation and business sense • project must have local support • project must not have serious environmental problems (v) Texas High Speed Rail project For this project, promoters must address the following issues in their proposals: • local procurement of materials • minority business opportunities • maximum attention to environmental concerns • strategic economic development opportunities • maximum public input/public meetings
The WANT criteria. After the MUST criteria are established, the other criteria are categorized as WANTs. According to this technique, the proposals would be judged on their relative performance against a set of WANT criteria, not on whether or not they fulfil them. The function of these criteria therefore is to give the assessors a comparative picture of proposals - a sense of how the proposals perform relative to each other and which could perform best. Each MUST and WANT criterion could also be subdivided into its own set of sub-criteria. For example, in the evaluation of the Hong Kong BOT tunnels, the sub-criteria for the financial package included toll structure, mechanism for toll adjustment, debt/equity ratio, immunity against market risks, drawdown of loans and debt repayment schedules, guarantees and undertakings, royalty and profitsharing mechanism, protection against uncertainties such as interest rates and foreign exchange variations. The MUSTs therefore enable the governments to decide who gets shortlisted whereas the WANTs lead to the award of the concession. Government responses. Through a study of the RFPs, the author found that the criteria commonly used by governments in evaluating BOT proposals could be grouped into five main criteria. The five criteria are: (i) Degree of attractiveness of financial package. (ii) Financial returns to government and benefits to community. (iii) Relative soundness of technical solution for project implementation. (iv) Relative experience and expertise of the project promoter in similar projects. (v) Degree of environmental impact.
Government responses on W A N T criteria
Responses by governments
2 3 4 5
24 29 27 23
6 1 3 7
the various risks such as future changes in taxation and future changes in property enhancement etc. (ii) Victoria "s Loy Yang B power plant, Australia • parent company support • ability to close transaction according to timetable (iii) Tate's Cairn Tunnel, Hong Kong • ability of government to control/regulate the level of toll charges (iv) 2nd Severn Bridge crossing, UK • realistic project construction schedule (v) Skye Bridge crossing, UK • whole life cycle cost of the proposal • maintainability and durability of the design (vi) California's transportation projects, US • ease of implementation • technical innovation • degree of support for achieving civil rights objectives regarding the utilization of minority and women business enterprises (vii) Texas High Speed Rail Project, US • degree of public convenience and necessity of project
Weighting the criteria. Once the WANT criteria have been identified, each one is weighted by the Selection Committee according to its relative importance in achieving the government's objective of, say, a low and stable toll structure. The most important criterion would be identified and given a weight of 10. All other criteria would then be weighted in comparison with the first, from 10 (equally important) down to a possible 1 (not important). No attempt should be made to rank the criteria. Evaluation of proposals against MUST criteria. In this evaluation, a proposal either meets all the MUST criteria or it does not. As shown in Table 4, if it does not, it will be given a NO GO and is immediately excluded. In actual practice, the government will hold a meeting with the promoter to explain the deficiencies of the proposal and it is up to the promoter to make good the deficiencies.
Evaluation of proposals against WANT criteria. At this The government respondents were given these five criteria and were asked whether the five criteria could be accepted as WANT criteria and used in evaluating the BOT projects that they were involved in. They were also asked to suggest other WANT criteria. The results are shown in Table 3. The majority of the respondents therefore agreed with the five WANT criteria. Other criteria suggested by them for their BOT projects were as follows: (i) Sydney airport's rail link, Australia • planning and development issues - the estimated effect of new transport infrastructure on the development of nearby property • a comprehensive risk analysis on who would bear
stage of evaluation, all criteria have been made visible and the WANTs have been weighted. The proposals will be examined and the relative advantages of each proposal judged. As each promoter is evaluated on the basis of all the WANT criteria, its overall performance and ability to produce desirable results will become clear. Table 5 shows how the evaluation of the relative performance of the Table 4 Evaluating proposals against MUST criteria (a)
In Columns (b) and (d), the government would give the reasons as to why the proposal has passed or failed the MUST criteria. By listing this information, the process has become visible and transparent.
Evaluation o f proposals f o r B O T projects: R L K Tiong
Table 5 Evaluating proposals against WANT criteria (a)
Proposal A information
From the above table, the questions that need to be answered are: How does each proposal perform across the board? How does it measure up against the other proposals on total performance? The questions can be answered by computing and examining the total raw score and the total weighted score of each proposal. A weighted score is the raw score of a proposal multiplied by the weight of the criterion to which the score refers. The total weighted score of each proposal functions as visible comparative measurement of the alternatives.
proposals scored against all the WANT criteria can be done. Sensitivity tests. Where appropriate, sensitivity tests can be applied to the evaluation matrix to ensure that the result would be sufficiently robust and would not be easily altered due to a slight variation in one or more of the weighting factors. The tentative choice. The total weighted score would give the government a tool for selecting a tentative choice. Although the tentative choice may graduate to the status of final choice, governments are often very cautious and would assess all the risks involved. They would hold further negotiations with the promoter to obtain a better deal and to ensure that a security structure of contracts is in place before the concession is awarded. Conclusion
The evaluation and selection process in a BOT tender was discussed in this paper. The current techniques used in evaluation of BOT proposals are based on the net present value method, the score system and the Kepnoe-Tregoe decision-making technique. This paper concludes that government's evaluation goal should be to select a balanced proposal that is financially attractive and technically costeffective. In the final selection, the following criteria are commonly used: n degree of attractiveness of financial package; • financial returns to government and benefits to the community; • relative soundness of technical solution for project implementation; • relative experience and expertise of the project promoter in similar projects; • degree of environmental impact. Indeed a better understanding of the evaluation and selection process and the criteria used by governments will definitely help the promoter to respond appropriately and positively in the tendering and negotiation process. References 1 Demsetz, H "Why regulate utilities?' Journal of Law and Economics (1968) 55-66
2 Kepnoe, C H and Tregoe, B B The New Rational Manager Princeton Research Press, Princeton, NJ, US (1981)
Bibliography 1 Anderson, G 'Lessons from regional experiences' Conference on New Opportunities and Issues in BOT Projects Institute for International Research, Jakarta, Indonesia (1989) 2 Attajarusit, T 'The Thai Government's perspective on BOT projects' The Asian Conference on Planning, Packaging & Implementing BOT Projects Singapore (1988) 3 Guidelines for Conceptual Project Proposals for Toll Revenue Transportation Projects California Department of Transportation, USA (1990) 4 GuidelinesforPrivateSectorParticipationinlnfrastructureProvision New South Wales, Australia (1990) 5 Invitation To Promoters for the Development, Financing, Construction and Operation of a Channel Fixed Link Between France and the UK Department of Transport, UK (1985) 6 Invitation To Promoters for Development, Financing, Construction and Operation of a Third River Crossing at Dartford Department of Transport, UK (1986) 7 Porter, J E and Matson, C R 'A franchized tollway - the design, financing and management of the Tate's Cairn Tunnel, Hong Kong' Conference on Tunnelling Institution of Engineers, Sydney, Australia (1990) 8 Project Brief for Eastern Harbour Crossing, Hong Kong Highways Department, Hong Kong (1986) 9 Proposal For Guangzhou-Shenzhen-Zuhai Superhighway Gordon Wu, Hopewell Holdings, Hong Kong (1983) 10 Revised Terms of Reference for Submission of Investment for Mass Rapid Transit System in Bangkok, Part 1, Stage 1 Expressway and Rapid Transit Authority, Thailand (1987)
Robert Tiong graduated from the University of Glasgow, UK with first class honours in civil engineering. He is a professional engineer in Singapore, holding a master's degree in construction management from the University of California, Berkeley and a PhD from Nanyang Technological University. He is currently a Senior Lecturer at NTU and the Coordinator for the MSc Programme in International Construction Management. He teaches international project financing to the MSc students as well as to PRC construction professionals under a World Bank sponsored training project. His current research interests are in international project financing and privatization of infrastructure projects with special emphasis on BOT projects. He has published extensively on the subject including a monograph on 'The Structuring of BOT Projects" and a paper on "Critical Success Factors in Winning BOT Contracts '.
Professor Alum is an Associate Professor in the School of Civil and Structural Engineering, Nanyang Technological University, Singapore and is former Director of the Centre for Advanced Construction Studies in NTU. He has more than 35 years of practical and academic experience in his areas of specialization of construction technology and project management. He has contributed many papers to various international conferences and journals and is actively involved in research and consultancy related to his areas of specialization.