Concerning the Results of the External Evaluation of Osaka National Research Institute
In February 1998 the Osaka National Research Institute (ONRI: Teruo Kodama, Director General) convened an External Evaluation Committee, meeting in the institute. The committee consisted of 10 members from outside the institute and was chaired by Professor Emeritus Kazuo Fueki of the University of Tokyo. It evaluated the overall operations of ONRI and presented the director with suggestions. This report compiles the results of the committee evaluation.
The report contains committee member opinions on the following items:
1) Selection of priority areas
2) Selection of research themes
3) Research system
4) Securing, training, and evaluating researchers
5) Research budget and buildings/facilities
6) Enhancement of research through interchange (joint research, etc.)
7) Research achievements
These points are covered below.
After having carefully considered the results, ONRI intends to put the evaluation to positive use in future institute operations.
We look forward to opinions from a broad range of people, thoughts we will weigh in the research activities, evaluation system, and other elements of ONRI operations. Let us know your opinions by email or fax.
Email: tokatsukan@onri.go.jp
Fax: 0727-51-9621 Research Manager
Osaka National Research Institute
External Evaluation Results
1. The initiative of the director and other executive members is fully manifested across the entire scope of institute operations, and is accorded a high estimation.
2. As a national research institute (NRI), ONRI senses the needs of industry and society, uses these needs to orient its research, and consciously pursues that research with well-defined objectives. ONRI also takes the regional character of Kansai industry into consideration, thereby conceiving national projects in areas such as human life engineering, and evincing leadership as a NRI. It has achieved a good balance between its position as a NRI and making a contribution to the local community, thus being active in a way that cannot be matched by other NRIs.
3. External Evaluation Categories
1) Selection of priority areas
ONRI's priority areas comprise a combination of two primary elements: first, energy/environment areas relating to the national mission, and the optic sector needed by industry, and second, surface/interface and other pioneering, shared-foundation areas. ONRI is shifting emphasis to materials research, which, in light of this institute's potential, was deemed generally appropriate.
On its own initiative ONRI created an external committee called the "Osaka Industrial Technology Administration Advisory Group," by which it solicits outside opinions in determining institute operations, a laudable process from the perspective of running the organization. In consideration of future activities, ONRI should perhaps have a committee including knowledgeable persons from the outside which formulates medium- and long-term blueprints. It was the opinion of most Evaluation Committee members that ONRI need not make any changes or curtail in the priority research areas. However, some committee members expressed the opinion that ONRI should further narrow its focus on materials, and that it should henceforth assign priority to energy conversion, human life engineering, and environmental catalysts.
2) Selection of research themes
The research themes currently in progress have generally been appropriately chosen. Also the use of the virtual laboratry and other means to actively incorporate the ideas of young researchers and motivated researchers is commendable as a fitting way to proceed with research. It was noted that measures for flexibility, such as changing budgets and terms in accordance with theme progress, are also important.
3) Research system
While the research division/department system is good for deciding research areas and carrying on continuous research, the virtual laboratry system is also necessary for interdisciplinary research and creating new research areas. Therefore conjoint use of the ONRI division/department system with the virtual laboratry system and the Basic Integrated Special Research Office is deemed suitable. The virtual laboratry is a system that respects the researcher's enthusiasm for research and makes prompt responses possible. Thus with respect to research achievements along with of course precise evaluations of researchers themselves, there must also be evaluations of the virtual laboratry on a set term basis. ONRI conducts an evaluation including a major review of the virtual laboratry every three years with the participation of knowledgeable persons from the outside, with this system being deemed appropriate.
4) Securing, training, and evaluating researchers
The evaluation indicated ONRI efforts to hire superior young researchers through deciding each year's hiring plans and hiring researchers, including core researchers, for definite work terms, as it watches the progress of research in each priority area. Additionally, the advantages of the Kansai locale help the institute secure good researchers. But personnel flow is rather poor, as a limited number of researchers are hired each year. Also, providing motivated and capable young researchers with self-awareness and with a place to employ research funds freely is an important policy measure in the operation of a NRI. It is laudable how ONRI uses its virtual laboratry system to foster research leaders among its young researchers. Further, the director's discretionary authority is put to use in engaging young persons as research leaders and treating them well in the pursuance of research themes within priority research and priority basic research. This serves as an incentive to young researchers.
5) Research budget and buildings/facilities
The measures of ONRI and its executive staff to secure funding for research and facilities are highly commendable. Under its recent supplementary budget, the institute is building new buildings and facilities and updating existing ones. However, the evaluation pointed out that, from the private-sector point of view, it is perhaps difficult to keep up with rapid changes in business and society with improvements and updates at 10-year intervals.
6) Enhancement of research through interchange (joint research, etc.)
Overall, ONRI has better industry-academia-government ties and interchange than other NRIs. Especially in the areas of energy and environmental materials and cutting-edge shared-foundation research, ONRI maintains appropriate industry-academia-government links. Some evaluators, however, expressed the opinion that in order to achieve extensive reform of industry-government interchange, ONRI needs an institutional review that would make possible commissioned research (a new commissioned research system different from the present one) and a concurrent task system. Overall, ONRI actively conducts international interchange, but foreign commendations are not commensurate with the extent of interchange, the same level as other NRIs. To facilitate international interchange ONRI must expand the corresponding budget and provide facilities.
7) Research achievements
Regarding the number and quality of research papers released, the Evaluation Committee found that, in terms of number, ONRI is on a par with other Ministry of International Trade and Industry (MITI)-affiliated NRIs. With 150 full-time researchers, and considering co-authoring, there should perhaps be even more papers released. Evaluating quality would require reading the papers. Therefore, as on this occasion the committee lacked the documents and time, the evaluation did not consider the substance of papers.
In terms of the numbers of patents and other rights, and the extent of their use, ONRI stands above other NRIs. The institute should continue to place importance on patents. Important in evaluating patents are, of course, the number of applications, as well as how many are put into actual use, but statistics show that NRIs overall are tending to decline here. It is hoped that the Agency of Industrial Science and Technology (AIST) will endeavor fully to secure budgeting for domestic and foreign patent applications.
ONRI is about average among NRIs for commendations and awards. If possible the institute should also try to receive an international award. ONRI seems to be presenting about the same number of papers at academic conferences and the like as other AIST-affiliated research institutes. As with published papers, however, nothing can be said of quality without further investigation.
As indexes are needed for evaluation, ONRI should begin by formulating these on its own and creating indexes for quantitative integrated evaluation. Because self-evaluations are also required, researchers should determine their own evaluation indexes and use them to evaluate research achievements.
ONRI has plenty to show for its work on molten carbonate fuel cells (MCFCs), part of the national mission in the energy/environment area. It also has a number of major achievements in accommodating the needs of industry, and has made a bigger contribution to industry than other NRIs.
Arrangements for disseminating research achievements are insufficient in terms of assistance for patent applications, and need to be enhanced. Also needed are improvements in operating the system to relax controls on concurrent tasking.
Future Improvement Points for the Osaka National Research Institute
1) Selection of priority areas
Establish a body provisionally named the ONRI Advisory Panel and including knowledgeable persons from the outside. Use this body's opinions in reviewing and formulating the next medium- and long-term vision. The formulated vision (draft) should, as always, be submitted for deliberation to ONRI's existing external board (the Osaka Industrial Technology Administration Advisory Group).
2) Selection of research themes
ONRI shall continue working on a flexible evaluation and implementation system that actively incorporates creative research seeds based on the free thinking of researchers. For research themes under the director's discretionary authority, ONRI shall consider making it possible not only to effect flexible budget changes, but also to lengthen or shorten the research term in accordance with research evaluation results.
3) Research system
In order to perform an evaluation every three years, including a major review of the virtual laboratryoratry, with the participation of knowledgeable persons from the outside, ONRI in March 1998 established the Osaka National Research Institute Committee for Developing Research Evaluation Guidelines and Standards, comprised of mainly Institute Council members, thereby initiating the development of such guidelines and standards. Present conjoint use of the ONRI department/office system with the virtual laboratry system is assessed as being efficient, but the institute intends to give more in-depth consideration to the future of these systems, while keeping in mind such matters as the establishment of a research group system.
4) Securing, training, and evaluating researchers
ONRI hopes for understanding in that poor personnel flow is due not only to the insufficient efforts of the institute, but also to social factors. Henceforth ONRI shall consider the active use of a limited-term system for appointing researchers invited from the private sector, as well as researchers from both industry and academia, including exchanges of researchers between universities and ONRI, and corporate researchers loaned to ONRI.
5) Enhancement of research through interchange (joint research, etc.)
While conferring with the Agency of Industrial Science and Technology (AIST), ONRI shall appeal to involved government agencies and departments in order to make possible still further improvements. Especially with regard to concurrent tasking, ONRI has already asked AIST for improvements. ONRI also intends to work on securing a budget to facilitate international interchange, and shall request a budget that makes it possible to build a new Center for the Advancement of International Research, thereby providing facilities for such interchange.
6) Research achievements
It is essential to secure funding and develop the organization for patent applications, which are an effective way to disseminate information about research achievements. While conferring with AIST, ONRI shall make strong appeals to involved government agencies and departments, in order to make possible still further improvements.
by M.Taniguchi, Osaka Natl. Res. Inst.
