Evaluation of innovation networks based on standardized management system

Abstract:

The interference in a phenomenon as a National Innovation System (NIS) is complex, especially in developing economies. Some studies discuss the absence of tools and practical actions that allow a continuous improvement of these systems. In Brazil, there is a case of pioneering action developed by EMBRAPII, an agency for Industrial Research and Innovation. EMBRAPII has recently published a standard of management system based on business processes. That tool aims to provide guidance to support the management of R&D Institutes, partners of EMBRAPII. This paper proposes a model of evaluation as a mechanism to improve a network of R&D Institutes using that standard management system. The model is structured based on the integration of three key concepts: performance measurement system, evaluation processes, internal and external audits. The constraints and assumptions about the use of a model such as this are discussed. The main conclusion is that there are no similar models in literature and application of this model can be an important tool for developing countries that need to boost their innovation network.

Keywords:
National Innovation Systems; R&D institutes; Innovation; Evaluation of innovation networks; Internal audit; External audit

Resumo:

A intervenção em um fenômeno como um Sistema Nacional de Inovação (SNI) é algo complexo, principalmente em economias em desenvolvimento. Estudos constatam a ausência de instrumentos e ações práticas que permitam atuar no aprimoramento contínuo desses sistemas. No Brasil, alvo do estudo, há um caso de ação pioneira desenvolvida pela Empresa Brasileira de Pesquisa e Inovação Industrial (EMBRAPII), que publicou recentemente um padrão de sistema de gestão baseado em processos de negócio que serve de orientação para apoiar a gestão das instituições de pesquisa científica e tecnológica (ICT) parceiras da instituição. O artigo apresenta uma proposta de modelo de avaliação como mecanismo para a melhoria de uma rede de ICTs que utiliza o padrão de sistema de gestão, visando avanços na gestão da inovação e impactos positivos no SNI. O modelo está estruturado com base na integração de três conceitos principais: sistema de medição de desempenho, processos de avaliação, auditorias internas e externas. As limitações e hipóteses sobre a utilização de um modelo como esse são discutidas. As principais conclusões são que não há modelos similares na literatura e que a aplicação desse modelo pode ser um instrumento importante para países em desenvolvimento que precisam dinamizar sua rede de inovação.

Palavras-chave:
Sistemas Nacionais de Inovação; Instituições de pesquisa; Inovação; Avaliação de redes de inovação; Auditoria interna; Auditoria externa

1 Introduction

Governments from all over the world consider National Innovation Systems (NIS) essential to increase the innovative potential of countries. More than a decade ago, Lundvall et al. (2002) recognized increased interest on National Innovation Systems’ perspective and networks in Latin America, Asia and Africa.

R&D Institutes play a significant role on the innovation scenario of a country and its evolution relies on a set of technical and organizational skills. The Oslo Manual (OCDE, 2005) points out is also scientific and technology institutions that feed commercial innovation, providing scientific knowledge. They usually act on the precompetitive stage of the innovation process, intermediating new knowledge and technologies development to apply it on new products and technologies development (on the market).

Aligned with Lundvall et al. (2002) and Oslo Manual (OCDE, 2005), it is possible understand that one way to improve the NIS is directly related to operational improvement and expansion of capacity in providing services by R&D Institutes. The NIS, in these cases, shall improve itself on abilities such as: 1) plan technological strategies; 2) plan compatible business that support its actions; 3) manage portfolio of insights (demand) and its projects; 4) establish business processes which allow operational improvement; 5) develop its skills constantly.

For Viotti (2002), there are relevant differences among developed and under developed countries’ NIS. Liu et al (2014) made a comparative research in which the Brazilian NIS was suited on a group of emerging countries characterized by a strong focus on the process of knowledge production and failure on business benefit acquisition.

Arbix et al. (2005) showed that cooperation with R&D Institutes is an important feature for the Brazilian NIS development. On a study about exporting and innovative companies, with data from the main domestic foundations (PINTEC, SECEX and RAIS), it was detected among the Brazilian exporting companies, that released technological innovation, a higher concentration of companies with interest on cooperative arrangements and government investment, including R&D Institutes participation. Assad & Souza (2009), in specific case study, validate the importance of R&D Institutes involvement on the innovation of medical field.

Another specific characteristic of Brazilian NIS is that its potential could be more exploited if considered data showing several challenges faced by R&D Institutes when supporting the productive system. There are structural issues such as the legal framework regarding payment of services provided and foundations engagement (Jannuzzi et al., 2008), lack of clear definition of the researchers’ role (Assad & Souza, 2009) and intellectual property issues (Kamimura & Cornetta, 2011) that discourage investment and partnership. And managerial issues such as the need for a more professional operation and improvement of laboratories best practices and management models, as argued by Assad & Souza (2009) on medical field, and Oliveira & Telles (2011) on several industrial sectors.

The challenge then is how to spread management models and establish good practices, including techniques for prospecting, planning and developing technologies and products in R&D Institutes network. According to Chandler (1962), Penrose (2006), Rutman (1977), Bin (2008) and Chiesa et al. (1996), the evaluation of these innovation networks has a fundamental role.

A chance to investigate this issue is the recent case of the Brazilian Agency for Industrial Research and Innovation (EMBRAPII). The strategic plan related to industrial innovation established by Federal Government comprised among the initiatives, the foundation of EMBRAPII as an institution aiming to acting particularly on the expanding R&D Institutes activities. EMBRAPII was founded to accredit and manage a network of R&D Institutes able to prospect and develop innovative technologies, in association with the productive sector. Based on the differential of other similar networks as the Fraunhofer Institute in Germany and Carnot in France, EMBRAPII developed a standard of Management System specific to its network needs, called EMBRAPII Operational Excellence System (EOE System). The purpose according to the standard is to guide business processes of the EMBRAPII network and to be reference to establish a common language among the stakeholders in order to support EMBRAPII on qualification and monitoring accredited R&D Institutes. The EOE System is available at EMBRAPII website.

This article proposes an evaluation model for R&D Institutes network based on the EOE System. The proposal is designed to suit in particular to the Brazilian NIS, represented in this study by EMBRAPII innovation networks institutions. The main contribution conducted by an action-research is the model introduction and the discussion about its components and aspects that affect the development and evolution of practical instruments related to the improvement of EMBRAPII’s innovation network.

This paper is organized in six sections. The bibliographic review is presented on the following section. Section 3 contextualizes the case studied (and the research issue). On section 4 is explained and analyzed the proposed model, including notes about the research outline by means of the knowledge acquired on the literature. Section 5 contains final considerations and conclusions made by the research team.

2 Bibliographic review

2.1 Evaluation of innovation networks

The agents of a National Innovation System (NIS) and its type of performance and relationship are quite different among countries and regions. The OECD (2009) mentions some principles that should be applied by those who work with innovation, such as seeking and arranging knowledge in a globally way, integrating cooperative networks and compounding partnership, and handle the needs of developing countries (OECD, 2009).

Intervention in a phenomenon as a National Innovation System is something complicated, particularly on developing economies, among which Brazil is embedded. According to Andreassi et al. (2006) on developing countries the innovation systems and innovation networks are immature. According to Viotti (2002), there are relevant differences on technical changes processes of these countries compared to developed countries. Furthermore, Lundvall et al. (2002) indicate absence of concepts, instruments and practical actions that allow continuous improvement of NIS. Levén et al. (2014) also points out that is essential to have a common mechanism for those involved on the network, to establish, promote and deploy processes in an innovation network. This article is focused on the absence of instruments and practical actions that allow continuous improvement of NIS.

Nobelius (2004) suggests that one of the main difficulties for innovation management is turning research works into commercial products, such as turning creation into innovation. Some of these difficulties derive from internal and external communication problems, cultural barrier, use of improper processes and methods not necessarily structured by superior and mature technologies. Such difficulties require an efficient management of innovation process (Nobelius, 2004; Schulz et al., 2000).

In the case of Brazil, recently, Liu et al. (2014), while studying the efficacy perspective on innovation systems, pointed out the country inside a group of emerging countries characterized by a strong focus on knowledge production process, along with India, Mexico, Thailand and Malaysia. The authors highlighted the group capacity to manage costs with R&D to uphold a reasonable productivity level on industry and services. According to the authors, despite being strong on knowledge production, these countries do not show the same strength on application and acquisition of business benefit. Oliveira & Telles (2011) based on previous experiment, highlighted that it is necessary that the R&D institutions are recognized by their technical skills on the project management field and be highly efficient on its management as a whole. The aspects presented along with EMBRAPII study opportunity led the research team to consider Brazil as a country target of study.

Chandler (1962) and Penrose (2006) characterize the management activity as a set of decisions related to coordination, evaluation and planning of the company’s work and allocation of its resources. In this context, evaluation is, first of all, a process that applies scientific procedures in order to gather reliable and valid evidences about how and to what extent certain activities generates certain effects or results (Rutman, 1977). Among the benefits of an evaluation stands out priority identification and provision of structural conditions to support activities conducted by the agents that compound an innovation system (Bin, 2008).

Lengrand (2006) discusses about four evaluation types intended to innovation programs: 1) Ex-ante evaluation: taken prior to implementation; 2) Midterm evaluation: review of progress or results at some point of their development; 3) Real time evaluation: detailed monitoring in the execution; 4) Ex-post evaluation: analyze results after being carried out. This typology was later accepted for creating the evaluation model of EMBRAPII innovation network.

Brazilian studies on Science and Technology (S&T) evaluation have been conducted in a comprehensive way. As in Salles-Filho et al. (2010), that show a study about impact evaluation in innovation programs. Bin et al. (2003) and Paulino et al. (2007) study technological programs evaluation for environmental impact. Furtado et al. (2002), explore S&T indicators for evaluation of university and industry cooperation programs. They discuss the networks evaluation in a broad way, but do not present instruments that guides how to improve each aspect of the organization, as the standards of operation and business processes of innovation network, for instance. Therefore, the main issue would be how to establish and promote an operational evaluation model that guides R&D Institutes in developing countries, ensuring that their activities are in accordance with the common goal of the innovation network they belong to. Such model should allow a detailed monitoring of R&D Institutes operation.

The overview about evaluation principles of R&D Institutes presented by Zackiewicz (2005) assisted the development of this study on elaborating an evaluation model for research institutions. On the Chart 1 is presented some comments of the authors related to the selected literature notes.

Chart 1
Principles of S&T evaluation considered by the author.

Zackiewicz (2005)	Author’s comments
To evaluate is to interpret an attribute by means of a criterion	This proposal suggests the use of EOE standard as the main criterion for R&D Institutes to perform an evaluation of its business processes and its operation as a whole. The EOE standard will be presented on section 3.
To decide is to take an action from an evaluation	This proposal suggests three evaluation aspects to support decision making on institutions: performance measurement system and internal and external evaluation. These will be presented and debated on subsections 4.1 and 4.3.
All evaluation methods are social constructions	This proposal suggests there is a need to build groups or teams that can learn, make evaluations and help to develop the EOE standard. This aspect will be presented and debated on section 4.4.

As previously mentioned, managers of research institutions face a challenge. It means finding ways to disseminate and use standards, such as the EOE standard, to improve R&D Institutes network they assist or have interest on developing, combining the different instruments of networks evaluation for this purpose.

Therefore, it is necessary to know the most common evaluation models, which shall be presented hereinafter.

2.2 Performance measurement system

The development of the Performance Measurement System (PMS) remains a topic of increasing concern for both the Academy and Industry (Nudurupati et al., 2011, p. 280; Valmohammadi & Servati, 2011, p. 494; Neely et al., 2005, p. 1228). In the Academy, the performance measurement is widely discussed with the proposition of PMS new models and its development methods. However, many authors use different terms for the same concepts. The concept of performance measurement, for instance, has different definitions for Neely et al. (2005, p.1229), Bourne et al. (2002), Atkinson (2000), Bititci et al. (2000), Kaplan & Norton (1997).

Therefore, is important to present some concepts for a better understanding of the topic. The definitions adopted in this paper are in accordance with Neely et al. (2005, p. 1229):

• Performance measurement. It can be defined as the process of measuring the efficiency and effectiveness of an action.

• Performance indicator (Originally, Neely et al. (2005, p. 1229) use the term performance measurement, however it was adopted performance indicator, most common word on the field). It can be defined as a metric used for measuring the efficiency and/or effectiveness of an action.

• Performance Measurement System (PMS). It can be defined as a set of performance indicators used for measuring both efficiency and effectiveness of actions.

According to Gerolamo (2003, p. 59), among the PMS models proposed between the 1980s and 1990s the most researched and cited models were Performance Pyramid, Cross & Lynch, (1989); Balanced Scorecard (BSC), Kaplan & Norton, (1997); and Performance Prism, Neely et al. (2002).

In the specific case of performance evaluation for productive arrangements, operating on cooperation networks it can be cited Cardoza Galdamez (2007) who studied clusters of small and medium enterprises. Hansen & Oliveira (2009), who developed a model for systematic evaluation of competitive performance of productive arrangements. It is also highlighted by Lenz’s (2007), which studies about horizontal cooperation networks. According to Lenz (2007) the Prism and the Balanced Scorecard models of performance may be basis for adjustment of the measuring performance of individual companies for horizontal cooperation networks.

The purpose of this choice is related to the Prism model focused on obtaining stakeholders’ satisfaction, considered closer to the networks reality, in which the agents (network participants, government, suppliers, and costumers) have significant role on its operation, setting a table of wide and complex social relationships. However, the prism does not suggest a way of deploying prospects that take into account an easy usage on practice, such as the Balanced Scorecard. Due to these characteristics objectives, indicators, goals and initiatives of the model proposed by Lenz (2007) followed Kaplan & Norton (1997) recommendations.

Thus, the perspective of stakeholders’ satisfaction of Neely et al. model (2002) and objectives outcome of Kaplan & Norton (1997) were chosen to compose the model proposed by Lenz (2007) and it guides the outline of this proposal.

2.3 Audit in innovation networks

The performance measurement is useful but is only one part of the solution, according to Chiesa et al. (1996). For the authors, it is essential to understand how it emerged and to have continuous improvement, it is necessary to review not only innovation skills of an institution, but the processes through which innovations are developed and exploited.

Chiesa et al. (1996) state that performance auditing helps on identifying problems and needs, but does not explain why are gaps on current performance and no action plan to fill the gaps is provided. As defined in Brazilian rule NBR 19011: 2002 (ABNT, 2002), audit is a systematic, documented and independent process for gaining audit evidence and objectively evaluating it, in order to determine to which extent criteria of process are met and, according to the author, meets these needs.

Authors like Hammer (2007), developed specific frameworks of process auditing to assist executives on planning companies’ changes, make progress and identify barriers. Chiesa et al. (1996) show a framework for technical innovation management auditing. The model approaches management processes and organizational mechanisms over which innovation is performed and provide basis for a detailed auditing of current innovation practices.

Yam et al. (2004) recorded the auditing executed on 213 Chinese companies in Beijing, which assisted on evaluating technological innovation skills and its impact on companies’ competitive performance. Boly et al. (2014) used an evaluation methodology of innovative processes in 39 small and medium companies in France in order to recommend improvement actions.

To perform evaluation of a program, according to the Brazilian Association of Technical Standards – ABNT 17021-2007 (ABNT, 2007), three conditions are prerequisites and must be tested in advance: 1) a program clearly defined; 2) objectives and results clearly specified; 3) an intelligent scheme of causalities linking program to objectives and results. This work considers EMBRAPII context and its Management System standard – EOE System, objectives and strategical goals, to be presented later on sections 3 and 4.

ABNT 17021-2007 (ABNT, 2007) describes concepts of management systems certification, useful for this work, and according to the standard can provide independent statement that the organization’s management system: 1) is in accordance with specified requirements; 2) is able to reach its declared policies and respective objectives with coherence; 3) is effectively implemented. According to the standard, accordance evaluation such as the certification of a management system provides significance for the organization, its customers and interested parties.

3 EMBRAPII and the context of proposal application

3.1 EMBRAPII presentation

The Brazilian Agency for Industrial Research and Innovation – EMBRAPII – is a result of a RD&I program established by the Federal Government in 2013. It seeks to fit the gaps of RD&I programs and to consolidate itself as an agent of innovation with high efficiency. Its goal is to register and manage a network of R&D Institutes capable of prospecting and developing innovative technologies. The program is similar to those existing in countries like Germany with the Fraunhofer Institutes, France with the Carnot Institutes and England with Catapult Institutes.

One of the challenges faced by EMBRAPII was the issue identified by Liu et al. (2014) and Oliveira & Telles (2011) related to the research institutions on the Brazilian NIS with low experience, regarding technology transfer. As mentioned on section 2, Brazil is characterized by a high production of “papers”, but few insertion and impact of this production on the commercial and industrial development. Therefore, besides composing the network, as in French and German cases, there is the extent that on Brazilian program is essential to incorporate the issue of developing institutes belonging to the network in order to impact NIS as a whole.

As mentioned by Oliveira & Telles (2011), cooperation experiences between university-company, from 2008, by the IPT (Instituto de Pesquisas Tecnológicas) focused efforts to identify relevant themes for innovation, to encourage the creation of an office of projects and planning of new technologies. However, still at pilot phase, EMBRAPII faced the absence of a consolidated model capable of providing conceptual basis to guide operational activities for its Units, shown on section 2. Such absence took EMBRAPII, along with a researcher group, to develop a specific innovation management system to its accredited Units, the R&D Institutes. The system was called EMBRAPII Operational Excellence System (EOE System) and is described as follows.

3.2 EOE system - general overview

The introduction stage of an innovation, capable of generating competitive advantage, initiates by identifying needs and opportunities and, in general, takes place in the company partner of the EMBRAPII System, which develops projects with the accredited R&D institutions.

EMBRAPII elaborated an operational standard to make research institution a strategic partner that operates where the NIS needs the most: on the intermediate stage. This stage is placed between the development of the first proofs of concepts and the development of products and processes, in the context of companies able to put the final product for sale, with embedded technology. Therefore, EMBRAPII performs a wide range of project types, always in partnership with companies, in order to facilitate introduction of new technologies in products, processes or business models.

Based on prior knowledge and according to studies of authors of the Innovation, Technology and Product Management field (Clark & Wheelwright (1993), Clausing (1993), Cooper (2006), Creveling et al. (2003), Schulz et al. (2000), Sheasley (2000), EMBRAPII sought to address actions for the R&D Institutes’ improvement on the EOE standard, providing them capability to: 1) plan technological strategies; 2) plan compatible business that support its actions; 3) manage portfolio of insights (demand) and its projects; 4) establish business processes which allow operational improvement; 5) develop its skills constantly.

The EOE system is a reference standard that can be used as a tool from which EMBRAPII shares and encourages practices improvement in all Units in order to obtain an efficient management. The purpose of using this standard is to guide business processes’ network and establish a common language among actors to support EMBRAPII system on the qualification and monitoring of accredited R&D Institutes.

The system consists of three models: a model of actors, a model of business processes and a set of Key Performance Indicators (KPIs).

• EOE actors is a model that describes roles or a set of responsibilities explicitly identified on the Organizational Structure of each EMBRAPII Unit.

• EOE business processes is a model that describes a set of processes or practices that produce verifiable results (deliveries) and valuable for the costumers of EMBRAPII Unit. It was selected and developed four main processes: 1) Business Opportunities Development; 2) RD&I Management; 3) Communication; 4) Financial and Administrative Management.

• Key Performance Indicators are the performance indicators used to show how well EMBRAPII Unit goals are being achieved.

Figure 1 represents the general overview of the interaction between EMBRAPII and its Units by means of the components of EOE System (For further information access EMBRAPII website).

Figure 1
EOE System - general overview.

This work considers that any organization that drives research institutions, such as EMBRAPII, can develop a standard management system for its innovation network, as EMBRAPII did with its EOE System. However, it is important that such organizations are concerned about the use of these standards and their application, disseminating and using tools to ensure their implementation.

The proposal of R&D Institutes network evaluation model that uses predetermined management systems is presented on section 4.

4 Building an evaluation model of R&D Institutes network

The work was organized in action-research stages. On the first stage, a research team was built, which one Master’s Degree student and two researchers from University of São Paulo, specialists on Innovation Management and Product Development, composed. The researchers have broad experience as users of Brazilian NIS development programs as well as of cooperative projects development among universities and enterprises.

The first step after the team building was the analysis of literature and prior knowledge of the team, which enabled proposing a set of requirements and general purposes of the model: 1) Assure the agent organization’s and its accredited institutions’ growth; 2) Be a guide for continuous improvement of the institutions management and their business processes; 3) Serve as a common language for using standardized management systems among actors of an innovation network composed by research institutions.

Simultaneously, a detailed bibliographic review was made by seeking articles proposing evaluation of innovation networks, performance measurement systems and audit in innovation networks, as mentioned on section 2. In this context stood out the urge to have means contributing for technological development and for building a business environment beneficial to innovation in Brazil.

On the second stage, the EOE System was studied and identified as a guidance tool for Brazilian R&D Institutes network and defined as a good example to develop the study. After all, as mentioned on section 3, it has the required characteristics for being a model for evaluation process. The result analysis of the selection of the most appropriate concepts and tools for evaluation model was performed among pairs of the research team that sought to bring together the most relevant aspects needed for the proposal.

Applying the knowledge acquired, on the third stage was selected and developed four main components to form the evaluation model.

A summary of activities relating to the evaluation model establishment is represented on Chart 2.

Chart 2
Activities relating to evaluation model establishment.

Stage 1	Building the research team
	Analysis of literature and the team’s prior knowledge
	Identifying needs
	Requirements proposal and general purposes of the model
Stage 2	Study of EOE – basis for evaluation process
	Analysis of concepts and tools for evaluation model
Stage 3	Development of the evaluation model

The components of the evaluation model are presented and detailed hereinafter from subsections 4.1 to 4.4.

1. 1

   Performance Measurement System (PMS). It is point out as a set of initial performance indicators used for measuring both efficiency and effectiveness of actions taken in each research institution and on the system as a whole.

2. 2

   Internal Evaluation Process. It was thought to be an improvement process established by the research institution, according to its needs and experience, and to ensure the continuous search for identification of problems and their improvement in daily activities.

3. 3

   External Evaluation Process. It was developed as a set of follow-up actions of the results of each R&D Institutes and its evolution, run by the agent organization or by Independent Organization at its order. Three objectives have been proposed: 1) evaluate the adoption of institution processes to the proposed standards in management systems, such as EOE System; 2) evaluate efficiency on establishing and achieving goals of resources appliance; 3) evaluate actions efficiency and its impact on society.

4. 4

   Technical group of Management System improvement. A technical group to discuss and introduce development for the management system as a whole, based on experience acquired by the institutions, in order to support the agent organization on continuous improvement of its management system.

Figure 2 shows a general overview of the proposed evaluation model. It is suggested that the first three components are independent principles of the evaluation model, with distinct features. Component 4 is understood as transverse to the others and, aligned with Bin (2008), corroborates to identify priorities and to provide structural conditions for basing actions to be taken by agents of an innovation system.

Figure 2
General overview of R&D Institutes networks evaluation model (own elaboration).

4.1 Performance measurement system

The adoption of a performance measurement system was conceived to be a tool enabling a continuous evaluation of the research institutions performance in relation to achieving the managing organization’s goals. It is proposed that its operation concerns regular update and evaluation of the performance indicators of business processes, predetermined by rules of management system, as the EOE system cited on section 3, for instance.

Aligned to the proposals defined by Neely et al. (2002) on section 2, it is suggested that the goal of such tool is to guarantee ongoing monitoring so the agent organization can act and give the necessary support in case of difficulty or delay on the execution of actions made by the accredited institutions. It also shall assist on transparency of the system, showing society a sum up of projects and progress in each institution in the face of the investment made.

The verification shall be made by follow-up of information that creates the indicators, which might allow an analysis of the environment and operational results of accredited institutions. As mentioned on section 2 and according to Kaplan & Norton (1997) instructions, the indicators shall be aligned to the organization’s strategic goals that run on promoting innovation in their countries. As on the Brazilian case of EMBRAPII, the indicators are aligned to the goals in its management contract with the federal government (For further information access EMBRAPII website (EMBRAPII, 2014)).

4.2 Internal evaluation process

Business processes are among the requirements for adaptation to management systems, such as the EOE system. It is important that research institutions testify the existence of structured processes for continuous improvement of its business processes in the case studied, as the processes of EOE System cited in section 3.

The methods and techniques for improvement are up to the institutions. However, there is a set of actions for use of management system standards, as instructed:

Submission of explanatory documents about the institutions internal system and how it is related to the current standard processes;

• Documents of changes made on the Management System;

• Establishment of an specific internal organizational structure (Committee, group, team and department) responsible for Improvement Program;

• Incorporation of training activities on continuous improvement techniques and tools and practices of predetermined standards.

It is suggested that specific training activities on such standards, including processes, indicators and information systems used by the research institution are developed along with the agent organization, considering the following aspects: objectives, target audience, goals, courses and events grid, calendar for implementation, and responsible for training.

4.2.1 Continuous improvement system

It is important that research institutions take into consideration a management system able to ensure continuous search to identify problems and result in improvements. Thus, is suggested a plan development for implementing this system. To be considered as such, a Continuous Improvement System shall consider four aspects.

• Philosophy. Establish an improvement model and mission, vision and principles for continuous improvement on the organization.

• Processes. Determine processes standard for improvement projects, as the traditional DMAIC, PDCA or another one already used by the organization, with tools and actions.

• People. Development of activities that ensure employees’ training to embody the mindset about management philosophy, having direct impact on the organization’s culture.

• Problems solution. Setting the organization for detection, prioritization and solution of problems on processes.

The definitions of improvement system, optimization process and continuous improvement of processes can be seen on standards such as the Brazilian ISO/NBR 9001:2005 (ABNT, 2005) and on the outline CMMI-DEV, v 1.2 (SEI, 2006). It shall be mandatory to establish an internal continuous improvement system for all research institutions part of an innovation network. The methodology chosen and the type of standards used can be chosen by the R&D Institute. Below are some examples of improvement models that can be used as reference.

• Maturity model. A model holding processes’ essential elements, described under a development improvement path, from immature processes to mature, disciplined, with improved quality and effectiveness processes (SEI, 2006).

• Six-sigma and six-sigma teams. An approach ruled for quality improvement centered in evaluation methods and statistics measurement.

• Kaizen. continuous improvement (incremental) focused on activities that do not add value, that is, waste removal. It relies on Kaizen Events which are short period events with fast results commitments.

• Problem-Solving Method. It is a dynamic process seeking to solve a certain situation. It aims to increase the likelihood of successfully resolve a situation where a problem has arisen. Several concepts are used to ground its practice, being PDCA (Plan, Do, Check, Act) the most known.

• Good Laboratory Practice (GLP). It is a quality system regarding organizational process and conditions in which non-clinical studies related to health and global environment are planned, performed, monitored, registered, filed and reported.

4.2.2 Organizational structure for continuous improvement

To ensure implementation of internal improvement system as well as to assist predetermined rules and standards requirement, it is suggested that research institutions show an organizational structure composed by qualified professionals with certifications and specific coaching on the adopted methodology. The specific definition is up to each institution.

The minimum requirement is to have a structure formed by areas, groups and/or individuals who have as functions well defined responsibilities regarding continuous improvement, establishing (For detailed explanation see ISO/NBR 9001:2005 (ABNT, 2005), CMMI-DEV, V 1.2 (SEI, 2006) and Meier & Liker, 2007):

• Improvement teams. Team responsible for planning and execution of activities of process improvement in organizations, according to the process action plan.

• Improvement program. A set of related projects and supportive infrastructure including purposes, methods, activities, plans and measures of success.

• Quality department. Specific sector of the organization that holds staff and specific expertise to manage the improvements system.

• Manager committees of process or of continuous improvement. Group of people responsible for evaluating, approving or rejecting proposed changes and for ensuring implementation of approved changes.

The structure can be simplified or sophisticated according to research institution’s needs.

4.2.3 Individual performance evaluation program

The performance management is characterized as a subsystem of personnel management; these people in turn, at research institutions, are in possession of knowledge and skills applied in business processes and innovation projects. Thus, the evaluation of individual performance is closely related to operational performance of the agent organization and of innovation network as a whole, including its processes and indicators, contributing to achieve its goals.

Research institutions can have an individual performance evaluation system simplified according to their realities. It is essential to establish an agreement on the purpose of its development within R&D Institutes since these systems can have different aims. Among them, highlights work scheduling, inputs acquisition for training and staff development, team potential identification, skills identification, feedback about results and recognition through awards.

The performance evaluation process involves planning, monitoring and work evaluation activities, which main goal is to improve people performance, besides improving work quality and life quality within the institution.

Some performance evaluation models are mentioned below for exemplification purposes.

• Participatory evaluation for objectives. This method, according to its authors, is democratic, participatory, involving and encouraging. It consists of six moments: establishment of agreed objectives, personal commitment to achieve these objectives, negotiation on allocation of means and resources to achieve the objectives, performance, monitoring of results and comparison with established objectives, intensive retroaction and joint and ongoing evaluation.

• 360^o or 180^o evaluation. Performance evaluation taken in circular or semicircular way by all elements that keep any sort of interaction with the individual evaluated. The method is based on great participation of all members of the team; individual performance information is gathered from everyone around.

The research institution can choose these or other methods to evaluate its staff. It should not be mandatory to establish an individual performance evaluation program, but it is recommended due to benefits it can provide, in particular for developing the institutions skills.

4.3 External evaluation process

It is suggested that every research institution, acting on innovation networks and according to predetermined management system standards, participate on external evaluation actions carried out by three mechanisms: information system, technical auditing and Certification process.

4.3.1 Information system

The agent organization, as EMBRAPII, shall use an information system for managing information flow, updating performance indicators of management system and carrying out competitive benchmarking among accredited institutions. This requirement attendance shall occur without making substitutions or any other changes on current information systems in institutions. For this system to work is fundamental to perform the following processes: 1) Implement information flow; 2) Hold a data repository.

4.3.2 Technical auditing

It is a specific auditing on Standard Management System, or on part of it, defined and performed ad-hoc by the agency (EMBRAPII in this case), or a third registered organization, according to specific needs. The audit can be made through technical visits and/or analysis of documents particularly required.

The visits have explanatory nature, aiming to gather detailed information on management system of institutions, specific projects, evaluation of problems, improvement and action plan. The research institutions shall keep update registers related to processes provided on standard rule for follow-up like this, following guidelines of subsection 4.2.1 Continuous improvement system.

These audits are addressed to be executed by specialists of standard rules, as the case of EOE System, and it can also be performed by expert professionals on technologies related to the projects, depending on the evaluation scope of the audit in question.

4.3.3 Management system certification auditing

This routine was conceived as a mechanism for closer monitoring, in case of detection of inconsistencies, problems or opportunities for improvement on certified research institutions. It shall certify its Management System in order to guarantee accordance with the standards of the agent organizations. Such certification would be a mean of ensuring that institutions introduced the necessary actions to manage relevant aspects of its activities.

Certification will provide independent demonstration that the management system of research institution: a) conforms to specified requirements, b) is capable of achieving a coherent goals and c) is implemented effectively, providing value to own agency organization, its customers and stakeholders, as referenced in the literature and presented in section 2.

The agency can hire a third party auditing to evaluate, grant and maintain the certification for the R&D Institutes. It can may use external institutions base on the trust on the principles of audits of this nature, ensuring in this way:

• Integrity. Engagement of professionals acting with impartiality, honesty and diligence.

• Confidentiality. Discretion on the use and protection of received information.

• Independence. Feedback and evaluation of independent auditors, who are exempt of tendencies and conflict of interest.

• Evidence-based approach. Rational method to achieve reliable and verifiable conclusions.

• Fair presentation. Truthfully and accurately reporting.

• Responsibility. Implies evaluating sufficient objective evidence on which it is possible to ground a certification decision.

It is advised to have specific documentation showing the certification scope and ensuring that all people, processes and fields that perform projects with the research institutions are involved.

4.4 Technical group of Management System Improvement

As mentioned on section 2, evaluation of research institutions is a social construction. In this regard, technical groups of improvement are designed to ensure learning and improvement of the standard used by different agents part of the innovation network.

On this proposal is considered essential that each institution designate one or more members to participate of the Technical Group of Management System Improvement. On the case studied, for EOE System. This group is an executive board whose mission is counseling the Agency (in this case, EMBRAPII) in issues related to the standard established by them. This board’s nature is consulting and its scope is technical. It shall manifests itself under technical aspects of the standard such as indicators, principles, processes, recommended practices and other standard’s contents.

The goal is to transfer the experience gained by R&D Institutes with the management system in order to subsidize the Agency in continuous improvement the Standard. The existence of the group can help the changes are in line with the expected changes to the internal improvement system of institutions.

5 Final considerations

This paper proposed an evaluation model of research institution networks, based on a mix of internal and external evaluation processes, aligned to a Management System Standard. The model aims to support improvement in National Innovation System’s development. Its idealization emerged from previous experience and bibliographic review. It was considered a lack of instruments and practical actions regarding NIS improvement and the significance of innovation networks evaluation and its benefits as main aspects for building the proposal. Principles of evaluation taken from literature were use as basis to justify the use of EOE System Standard as main evaluation criterion and to develop its main elements.

Knowledge of S&T evaluation approaches, performance measurement systems and auditing in innovation networks were essential to elaborate the research. As mentioned on section 2, it was considered: 1) use of Management System Standards as main criterion for R&D Institutes to perform an evaluation of its business processes and its operation as a whole; 2) use of three evaluation elements to support decision making in institutions, such as: performance measurement system and internal and external evaluation processes; 3) the need to build groups or teams that can learn, perform evaluations and help with the develop of Management System Standard adopted by networks, such as the example of EOE System, used by EMBRAPII.

The use of the model as well as the standardization process, guided by management systems such as EOE System, require institutional support at the R&D Institutes, so that the necessary resources are allocated for this project. This aspect can be considered a limitation to the model success. Continuity of this work shall enable new adjustments until it becomes as good as possible to their purposes. The research team intends to continue monitoring these experiences and hopes that this article may encourage other research groups to investigate this issue, following this case or replicating it in other countries.

The model can be used in similar applications, such as in networks of university laboratories, technological parks or even R&D Institutes singly. If its credit is verified, it can still generate the creation of a national certification, and international then, in third party type. In addition to the benefits cited for the development of research institutions, the Standard would offer an interesting mechanism to guide companies in building partnerships for technological development by providing them with an assessment of the preparation in terms of quality of services of these institutions.

Given the absence of similar models, the paper presents this new perspective that can open a new line of research in technology management: on evaluation and auditing to management innovation. The model deserves future research effort in the sense that it can complement it with the possibility of creating a research institutions management platform, thus contributing to the adjustment of the innovation network strategies.

• Financial support: CNPq and EMBRAPII.

Referências

• Andreassi, T., Campanário, M., Stal, E., & Sbragia, R. (2006). Inovação. Como vencer este desafio empresarial São Paulo: Clio.
• Arbix, G., Salerno, M., & De Negri, J. (2005). O impacto da internacionalização com foco na inovação tecnológica sobre as exportações das firmas brasileiras. Dados, 48(2), 395-442. http://dx.doi.org/10.1590/S0011-52582005000200006
  » http://dx.doi.org/10.1590/S0011-52582005000200006
• Assad, A., & Souza, R. (2009). Challenger of Innovation in the area of health: learning in the ongoing debate. Cadernos de História da Ciência, 5(2), 9-19.
• Associação Brasileira de Normas Técnicas – ABNT. (2002). NBR/ISO: 19011: Diretrizes para auditoria de sistemas de gestão da qualidade e/ou ambiental Rio de Janeiro: ABNT.
• Associação Brasileira de Normas Técnicas – ABNT. (2005). NBR ISO 9000: Sistema de Gestão da Qualidade - Fundamentos e Vocabulário Rio de Janeiro: ABNT.
• Associação Brasileira de Normas Técnicas – ABNT. (2007). NBR/ISO: 17021: Avaliação de conformidade. Requisitos para organismos que fornecem auditorias e certificação de sistemas de gestão Rio de Janeiro: ABNT.
• Atkinson, G. (2000). Measuring corporate sustainability. Journal of Environmental Planning and Management, 43(2), 235-252. http://dx.doi.org/10.1080/09640560010694
  » http://dx.doi.org/10.1080/09640560010694
• Bin, A. (2008). Planejamento e gestão da pesquisa e da inovação: conceitos e instrumentos (Tese de doutorado). Instituto de Geociências, Universidade Estadual de Campinas, Campinas.
• Bin, A., Salles-Filho, S. L. M., Paulino, S. R., & Rodrigues, G. S. (2003). Impactos ambientais na agricultura: um método de avaliação de programas tecnológicos. Revista Ciências Administrativas, 9(1), 76-83.
• Bititci, U. S., Turner, T., & Begemann, C. (2000). Dynamics of performance measurement systems. International Journal of Operations & Production Management, 20(6), 692-704. http://dx.doi.org/10.1108/01443570010321676
  » http://dx.doi.org/10.1108/01443570010321676
• Boly, V., Morel, L., Assielou, N. D. G., & Camargo, M. (2014). Evaluating innovative processes in french firms: methodological proposition for firm innovation capacity evaluation. Research Policy, 43(3), 608-622. http://dx.doi.org/10.1016/j.respol.2013.09.005
  » http://dx.doi.org/10.1016/j.respol.2013.09.005
• Bourne, M., Neely, A., Platts, K., & Mills, J. (2002). The success and failure of performance measurement initiatives. International Journal of Operations & Production Management, 22(11), 1288-1310. http://dx.doi.org/10.1108/01443570210450329
  » http://dx.doi.org/10.1108/01443570210450329
• Cardoza Galdamez, E. V. (2007). Proposta de um sistema de medição de desempenho para clusters industriais de pequenas e médias empresas (Tese de doutorado). Escola de Engenharia de São Carlos, Universidade de São Paulo, São Carlos. http://dx.doi.org/10.11606/T.18.2007.tde-05032008-124613
  » http://dx.doi.org/10.11606/T.18.2007.tde-05032008-124613
• Chandler, A. D. (1962). Strategy and structure: chapters in the history of the American enterprise Cambridge: The MIT Press.
• Chiesa, V., Coughlan, P., & Voss, C. A. (1996). Development of a technical innovation audit. Journal of Product Innovation Management, 13(2), 105-136. http://dx.doi.org/10.1016/0737-6782(95)00109-3
  » http://dx.doi.org/10.1016/0737-6782(95)00109-3
• Clark, K. B., & Wheelwright, S. C. (1993). Managing new product and process development: text and cases New York: The Free Press.
• Clausing, D. (1993). Total quality development: a step-by-step guide to world-class concurrent engineering New York: American Society of Mechanical Engineers.
• Cooper, R. G. (2006). Managing technology development projects. Research Technology Management, 49(6), 23-31.
• Creveling, J.L., Slutsky, D., & Antis, C., Jr. (2003). Design for six sigma: in technology & product development New Jersey: Prentice Hall.
• Cross, K., & Lynch, R. (1989). Accounting for competitive performance. Journal of Cost Management for the Manufacturing Industry, 3(1), 20-28.
• Empresa Brasileira de Pesquisa e Inovação Industrial – EMBRAPII. (2014). Sistema de Excelência Operacional EMBRAPII (EOE) Brasília: EMBRAPII. Recuperado em 10 de novembro de 2014, de www.embrapii.org.br
• Furtado, A. T., Terra, B., Passos, C. A. S., & Plonsky, G. A. (2002). Indicadores de C&T para avaliar os Programas de Cooperação entre Universidade e Indústria: Uma Análise do PADCT III. In: Anais do XVII Simpósio de Gestão da Inovação Tecnológica (15 p., CD-ROM). São Paulo: NPGT-USP.
• Gerolamo, M. C. (2003). Proposta de sistematização para o processo de gestão de melhoria e mudanças no desempenho (Dissertação de mestrado). Escola de Engenharia de São Carlos, Universidade de São Paulo, São Carlos.
• Hammer, M. (2007). The process audit Canada: Harvard Business Review.
• Hansen, P. B., & Oliveira, L. R. (2009). Proposta de modelo para avaliação sistêmica do desempenho competitivo de arranjos produtivos: o caso do arranjo coureiro-calçadista do Vale dos Sinos. Produto e Produção, 10(3), 61-75.
• Jannuzzi, A., Oliveira, T., & Teixeira, C. (2008). Análise da interação de uma Instituição Científica e Tecnológica (ICT) com o setor produtivo: o caso do Instituto Nacional de Tecnologia (INT) Rio de Janeiro: INT. Recuperado em 5 de novembro de 2013, de http://biblio.int.gov.br/phl82/INT_DOCELE/Analise%20da%20Intera%C3%A7%C3%A3o.pdf
  » http://biblio.int.gov.br/phl82/INT_DOCELE/Analise%20da%20Intera%C3%A7%C3%A3o.pdf
• Kamimura, Q. P., & Cornetta, V. K. (2011). Inovação e propriedade intelectual no setor farmacêutico: desafios governamentais e efeitos na saúde pública. In Anais do XXXI Encontro Nacional de Engenharia de Produção (pp. 1-9). Rio de Janeiro: ABEPRO. Recuperado em 5 de novembro de 2014, de http://www.abepro.org.br/biblioteca/enegep2011_TN_STO_135_855_18348.pdf
  » http://www.abepro.org.br/biblioteca/enegep2011_TN_STO_135_855_18348.pdf
• Kaplan, R. S., & Norton, D. P. (1997). A estratégia em ação: balanced scorecard (26 ed.). Rio de Janeiro: Campus.
• Lengrand, L. (2006). Smart innovation: a practical guide to evaluating Innovation Programs Brussels-Luxembourg: ECSC-EC-EAEC. 199 p.
• Lenz, G. S. (2007). Medição de desempenho: uma proposta de sistema de avaliação de performance para redes horizontais de cooperação (Dissertação de mestrado). Faculdade de Administração, Contabilidade e Economia, Pontifícia Universidade Católica do Rio Grande do Sul, Porto Alegre.
• Levén, P., Holmström, J., & Mathiassen, L. (2014). Managing research and innovation networks: evidence from a government sponsored cross-industry program. Research Policy 43(1), 156-168. http://dx.doi.org/10.1016/j.respol.2013.08.004
  » http://dx.doi.org/10.1016/j.respol.2013.08.004
• Liu, J. S., Lu, W. H., & Ho, M. H. C. (2014). National characteristics: innovation systems from the process efficiency perspective. R & D Management, 45(4), 317-338. http://dx.doi.org/10.1111/radm.12067
  » http://dx.doi.org/10.1111/radm.12067
• Lundvall, B.-A., Johnson, B., Andersen, E. S., & Dalum, B. (2002). National systems of production, innovation and competence building. Research Policy, 31(2), 213-231. http://dx.doi.org/10.1016/S0048-7333(01)00137-8
  » http://dx.doi.org/10.1016/S0048-7333(01)00137-8
• Meier, D., & Liker, J. (2007). O modelo Toyota: manual de aplicação Porto Alegre: Bookman.
• Neely, A., Adams, C., & Kennerley, M. (2002). The performance prism: The scorecard for measuring and managing business success London: Prentice Hall.
• Neely, A., Gregory, M., & Platts, K. (2005). Performance measurement system design: A literature review and research agenda. International Journal of Operations & Production Management, 25, 1228-1263.
• Nobelius, D. (2004). Linking product development to applied research: transfer experiences from automotive company. Technovation, 24(04), 321-334. http://dx.doi.org/10.1016/S0166-4972(02)00073-1
  » http://dx.doi.org/10.1016/S0166-4972(02)00073-1
• Nudurupati, S. S., Bititci, U. S., Kumar, V., & Chan, F. T. S. (2011). State of the art literature review on performance measurement. Journal Computers and Industrial Engineering, 60(2), 279-290.
• Oliveira, J. F. G., & Telles, L. O. (2011). O papel dos institutos públicos de pesquisa na aceleração do processo de inovação empresarial no Brasil. Revista USP, 89, 204-217.
• Organização para a Cooperação e Desenvolvimento Econômico – OCDE. (2005). Manual de Oslo: proposta de diretrizes para coleta e interpretação de dados sobre inovação tecnológica (3 ed.). Brasília: OCDE, FINEP.
• Organização para a Cooperação e Desenvolvimento Econômico – OCDE. (2009). New nature of innovation Copenhagen: OCDE. Recuperado em 6 de fevereiro de 2014, de http://www.oecd.org
  » http://www.oecd.org
• Paulino, S. R., Salles-Filho, S., & Bin, A. (2007). Avaliação de programas tecnológicos: impactos ambientais da inovação na agricultura. In R. Casas, C. Fuentes & A. O. Vera-Cruz (Org.), Acumulación de capacidade tecnológicas, aprendizaje y cooperación en la esfera global y local (Vol. 1, pp. 3-26). Ciudad de México: Universidad Autónoma de Metropolitana.
• Penrose, E. (2006). A teoria do crescimento da firma ( T. Szmrecsányi, Trad.). Campinas: Editora da Unicamp.
• Rutman, L. (Ed.). (1977). Planning an evaluation study. In L. Rutman. Evaluation research methods: a basic guide (pp. 13-38). Beverly Hills: Sage Publications.
• Salles-Filho, S., Avila, F. D., Sepulveda, J., & Colugnati, F. A. B. (2010). Multidimensional assessment of technology and innovation programs: the impact evaluation of INCAGRO-Peru. Research Evaluation, 19, 1, 361-372. http://dx.doi.org/10.3152/095820210X12904313498360
  » http://dx.doi.org/10.3152/095820210X12904313498360
• Schulz, A. P., Clausing, D. P., Fricke, E., & Negele, H. (2000). Development and integration of winning technologies as key to competitive advantage. Systems Engineering, 3(4), 180-211. http://dx.doi.org/10.1002/1520-6858(2000)3:4<180::AID-SYS2>3.0.CO;2-H
  » http://dx.doi.org/10.1002/1520-6858(2000)3:4<180::AID-SYS2>3.0.CO;2-H
• Sheasley, W. D. (2000). Taking an options approach to new technology development. Research Technology Management, 43(6), 37-43.
• Software Engineering Institute – SEI. (2006). Improving process for better products. CMMI-DEV, V1.2 Pittsburg: SEI.
• Valmohammadi, C., & Servati, A. (2011). Performance measurement system implementation using Balanced Scorecard and statistical methods. International Journal of Productivity and Performance Management, 60(5), 493-511. http://dx.doi.org/10.1108/17410401111140400
  » http://dx.doi.org/10.1108/17410401111140400
• Viotti, E. B. (2002). National learning systems - a new approach on technological change in late industrializing economies and evidences from the cases of Brazil and South Korea. Technological Forecasting and Social Change, 69(7), 653-680. http://dx.doi.org/10.1016/S0040-1625(01)00167-6
  » http://dx.doi.org/10.1016/S0040-1625(01)00167-6
• Yam, R., Guan, J. C., Pun, K. F., & Tang, E. P. Y. (2004). An audit of technological innovation capabilities in chinese firms: some empirical findings in Beijing, China. Research Policy, 33(8), 1123-1140. http://dx.doi.org/10.1016/j.respol.2004.05.004
  » http://dx.doi.org/10.1016/j.respol.2004.05.004
• Zackiewicz, M. (2005). Trajetórias e desafios da avaliação em ciência, tecnologia e inovação (Tese de doutorado). Instituto de Geociências, Universidade Estadual de Campinas, Campinas.

Publication Dates

• Publication in this collection
  28 Sept 2017
• Date of issue
  Jul-Sep 2017

History

• Received
  30 June 2015
• Accepted
  09 Dec 2015