| A- Functional supportability (functional completeness, functional correctness and functional adequacy) (QA=7, PI=7) |
G
PS
US |
1) The user needs referring to treatment of information are addressed by how many functions made available in the system? |
0-20% (none/almost none)
20-40% (few)
40-60% (about half)
60-80% (most)
80-100% (all/almost all) |
ISO/IEC 25023 - Functional implementation coverage metric |
| 2) How many functions in the system make clinical documentation available in a correct and complete manner? |
G
PS |
3) How many functions available in the system implement functionalities that observe legal information rules (CID10, DRG, data transmission, etc.)? |
| 4) User needs in support for decision-making are addressed by how many functions available in the system for such purpose? |
| PS |
5) For the system functions that need verification and prevention of medication errors, how many implement this functionality? |
G
PS
US |
6) How frequently does the system present incorrect or imprecise information? |
Always/ In general/ Eventually/ Rarely/ Never |
ISO/IEC 25023 - Computational accuracy metric |
| 7) Do the system functionalities integrate different areas or departments? |
Never/ Rarely/ Moderately/ In general/ Always |
ISO/IEC 25023 -Functional adequacy metric |
| B- Performance efficiency (Behavior with regard to time and capacity) (QA=4, PI=5) |
G
PS |
8) Is the response time for a job in the system satisfactory? |
Never / Rarely / Moderately In general, Frequently /Always |
ISO/IEC 25023 - Turnaround time metric |
G
PS
US |
9) Is the response time for an online operation in the system satisfactory? |
ISO/IEC 25023 - Response time metric |
| 10) Does authentication of the system occur within a satisfactory timeframe? |
| 11) Does the system generate clinical documentation within a satisfactory response time? |
|
|
Indicator: The system provides its users with adequate simultaneous access, with satisfactory performance. |
ISO/IEC 25023 -Number of simultaneous accesses metric: NAS=verified number of simultaneous accesses by users, with performance within adequate standards; NMA=maximum number of simultaneous accesses predicted, estimated in system requirements EFDESEMP=NAS/NMA |
| C- Compatibility (Coexistence and Interoperability) (QA=2, PI=2) |
G
PS
US |
12) The system allows data availability for how many other systems that need to access it? |
0% (none)/
Up to 25% (few)/
Between 25 and 75% (partially)/
More than 75% (almost all)/
100% (all) |
ISO/IEC 25023 - Coexistence availability metric |
| 13) How many systems does the application have integration with through exchange of information and use of information that is exchanged, in cases where interoperability is required? |
ISO/IEC 25023 - Connectivity with external systems metric |
| D- Usability (learnability, operability, protection from user error, esthetics in the user interface and accessibility (QA=13, PI=9) |
G
PS
US |
14) For how many functions does the system have documentation available (manuals, tutorials and training material) and/or help online for users? |
0% (none)/
Up to 25% (few)/
25 to 75% (about half)/
More than 75% (most)/
100% (all) 0% (none)/
Up to 25% (few)/
25 a 75% (about half)/
More than 75% (most)/
100% (all) |
ISO/IEC 25023 - user documentation completeness and/or help metric |
| 15) How many functions in the system are easy to understand, as from the system documentation? |
Not specified |
| 16) In how many functions does the system have standardized access, with similar browsing and rapid access? |
ISO/IEC 25023 - Operational consistency metric |
| 17) For how many functions does the system produce adequate feedback, with clear messages, that allow understanding of tasks that are executed? |
ISO/IEC 25023 - Message clarity metric |
| 18) How many functions in the system are easy to operate, intuitive handling? |
Not specified |
| 19) How many functions of the system present clear interfaces (screens/forms/data entries/reports/graphs), as well as terms and concepts used in the system, which are clear and have no ambiguities? |
Not specified |
| 20) In how many system functions is the interface uniform, standardized? |
Not specified |
| 21) How many system functions incorporate facilities for people with special needs or elderly people? |
ISO/IEC 25023 -Physical accessibility metric |
G
PS |
22) Among the functions that require/ demand customization by the user himself, in how many does the system allow the user to make adaptations to address his local/specific needs? |
ISO/IEC 25023 - Customization possibility metric |
| PS |
23) In how many system functions can an action be reversed, in a simple, easy and safe manner? |
ISO/IEC 9126-3 - Ease of cancellation metric
ISO/IEC 9126-3 - Ease of undoing metric |
|
|
24) In how many system functions can the interface be modified, with the appearance customized by the user? |
ISO/IEC 25023 - user interface appearance customization metric |
|
|
25) For how many functions does the system block incorrect operations? |
ISO/IEC 25023 Incorrect operation hindrance metric |
|
|
26) For how many entries (attributes, fields) does the system not allow the entry of invalid or incorrect data? |
ISO/IEC 25023 - Entry validation verification metric |
| E- Reliability (maturity, availability, recoverability, tolerance to failure and non-questioning) (QA=6, PI=7) |
G
PS |
27) How frequently does the software go through maintenance to correct errors? |
Never or rarely/ Eventually/ Moderately/ Regularly/ Very frequently |
ISO/IEC 25023 - MTBF metric (media time between failures) and ISO/IEC 9126-3 - Failure detection metric |
| 28) Do the corrections, improvements or updates of the version that occur in the system cause instability or require excessive effort or time? |
Always/ Frequently/ Moderately/ Eventually/ Never |
MR= number of maintenance calls arising from previous maintenance, that generate side effects and do not allow for installation of system. NM= total number of maintenance calls CONF= MR/NM (metric proposed by author) |
| 29) How many system functions that require digital signatures require this resource? |
None/ Few/ About half / Most/ All |
ISO/IEC 25023 -Digital signature use metric |
G
PS US |
30) Do errors occur when the system is being used? |
Very frequently/ In general, Regularly / Moderately / Eventually/ Never or rarely |
Same metrics from question (27) |
| 31) How often is the data available to the user, when required? |
Always or almost always not available / Unavailable most of the time / There are regular periods when the system is unavailable / Eventually unavailable / Always available |
ISO/IEC 25023 -System in production rate metric |
| 32) When there is data loss in the system (power outage, equipment failure, etc.), we can say that the data: |
is lost, cannot be recovered / is rarely recovered / Sometimes data is recovered, sometimes not/ Frequently, almost always it is recovered / Always recovered |
ISO/IEC 9126-3 - Restorability metric |
| Indicator: The system has resources to store redundant data. |
ISO/IEC 25023 - Component redundancy metric: NCR = number of components installed redundantly to avoid system failure; NCRI = number of redundant components installed predicted in the system requirements CONF = NCR/NCRI |
| F- Safety (Confidentiality, Authentication, Integrity and Accountability) (QA=1, PI=3) |
G
PS |
33) In how many functions does the system allow access and operation information to be registered that can be audited, traced in the future? |
None/ Few/ About half/ Most/ All |
ISO/IEC 25023 -Access auditability metric |
|
|
Indicator: There is no risk of non-authorized access to information in the system |
a) ISO/IEC 25023 - Access controllability metric:
TCAI = number of types of access controls (verification of illegal operation) correctly deployed and verified;
TCAP = number of types of access controls predicted in system requirements.
SEG1 = TCAI / TCAP
b) ISO/IEC 25023 - Data encryption metric:
NIDCE = number of data items correctly encrypted/decrypted;
NIEP = number of data items with encryption predicted in system requirements.
SEG2 = NIDCE / NIEP |
| G- Maintainability (modularity, modifiability, reusability, analyzability and testability) (QA=0, PI=6) |
|
|
Indicator: The software integrates (easily) with new /systems. |
a) ISO/IEC 25023 - Condensability metric:
NCNA= number of components not affected by alterations in other components; NTC= total number of components
MANUT1 = NCNA/NTC
b) ISO/IEC 25023 - Modification success rate metric: PRAM = number of problems/complaints before specific maintenance; PRDM = number of problems/ complaints after (same) maintenance.
MANUT2 = (PRAM-PRDM)/PRAM (calculate average for one set of modifications in a given timeframe) |
|
|
Indicator: The system has reusable software components. |
ISO/IEC 25023 -Degree of reusability metric:
NCRS = number of reusable software components/artefacts, which can be used in more than one system or used to build other systems
NTCB = total number of reusable software components/artefacts in the library of reusable objects in the development environment of the software product
MANUT3 = NCRS / NTCB |
|
|
Indicator: Little effort is needed to locate causes of failure in software. |
a) ISO/IEC 25023 - Audit record for causes of errors metric:
NRCE = number of distinct causes of error records in system operations; NPRCE = number of records of causes of errors recorded planned sufficiently to monitor the status of system during its operation.
MANUT4 = NRCE / NPRCE
b) ISO/IEC 25023 - Diagnostic function metric: NFDF = number of functions for diagnosis of failures available for the system; NPFDF = number of functions for diagnosis of failures predicted in the system specification/requirements
MANUT 5= NFDF / NPFDF |
|
|
Indicator: The system can be efficiently tested after a modification. |
ISO/IEC 25023 - Test case coverage metric:
NCT = number of test cases prepared to test system; NTCT = total number of estimated test cases to verify function of system, with inclusion of test cases for integration with other systems.
MANUT 6= NCT / NTCT |
| H- Portability (Adaptability and capacity to be installed) (QA=0, PI=3) |
|
|
Indicator: The system operates in environments that are market standards (operational system, data base, development tools, etc.). |
ISO/IEC 25023 - Software environment adaptability metric:
NFAS = number of system functions successfully tested in other software environments (SGBDs, operational systems, development environments, etc.), besides the native environment; NTF = total number of system functions.
PORT1=NFAS/NTF |
|
|
Indicator: The system presents independence and mobility for storage and recovery of information (notebooks, tablets, PDAs, etc.). |
ISO/IEC 25023 - Hardware environment adaptability metric:
NFAH = number of system functions successfully tested in other hardware environments besides the native environment; NTF = total number of system functions.
PORT2=NFAH/NTF |
|
|
Indicator: The installation of the system in the user environment is easy and fast. |
ISO/IEC 25023 - Ease of installation metric:
NIS = number of successful installations, where installation occurred according to user convenience and time for installation was adequate; NTIS = total number of installations and attempts to install the system.
PORT3 = NIS / NTIS |
Source: 