Module A2: Data Collection and Quality Control

6.Data quality control

To be successful and useful, a school census must collect relevant, comprehensive and reliable data about every school. To do so, rigorous data collection methods using appropriate instruments must be adopted, accompanied by well-defined procedures and strict application of data quality norms and control practices.

Controlling data quality is essential throughout the entire data collection process. in this section, we look at the concept and practices of data quality control, and the roles and responsibilities of each level of the education administration.

6.1 Purpose of data quality control

When collecting data, it is important that the data collected are of high quality so that they can be reliably used as the basis to make sound decisions. To ensure data quality, data control measures must be applied at every stage of the data collection process, including when updating school records (see Section 6 in Module A1), creating data summaries from the records and completing the school census questionnaire at the school level. Additional data quality controls should be applied by district, provincial and central education officers, and when entering the data into computerized eMiS database as well as during data analysis, interpretation and use.

6.2 Sources of problems in data quality

Various problems can affect data quality and subsequent data analysis and use in decision-making. Five main sources of data quality problems can be identified as follows:

  • Poor school records:non-existent,incomplete or inaccurate school records(seeModuleA1) can make it difficult for school managers to complete the school census questionnaire with reliable information. if a lot of schools have incomplete or inaccurate school records, it can impact the overall quality of the school census.For example: if a school does not keep records of students without textbook by subject, it cannot report the number of such students nor of the missing textbooks during the school census. The resulting national data will be incomplete and under-represent the real situation, thereby leading to wrong understanding and decisions regarding textbook production and distribution.
  • Bad design of the school census questionnaire:inappropriate structure and presentation of the questions and tables, and missing or unclear explanations and instructions (see Sections 4.2 and 4.3), can cause errors while completing the school census questionnaire, if not also many omissions of important data.For example: A complex statistical table that presents multiple layers of headings and sub- headings that try to collect data on the number of students, repeaters, in-transfers, drop-outs by gender and age all in one table will be very difficult if not impossible to complete, which may cause the respondent to provide inaccurate or incomplete data.
  • Lack of understanding of data terms,concepts and categories:Respondents fill in the questionnaire without reading and fully understanding the data concepts, terms and categories.For example: Respondents don’t understand they need to subtract the number of students who have repeated Grade 1 from those students who are enrolled in Grade 1 when asked to provide data about new entrants into Grade 1.
  • Incorrect completion of questionnaire:Respondents enter either incorrect data or correct data but into the wrong cell, or have not completed all the essential cells of the questionnaire.For example:
    • ThenumberoffemalenewentrantstoGrade1was343,but34wasfilledinthequestionnaire. This will affect information about the gender balance in this school.
    • 343wasentered,butintothecellformalenewentrantstoGrade1.
    • School managers did not fill all essential cells in the questionnaire using the correct symbols given in the beginning of the questionnaire (see example on p. 40).
  • Inadequate and careless checking of the completed questionnaire: School managers, district education officers, inspectors and other stakeholders did not thoroughly check the completed questionnaires for errors and omissions.For example: The school manager and the district education officer checked to see whether all the questions and tables were completed, but failed to check for obvious data errors.

Data quality problems can be caused by heavy workload and unclear definitions of roles, responsi- bilities and tasks among those responsible for completing and verifying the school census question- naire. Poorly designed questionnaire, inadequate understanding of the instructions, intentional or unintentional misreporting, and many other factors can also affect data quality during school cen- suses. Having a thorough understanding of the factors that can affect data quality is essential.

6.3 data quality control during data collection

Data quality control can be done:

  1. Before and during school census data collection.
  2. During data entry and processing.
  3. When analysing, interpreting and using the data.

The following part introduces some general methods and practical tips for carrying out data quality control during the processes of data collection.

6.3.1 School record management

Quality data begins at the source. Standards and procedures about records management should be clearly defined and implemented in all schools. Schools must be required to maintain school records in a systematic and rigorous manner so information is available for regular school management and for use in responding to the annual school census questionnaire.

6.3.2 Design, pre-test and revision of the school census questionnaire

data quality control mechanisms should be built into the design of the census questionnaire and refined through the pre-test and subsequent revisions.

TIPS

  • Carefully investigate the socio-economic characteristics and aspects of culture that can affect school reporting and quality control practices, and consider how the questionnaire should be designed to minimize such problems.
  • Design the school census questionnaire with clear structure, presentation and explanations, and concise instructions.
  • Thoroughly revise the questionnaire based on the feedback gathered from the pre-test, before the finalized questionnaire is distributed to schools across the country.

6.3.3 Completing the questionnaire

TIPS

  • Respondents must read and fully understand the instructions for completing the school census questionnaire, and should ask district education officers for help when needed.
  • School managers must carefully check and re-check the data for omissions and errors during and after completion of the questionnaire.
  • District education officers and school inspectors should train relevant school personnel and maintain frequent communication with them to monitor progress while they complete the school census questionnaire, and to offer help with data or reporting problems.

6.3.4 Data verification and quality control

TIPS

  • Data quality checks and controls should be done as close to the data source as possible to make it easier for school managers and district education officers to trace data errors and omissions back to records and information at school.
  • School managers should check for data omissions, errors in calculations, inconsistencies within and across tables, and for the incorrect use of special symbols (see next section for details).
  • District education officers and inspectors should systematically check for late or missing responses, misunderstanding among school managers, and data omissions and errors (see next section for details).
  • Training in data quality control procedures should be provided to education officers at the district and provincial levels to ensure they can check and control the quality of data from the schools.

6.3.5 Data quality control during data entry and processing

Additional data quality controls can be applied during data entry and processing.

TIPS

  • Automatic data verification mechanisms can be designed and incorporated into data entry systems using computers or internet-based online systems, to indicate data omissions, errors and inconsis- tencies. For example, the system may automatically add up all the values in rows and columns and cross-check with the reported totals.
  • Once data entry is complete, calculating derived statistics such as percentages, ratios and rates can highlight missing data and errors. For example, it may signal errors when girls account for more than 100% of total enrolment, or when there are student-teacher ratios of more than 1000.
  • When school managers enter data online, built-in automatic data verification systems can immedi- ately signal any data omissions and errors on-screen, so that corrections can be made immediately.

6.3.6 Data quality control during data analysis and interpretation

The processes of deriving indicators and analysing and interpreting the information from the school census data can help to highlight and identify other data anomalies.

TIPS

  • Calculation of education indicators and comparing them among provinces, districts and schools can reveal unlikely or illogical results that can be traced back to data or calculation errors.
  • Further data errors and inconsistencies can be detected while interpreting the analytical results to draw conclusions.
  • Having other persons to review the analytical results may help to identify data anomalies that were not obvious during the initial analysis.
  • When disseminating information products and indicators, encourage the users to query the information or make comments. Standard procedures for referring queries back to the appropriate district education officer or school manager should be defined and implemented.

6.4 The role of stakeholders in data quality control for school census

Without serious data quality control, the data collection process can be like cooking food without tasting it. Stakeholders at different levels of the education administration can play specific roles in data quality control. These are described in more detail below.

6.4.1 Schools and school administrators

School, as the main source of school census data, can be considered the most important level for data quality control. At this level, attention may focus on two aspects:

School records

School records must be created and updated in a systematic, complete and accurate manner. School records can be subject to frequent review by the responsible school personnel to ensure the data in the records are always accurate and up-to-date.

When preparing to respond to the school census, school managers and staff including teachers should retrieve the relevant school records, extract the correct information, and carefully tally the data in order to answer the questions and properly fill in the tables. during such preparations, school managers should ensure the data in the school records match the definitions of data in the school census (see Section 3 in Module A1 for details). When needed, the school manager should refer any errors or omissions to the school staff who are responsible for gathering and recording the data in the school records, so they can explain the omissions and rectify the errors.

Completing the school census questionnaire

Errors may occur when data are being entered into the school census questionnaire. This may occur for a number of reasons.

  • There is a mismatch between the data requested and the data in the school records – This can occur if the school manager misunderstood the meaning of specific data categories in the school census questionnaire which are different in the school records, and used the wrong school data. School managers should always read the instructions and data definitions carefully before entering data, and consult district or local education officers if they need clarification or help.
  • Mis-codingofdatainthequestionnaire-Schoolmanagersshouldbecarefulinensuringthey use the correct codes defined by the Ministry of education when entering data.
  • Not checking responses-Schoolmanagersshouldcarefullycheckandre-checkthecompleted questionnaire to identify any remaining data errors or omissions before submitting it.

CHECK LIST FOR SCHOOL MANAGERS TO ENSURE DATA QUALITY:

  • Check the school code
  • Check the name of the school, full address, telephone number, etc.
  • Ensure all pages and questions have been properly completed
  • Calculate and re-calculate the vertical, horizontal and grand totals in tables to ensure the data are consistent
  • Cross-check the totals between two related tables to verify consistency
  • Check the balance between school income and expenditure
  • Check any fields marked ‘data not available’ to confirm that the data are really unavailable and that reasons for the data being unavailable have been specified
  • Check that any special symbols used in the questionnaire are consistent with the special symbols the Ministry of education has specified

 

Activity 14

For school managers, please compare the data quality control practices described above with the practices in your school, and answer the following questions:

  1. What role do you play in your school regarding data quality control?
  2. How do you ensure your school consistently reports high quality data during school censuses?
  3. What difficulties have you encountered with data quality control at your school? What caused these difficulties?
  4. Referring to the checklist above, what else should you check before you submit your completed questionnaire to a higher level?

6.4.2 District Education Officer (DEO)

The district education Officer is responsible for ensuring that all the schools in the district receive the school census questionnaire (or can access it online) and respond by the deadline. When requested, district education officers should be ready to provide assistance to schools to help them accurately complete the questionnaire.

After the schools, district education officers are the second most important level for data quality control. district education officers are normally familiar with the schools in their district so they are well-placed to make initial checks of the completed questionnaires submitted by the schools. Any data omissions, inconsistencies or errors identified at this level can be corrected easily by calling or visiting the school. Monitoring school records and helping schools to improve their school records management practices are also the responsibility of the district education officer. These tasks directly contribute to maintaining the quality of census data at the source.

While checking the completed school census questionnaire, the district education officers may apply the methods below:

TIPS

  • Coverage check
    • Check the school list to ensure that all schools have returned the completed questionnaire
    • Contact and remind the schools that have not responded
    • Offer assistance to schools that have problems with completing the school census questionnaire
  • Data check
    • Are all the pages, questions and tables properly completed? Are the answers legible? Contact the school to complete any missing data.
    • Have the special symbols been properly used to explain data limitations?
    • Do the vertical and horizontal totals correspond to the sum of the detailed data?
    • Are the same data consistent in different tables of the questionnaire?
    • Are there illogical data (e.g. the number of female students is larger than the total of male+ female student population)?
    • Use other innovative data checks.
  • Feedback on questionnaire design
    • Provide useful feedback to the Ministry of Education about difficulties encountered during school census so as to further improve the design of the questionnaire, instructions and the organization of the school census.

 

Activity 15

For district education officers, please compare the data quality control practices described above with the realities in your district, and then answer the following questions:

  1. What is your role in data quality control during school censuses?
  2. What difficulties have you encountered while maintaining data quality control? What caused these difficulties?
  3. What do you think of the checklist above? What else needs to be checked and how?

6.4.3 Provincial Education Office (PEO)

Data quality control accompanies the process of data collection and is repeated at every stage and level of the process, so as to eliminate in a step-by-step manner the remaining data errors and gaps. depending on the number of schools within the province and the capacity of staff at the provincial education office, the PeO may choose to perform the same kind of data quality control described for district education officers above, either for all the schools in the province or for a representative sample of them. Another way to control data quality is for the provincial education office to cross- check either a random sample or representative sample of the completed and quality-controlled school census questionnaires forwarded by the deO.

TIPS

  • Sample check: The PeO may select at random 10 per cent of the questionnaires delivered by the district education offices to verify if data checks and corrections have been properly carried out at the district level, and identify if any other data errors may be present. Any problem discovered will be referred back to the relevant district education office and schools for clarification and correction.
  • The provincial education office can provide useful feedback information and suggestions to the Ministry of education about the situation and issues in specific districts and schools with regard to school census data collection, in order to help to improve future school censuses.

 

Activity 16

For provincial education officers: Please compare the data quality control practices described above with the realities in your province, and then answer the following questions:

  1. What responsibility do you have for school census data quality control?
  2. How do the practices in your province compare to the practices described above?
  3. What difficulties have you encountered in school census data quality control in your province? What caused these difficulties? Please give examples.
  4. How can provincial education officers contribute to improved data quality control in the future?

6.4.4 Ministry of Education (MOE)

The Ministry of education is responsible for designing, pre-testing and producing the school census questionnaire to collect data from the schools. Good design, layout and usability of this question- naire has a large impact upon the quality of data that are collected during the school census. As described in Sections 3.3, 4.2 and 4.3, the following tips can help the MOe to minimize data prob- lems and improve data quality:

TIPS

  • Design of the questionnaire
    • Relevant objectives, contents and structure
    • Clear, attractive and efficient design
    • Thorough and easy-to-understand explanations and instructions
    • Pre-testing of the questionnaire
    • Gather maximum comments and suggestions from the field; seriously review them to identify errors, gaps, inefficiencies and operation problems
    • use other innovative data checks.
  • Revision of the questionnaire
    • corporate appropriate improvements in finalizing the questionnaire
    • Update the questionnaire periodically in accordance with changes in the education system and information needs.
  • Access to the questionnaire
    • Make sure all the schools can access the questionnaire and instructions, either in paper form or electronically (e.g. in electronic file or online).

For national ministries of education that operate computerized eMiS, automated data quality control mechanisms can be incorporated during:

  • Data entry: by building data checking routines into computerized data entry programmes, miss- ing data and errors in horizontal, vertical and grand totals, data inconsistencies across tables and data that exceed pre-determined norms can be flagged by the system.
  • Data processing: data cleaning and calculating various percentages, rates, ratios, ranges and other statistics can help to identify anomalies in the data.
  • Data analysis: observing and discovering irrationalities in the analytical processes and results, which can be traced back to problems with the quality of data.
  • Data use: encourage decision-makers and stakeholders to actively use the data and information produced, and to provide comments and queries regarding the quality of specific data.

In addition, it is the role of the Ministry of education to promote the systematic practice of school records management in all schools, which will decisively contribute to data quality in the school census.

Activity 17

For Ministry of Education officers, please compare the data quality control practices described above with those in your Ministry, and then answer the following questions:

  1. To what degree does your Ministry apply the tips above while designing, testing and revising the school census questionnaire? Please give examples.
  2. How does your Ministry carry out data quality control during data entry, processing and analysis? Please describe.
  3. What problems have you encountered with school census data quality control in your country? What causes these problems? How can these problems be solved?
  4. How can the Ministry of education improve data quality control in future school censuses?

Comments are closed.