I. Overview of Reliability

Reliability is also called stability or accuracy. It is used to reflect the degree of coincidence or consistency of the measurement results of the same objective thing for many times under the same conditions, and to explain the reliability of the data. When the same measuring tool is used to measure an objective thing repeatedly, the higher the consistency of the results, the higher the reliability of the measuring tool.

Reliability contains two meanings: first, whether the same individual can get the same result with the same measuring tool at different times, or repeatedly, or under different circumstances, that is, whether the measurement results change with time and place; The second is whether the influence of random error on the test results can be reduced, so as to reflect the real situation to be measured by the measuring tool, that is, whether the measurement results are stable, reliable and predictable. A good measuring tool must be stable and reliable, and the results of many measurements should be consistent.

Second, the classification of reliability

The classification of reliability can generally be divided into the following four categories:

● Test-retest reliability.

● Alternating form reliability.

● Reliability of internal consistency.

● Rater reliability of raters

1, retest reliability

Test-retest reliability refers to the degree of correlation between the results of repeated tests by the same group of respondents with the same measuring tool at a certain interval. Test-retest reliability can be tested in two ways:

Path 1: Calculate the correlation coefficient of two test results. If the correlation is statistically significant, it is considered that the reliability of the measuring tool is high, otherwise the reliability is low. It has also been suggested that the acceptable criterion for the retest reliability of measuring tools is that the correlation coefficient between the two tests is above 0.7.

Path 2: Test the difference between two related samples of two repeated test results. If the difference is statistically significant, it is considered that the reliability of the measuring tool is low; Otherwise, the reliability is high.

2. Replica reliability

Replica reliability method is to let the same group of respondents fill out two questionnaires at a time and calculate the correlation coefficient of the two questionnaires. The copy reliability law requires that the two copies should be completely identical in content, format, difficulty and the direction of corresponding questions except for different expressions. But in the actual investigation, it is difficult to make the questionnaire meet this requirement, so few people use this method.

3. Half-reliability

Semi-reliability is usually used when repeated adjustment is impossible. The calculation method is to divide the survey items into two halves, and calculate the correlation coefficient r of the measurement results of these two halves, also known as split-half reliability coefficient, as a standard to measure the reliability of the whole scale.

Split-half reliability divided the survey items into two halves, calculated the correlation coefficient of the scores of the two halves, and then estimated the reliability of the whole scale. Split-half reliability belongs to the inherent consistency reliability, which measures the consistency between the scores of two and a half projects. This method is generally not suitable for fact-based questionnaires (such as age and gender can not be compared), and is often used for reliability analysis of attitude and opinion questionnaires. The most common form of attitude measurement in questionnaire survey is the 5-level Likert scale.

4. Reliability coefficient of Kronbach

Cronbach's sQ coefficient was created by Cronbach in 195 1 to evaluate the internal consistency of the questionnaire. The α coefficient is between 0 and 1. The higher the Q coefficient, the higher the reliability of the questionnaire and the better the internal consistency.

Cronbach's A coefficient is not only suitable for two-level scoring questionnaires, but also suitable for multi-level scoring questionnaires.

The consistency between the scores of each item in a coefficient evaluation scale belongs to the reliability coefficient of internal consistency. This method is suitable for the reliability analysis of attitude and opinion questionnaire (scale).

Low reliability: a

5. Reliability of raters

The method to examine the reliability of raters is to randomly select a considerable number of questionnaires, and two raters will score them respectively according to the scoring rules. Then calculate the correlation coefficient according to the score of each questionnaire, and get the reliability of the raters. The reliability of raters can also be the correlation coefficient of two ratings of a rater. If scored by multiple raters or more than two raters, Kendall Harmony Coefficient and Kappa Coefficient can be used. Kendall harmony coefficient is used for grade data and kappa coefficient is used for qualitative data.

Third, methods to improve reliability.

Methods to improve reliability The higher the reliability of the questionnaire, the lower the interference from people, time and things, and the higher the degree of reflecting facts or convincing people. Therefore, how to effectively improve the reliability before the implementation of the questionnaire is the key to the success or failure of the questionnaire test. There are several ways to improve the reliability of the questionnaire:

1, appropriately extend the length of the questionnaire.

There are many questions in the questionnaire, which eliminates the influence of accidental factors to some extent and improves the reliability of the questionnaire. However, the increase of questionnaire length is not always in direct proportion to the increase of questionnaire reliability. When the reliability coefficient is small, the reliability coefficient of the questionnaire increases greatly with the extension of the questionnaire length. When the reliability coefficient is large, extending the length of the questionnaire has little effect on the reliability coefficient.

2. The difficulty of the questionnaire should be moderate.

When the topic of the questionnaire is too difficult, the score of the questionnaire is generally too low; When the topic of the questionnaire is too difficult, the score of the questionnaire is generally higher. Questions that are too difficult or too easy will reduce the difference of the questionnaire scores and the variance of the obtained scores, thus reducing the reliability of the questionnaire.

3. The contents of the questionnaire should be consistent.

The questionnaire with the same content requires the respondents to have the same ability, knowledge and skills. Therefore, in order to improve the reliability of the questionnaire, the content of the questionnaire should be as homogeneous as possible.

4. The examination time should be sufficient.

The questionnaire should ensure that the vast majority of respondents can complete the test within the specified time. When the respondents can't answer all the questions calmly, the scores of the questionnaire can't reflect the real situation of the respondents.

5, test procedures should be unified.

The questions of the questionnaire should be unified, and the guiding language, answering methods, winding methods and questionnaire testing time should be unified. These are the basic guarantees of high reliability of the questionnaire.

term of validity

I. Definition

Validity refers to whether the content of the questionnaire can truly reflect the content of the survey.

Second, the role

The effectiveness of 1. is the most important prerequisite for scientific measurement tools. When choosing a standard test or designing and compiling a measuring tool by yourself, we should first evaluate its effectiveness.

2. Validity is the degree of consistency between measurement and measurement. Some measurements can be completely suitable for one purpose, but not necessarily for other purposes.

Third, classification

According to the American Psychological Association's standards for education and psychological testing, it can be divided into three categories:

1, criterion-related validity: refers to the validity of the questionnaire in predicting the operational behavior of the respondents under certain conditions.

(1) Predictive validity: refers to the degree of correlation between the questionnaire results and a certain criterion in the future, and is often used to predict the possibility of individual success in a certain aspect in the future. For example, the simulated test results of the senior high school entrance examination can accurately predict a student's performance in school for four years. Only by looking at the high correlation between them can it be shown that the survey has predictive validity.

(2) Concurrent validity: The correlation between the questionnaire results and the current effective standards often proves the validity of the questionnaire. For example, conduct a confirmatory survey on the same group of respondents and compare their scores. If the correlation is high, it is valid at the same time.

2. Content validity: refers to the coverage or representativeness of the questionnaire to the investigated content. For example, if a questionnaire is to investigate the customer's satisfaction with the service of staff in various positions, including shopping guide, cashier and security guard, if the questionnaire only mentions shopping guide and cashier, it lacks content validity.

3. Structured validity: The concept of structured validity was first put forward by the joint committee of American Psychological Association (APA), American Association for Educational Research (AERA) and American National Association for Educational Measurement (NCME) (1954), and published in Technical Suggestions on Psychological Testing and Diagnostic Technology. It was originally used when there is no clear standard measurement of the measured attribute of the tester, but indirect measurement is needed to prove the validity of the theory. So far, structural validity refers to the degree of theoretical structure and characteristics to be measured in a test, and refers to the consistency between experiment and theory, that is, whether the experiment really measures the hypothetical (constructed) theory.

4. Aggregation validity: refers to the degree of aggregation of multiple observed variables with the same character or structure by different methods. If different measuring tools are used to measure the results of the same individual, high aggregation validity means that the observed variables will have higher factor load in the corresponding factor construction, and there is a high correlation between the observed variables, so the aggregation validity is high.

5. Discriminatory validity: it refers to testing the degree of difference between different factors. The commonly used test index is to compare the average variance of two-factor structure with the square of correlation coefficient between two-factor structure. If the former is greater than the latter, it means that there is good discrimination validity between the two factors.

Fourthly, methods to improve validity.

Validity represents correctness. The higher the validity of the questionnaire, the more the test results can represent the real characteristics of the subjects. The methods to improve effectiveness are as follows:

1, the questionnaire design should be scientific and reasonable;

The content of the questionnaire has a very important relationship with its validity. If the content is set scientifically and moderately, the validity will be high, on the contrary, it will affect the validity. For example, the setting of each topic should have a clear concept, the difficulty of the topic should remain unchanged, and the arrangement of the topic should be reasonable.

2. Carefully select samples:

To choose a representative sample, the more representative the sample, the more it can reflect the main characteristics of the questionnaire. Other things being equal, the greater the full range of sample scores, the higher the validity coefficient.

3. Pay attention to the recovery rate of the questionnaire;

If the recovery rate is low, it may not be representative and cannot correctly reflect the characteristics of the measured object.

4, the rationality of the goal:

Norm is the premise of effectiveness. We should construct appropriate operations according to the content. If we choose unreasonable standards, we will not be able to show any value.

5. Pay attention to the interviewee's status:

In the process of answering the questionnaire, the state of the interviewee is an important factor affecting the validity. If the interviewee is absent-minded or unwilling to answer, or even has a negative attitude, it will affect the accuracy of the questionnaire results.

6. Ensure that the test environment is not disturbed:

If the interviewee is in a noisy external environment or the external environment cannot be controlled by the researcher, then the interviewee will be disturbed, so it is necessary to eliminate irrelevant interference to avoid affecting the correctness of the questionnaire.