iNtgrty – Page 11 – Publishing and the Mentoring Network Journal

News

ARTICLE 36: Research Methods for Ph. D. and Master’s Degree Studies: Sampling Part 4 of 6

2nd Sep 2020 No Comments

Written by Dr. Hannes Nel

Introduction

Sampling types mostly differ only in the way in which the sample is composed.

Ultimately, samples are all small extracts from a larger population.

They are studied because the entire population is too large for viable research.

Obviously, the sample should display behaviour, phenomena, events, etc. that would be typical for the entire population.

That means that the research findings based on the sample should be generalizable.

This, however, is not always the case.

Researchers sometimes limit their research to the sample without even trying to achieve generalizability.

In effect, what is supposed to be a sample becomes the population.

I will discuss the following sampling methods in this article:

Probability sampling.
Purposive sampling.
Quota sampling.
Random sampling.
Reputational-case sampling.
Snowball sampling.
Stage sampling.

Probability sampling

A probability sample is drawn randomly from the wider population.

Therefore, it will be useful if you need to generalize.

Or if you seek representativeness of the wider population.

The name already reveals that it should be used with quantitative research.

Obviously, statistical calculations will form part of the research process.

Probability or representativeness is calculated with a wonted measure of accuracy probability.

For example, 97% (giving an error factor of 3%).

The risk of bias is less than if you were to rely on your gut feeling or subjective judgement.

However, all samples, regardless of whether they are based on statistical calculation or your judgement, are hampered by a measure of sample error.

Purposive sampling

Purposive sampling is also sometimes called purposeful sampling or judgmental sampling.

In purposive sampling the researcher collects data according to the aims of the research.

To develop a purposive sample that will sufficiently represent the population:

You will need to know the profile of your target group.
And you will select items for the sample according to a list of important characteristics.

You can select purposive samples after you have studied the bigger population from which to select the sample.

This might require fieldwork or studying written profiles, such as curriculum vitae.

It can also require fieldwork and literature study.

You will need to find certain types of individuals displaying attributes that are needed for your research.

In horticultural research you will probably use categories such as resistance to plant diseases, flower colour, size and shape, etc.

In social research categories such as age, gender, status, role or function in organizations, stated philosophy or ideology, may serve as starting points.

For study in-depth, you will need to select information-rich cases.

As your research progresses, you may discover new categories which would lead you to more sampling in that particular dimension.

Quota sampling

For quota sampling you will need to choose representative individuals from a specific subgroup.

For example, numbers in accordance with the demographic composition of the total population.

You can simplify the process by first preparing a matrix table that creates cells or strata.

You may wish to use gender, age, education, or any other attributes to create and label each stratum or cell in the table.

The research question and the focus of your study will determine which attributes you should select for your sample.

You should use a nonprobability method to fill the cells.

Each category in the overall sample must be filled using the same recruitment procedure for the resulting groups to be comparable.

Next, you need to determine the proportion of each attribute in the full study population.

Let’s say you want to study perceptions of violence among people over age 50.

You can, for example, group the people in your sample as younger than 30; 30 to 39; 40 to 49; 50 to 59 and 60 and older.

You will need to determine the proportion of people in each age group, although it will often be best to have the same number of individuals in each group.

The groups younger than 50 are included for comparison purposes.

Random sampling

In random sampling, each possible sample has an equal probability of being chosen as the sample.

The sample needs to be large enough to deliver findings that will apply to the entire target group with a reasonable accuracy.

This depends on the size of the population and the measure of heterogeneity.

The larger the population, the larger the sample drawn should be.

A random sample is mostly used when each element of the population has the same chance of being selected to be part of the sample.

This is sometimes called ‘simple random sampling’.

The procedure used to draw the sample needs to ensure that as little bias as possible is present.

Drawing names from a hat, putting the names on a dart board and throwing darts at them to choose sample items, the roll of a dice are examples of how a random sample can be composed.

Some dedicated computer software can also select a random sample from a list of names.

Reputational-case sampling

Reputational-case sampling is a variant of extreme-case and unique-case sampling.

Like in the case of the extreme-case and unique-case sampling, you will choose a sample recommended by an expert in the field.

For example, you might be looking for individuals who have particular expertise that is most likely to advance your research interests.

Snowball sampling

Snowball sampling is also called chain referral sampling or respondent-driven sampling.

Snowball sampling is an effective way to locate subjects with certain attributes or characteristics necessary in the study.

Snowball samples are particularly popular among researchers who are interested in studying various classes of deviance.

Sensitive topics are often studied, and the sample often come from difficult-to-reach populations.

In snowball sampling, you would identify a small number of individuals who have the characteristics in which you are interested.

These people are then asked to identify and put you in touch with others who also have the wonted characteristics.

For example, if you are doing research on people who suffer from arachnophobia, you need to find only one person who is intensely scared of spiders and would be willing to put you in touch with others who suffer from arachnophobia to get the snowball rolling.

It so happens that this sampling method is especially useful for sampling a population where access and obtaining co-operation are difficult.

The basic procedure of snowball sampling begins by identifying one or more people with relevant characteristics and then interviewing them or asking them to complete a questionnaire.

After interviewing the subjects or after they have completed a questionnaire, you would ask them for the names of other people who possess the same attributes as them.

In this manner, a chain of subjects driven by the referral of one respondent to another is formed.

Stage sampling

Stage sampling is an extension of cluster sampling.

It involves selecting the sample in stages, i.e. taking samples of samples.

Doing research on the value systems of universities, you can first select a cluster of universities as your sample, then certain faculties in each university as a sub-sample, then a number of students from each faculty as your sub-sub sample.

Summary

A probability sample is a random sample for which the measure of probable accuracy is calculated.

In purposive sampling you would collect data based on the aim of your research. The characteristics of the population from which you will draw a sample will be taken into consideration.

A quota sample is drawn from a population that represents the profile of your research target. The profile is based on your research question and the focus of your study.

In random sampling each possible sample has an equal chance of being chosen for your research.

In reputational-case sampling you will choose a sample recommended by an expert in the field.

In snowball sampling you would identify a small sample or perhaps just one individual who meets the requirements for your research. This individual or small group can recommend other individuals or groups that meet your requirements.

Stage sampling is the selection of a sample in stages, that is taking samples from samples.

Uncategorised

ARTICLE 35: Research Methods for Ph. D. Studies: Sampling Part 3 of 6

1st Sep 2020 No Comments

Written by Dr. Hannes Nel

Introduction

I will discuss event sampling, extreme-case sampling, matched sampling, multi-phase sampling, non-probability sampling and opportunistic sampling in this article.

Event sampling

Of all the types of sampling, event sampling is the most a research method rather than a data collection method.

You would typically select a small group of individuals to investigate.

It requires counting the number of times that an occurrence is observed or a statement, word, etc. is found in a data source, for example a document.

Counting words, statements, etc. in a document is easy if it is available in electronic format.

An example of this is to count the number of times that the words “he” and “she” respectively are used in a book to determine if there is gender discrimination against one or the other.

You can use your laptop computer to count the words for you.

Any word processor on computer has the facility to count words.

When your research requires you to observe certain repetitive actions, you will need to develop statements or criteria for the action.

This can be compared to the cricket scorebooks that some clubs still use, although computer programmes for even that have been developed.

The task becomes more difficult if it is not only necessary to count the number of times actions are repeated, but also the chronological order in which the actions took place.

An example of this is where you need to determine what kind of response a certain action invites, such as how people respond to a smile, compliment or insult.

Extreme-case sampling

Extreme-case sampling is used when information about unusual cases that may be particularly troublesome, or enlightening are sought.

Such a sample might represent the purest instance of a phenomenon that you wish to investigate.

Obviously, such cases are likely to be exceptions to the rule.

You might identify and investigate extreme cases to develop a richer, more in-depth understanding of a phenomenon.

You can also use extreme-case sampling to lend credibility to your research project.

Extreme-case sampling is often used in conjunction with other sampling methods.

In such events the extreme-case will be investigated in the context of the other sampling method.

The extreme-case can be used as an outlier, that is an observation that takes on an extremely high or extremely low value, thereby strengthening the norm.

An extreme-case can also be investigated on its own.

For example, where you do research on a specific individual who is extremely gifted, rich, strong, etc.

Matched sampling

In matched sampling one member of a control group is matched to a member of an experimental group.

Matching is done randomly, but the independent variables that are considered important for the research are taken into consideration.

Pairs of participants are matched in terms of the independent variables.

For example, age, gender, academic qualifications, etc. and only then assigned to the control or experimental group.

Matching pairs are used mostly when experimental research methods are used.

This implies that it is more valuable to quantitative research than to qualitative research.

However, experimental research methods can be used with qualitative research methodology.

Therefore, matching pairs can also be used to compose samples for such research.

Multi-phase sampling

Multi-phase sampling is a form of stage sampling.

The difference between multi-phase sampling and stage sampling is this:

In the case of stage sampling the criteria on each level are pretty much the same for each level of sampling in a particular region or field.

You will take samples of samples.

For example, your initial sample can be 1000 students randomly selected at a university.

Your next level sample can be 500 students who enrolled for psychology.

Your third level can be the 100 students who achieve the highest marks in the final exam.

In multi-phase sampling the criteria change on each ensuing level.

For example, the first stage criterion might be region.

The second stage criterion can be students who excel in numeracy skills.

The third stage criterion can be students with blue eyes.

Non-probability sampling

Convenience sampling, quota sampling, dimensional sampling, purposive sampling and snowball sampling can all be conducted in a non-probability sampling manner.

In non-probability sampling you, as the researcher, select the sample based on your own subjective judgement.

Non-probability infers that randomness ins not required, and the accuracy of the research findings need not be accurate to a good measure of probability, for example 90%.

In non-probability sampling, you will do research on a specific group to obtain knowledge about the group.

You will not claim general applicability of your findings.

The group, therefore, is important rather than any wider population.

This is often the case with small-scale research.

For example, just two or three faculties in a university,

Or just the lecturers in a small number of faculties.

Small-scale research often uses non-probability samples.

Meaning that not all available individuals have an equal chance of being selected for the sample.

The more expertise you have of the field of study, the better will you be able to select your sample.

Such samples are far less complicated to set up than a randomly selected sample.

However, keep in mind that they probably do not represent a substantive population.

Then again, they are less expensive than random samples and can achieve the purpose of the research if the general applicability of the findings is not important.

Small-scale research can also be used to pilot data collection tools, such as questionnaires, before they are used in actual research.

Opportunistic sampling

Opportunistic sampling means taking advantage of unanticipated events, leads, ideas, hints, and issues.

Accidental sampling is similar, but slightly different, as we saw in my previous post on convenience sampling.

The sample is composed of items upon which you stumbled by chance.

People who are available at the time of your study can be included in the sample.

They will need to provide valuable information and they must fit the criteria that you need.

A friend, who unknowingly shares valuable information with you at a party can be included in your sample.

Opportunistic sampling can be risky because the data that you collect can often not be generalized.

It might, furthermore, be difficult to corroborate the information.

Your source might not be reputable and might not have any credentials or qualifications making him or her an authoritative or expert source.

Summary

Event sampling is where you would do research on one or a small number of people, occurrences, or phenomena.

Extreme-case sampling also focus on just one or a small number of people, occurrences, or phenomena. Only here you will look for an unusual case or cases.

In matched sampling members of a control group are compared in terms of independent variables to members of an experimental group.

In multi-phase sampling the sample is broken down in terms of different criteria for each phase or level.

Non-probability sampling is selecting a sample that does not necessarily represent the population.

The result of this would be that your research findings cannot be generalized.

Opportunistic sampling means using any data sources that you can find without deliberately selecting them.

News

ARTICLE 34: Research Methods for Ph. D. Studies: Sampling Part 2 of 6

30th Aug 2020 No Comments

Written by Dr. Hannes Nel

Introduction

I discussed an introduction to sampling in my previous post.

I also discussed the first two types of sampling, namely boosted sampling and case study sampling.

To refresh your memory, boosted sampling is a type of sampling where you take specific steps to ensure that individuals who might otherwise have been omitted from the sample, are included.

Case study sampling is an investigation into a specific instance or phenomenon in its real context.

I will discuss cluster sampling, convenience sampling, critical-case sampling and dimensional sampling in this post.

Cluster sampling

A large and widely dispersed population brings with it, administrative problems.

Let’s say you want to investigate the impact of video games on teenagers.

It would be difficult to select and investigate a sample of, say, 1000 teenagers across the entire country.

You can reduce the time and work by selecting a smaller geographical area within the country.

This might have a negative effect on the generalizability of your findings.

It depends on your purpose with your research if this would be a problem or not.

But at least your research topic will now be viable.

This is an example of cluster sampling.

You can go one or more steps further by subdividing the geographical region into towns and towns into suburbs.

This is called ‘multi-stage cluster sampling’.

Limitations can be:

Younger teenagers will probably not be affected the same as older ones.
Teenagers in rural areas will not be affected the same as children living in urban areas.
There are many other possible variables that can have an impact on the results of the research.
For example, single-parent versus both parents in the family, personality traits, home conditions, etc.

The obvious risk with cluster sampling is that you can build in bias by excluding some possibilities.

The type and purpose of the research will determine if cluster sampling should be used or not.

Convenience sampling

Convenience sampling means selecting inputs and data from whoever happens to be available.

It is, thus, a rather opportunistic type of sampling.

It can also be accidental in nature, although there is a subtle difference between convenience sampling and opportunistic or accidental sampling.

Convenience sampling means making use of sources of information that you know of and that are easy to obtain.

Opportunistic or accidental samples are sources of information that you are not aware of in advance and that ‘falls into your lap’.

Convenience sampling is not a scientific way in which to collect data or to compose a sample.

It involves choosing the nearest individuals to serve as respondents.

It also means continuing the process until the required sample size has been reached.

Captive audiences such as students in a class are sometimes asked to discuss issues or complete questionnaires.

Convenience sampling is unlikely to deliver generally applicable findings.

Therefore, you will only use convenience sampling if achieving generalizable findings is not important to you and your research.

You must acknowledge that your findings are not applicable to a larger population than the small sample that you selected for your research.

It is even possible that your findings apply to your sample only.

Critical-case sampling

Critical-case sampling is sampling where it is important to obtain maximum applicability.

If the information and findings hold true for a critical case, it is likely to hold true for other cases and communities as well.

Critical cases are those that are likely to yield the most or most important information that will prove a hypothesis or solve a problem and that will have the greatest impact on the development of knowledge.

When selecting a critical-case sample, you would be looking for a ‘decisive’ case that would help you decide about which of several different explanations for a phenomenon, event or behavior is most plausible or most widely regarded as representing the general profile of a community.

For example, if one or a small number of people living in a jungle somewhere proves to have an immunity to the venom of a snake that can be found in that part of the world, one can probably deduce that there are people who have developed an immunity to the venom of that snake.

It does not necessarily mean that the tribe from which the sample was taken is immune to the venom of the snake.

This is a rather dramatic example and one will probably need to do more research and tests before you can claim with certainty that all people belonging to that tribe are immune to the venom.

It would also be interesting to do additional research on why the people are immune to the venom.

But please keep in mind that this example is a figment of my imagination – I don’t think there is such a tribe or snake.

Critical-case sampling is an efficient way in which to conduct research.

It need not be expensive because only one or a few items should be enough to prove your hypothesis or solve your research problem statement or question.

Dimensional sampling

Dimensional sampling is a refinement of quota sampling.

It can be used to reduce the sample size.

Dimensional sampling means selecting participants in the sample group in terms of a combination of criteria that you feel they should meet.

For example, when doing research on the correlation between perseverance and cognitive thinking ability, you can draw up a table with criteria for perseverance at the top (as column headings) and cognitive thinking ability down the side (as row headings).

A table like the one in the example can easily be converted into graphs of different types.

Summary

Cluster sampling is mostly used where the population is widely dispersed.

You will need to choose a cluster in the population that is representative of the population and that can realistically be investigated.

Convenience sampling means accepting any relevant data sources that you can easily find for your sample.

Convenience samples can be found by accident or coincidently.

Critical-case sampling focuses strongly on the purpose of your research.

The sample will often be too small to deliver generalizable findings.

Dimensional sampling is sampling where different characteristics of the sample are compared in the hope that the intersections of comparative data will deliver significant findings.

News

ARTICLE 33: Research Methods for Ph. D. and Master’s Degree Studies: Sampling Part 1 of 6

29th Aug 2020 No Comments

Written by Dr. J.P. Nel

Introduction

I will discuss sampling in this and the five postings following on this one.

How large should your sample be if you were to do research on factors that make some people more susceptible to the COVID-19 virus than others?

Would you even use a sample, or would that be irresponsible for a topic that threatens human life as we know it?

Should one entrust an individual with such a critically important issue, or would it be something that should be done by a large team of researchers in as many countries as possible?

The three questions that I just asked should already show that sampling is not suitable for all research topics.

It is valuable for academic purposes and can be used if the main purpose is to broaden our knowledge.

As we all know, sampling is used to study the social and medical impact of the COVID-19 virus.

We also know that the findings of such research are sometimes questioned.

And we know that findings are sometimes proven wrong.

Sampling is largely about data collection.

However, it also includes the analysis of data.

That is why I am discussing it as a research method rather than a data collection method.

Obviously, sampling cannot be a stand-alone research method.

It must always be linked to other research methods.

The type and size of sample that you choose for your research can determine if your research will succeed or not.

You should choose the size and type of sample after you have chosen the target group for your research.

In quantitative research, you might need to calculate the size of the sample based on what percentage probability of accuracy you will need.

Dedicated computer programmes can often work out the sample size for you.

In qualitative research factors like expense, time available and where your target group is will determine the size of your sample.

Where the target group is, impacts on distance and accessibility, which will determine what size sample you will be able to reach and deal with.

The data collection method or methods that you will use will also have an impact on the size of your sample.

Although a sample should be large enough to provide valid and generally applicable information for a community, it should be small enough for you to manage.

Sampling in qualitative studies focuses on the quality of the information collected and not on the number of participants.

A sample of ten to twelve people in a community of 100 people is often acceptable.

Researchers in natural sciences will probably frown upon such a small sample or the unscientific way the sample size is decided upon.

Keep in mind that the response rate to questionnaires is often low.

You will do well if you achieve a 10% or more response rate.

If you need to receive at least 100 completed questionnaires back, you will need to send out at least 1,000 questionnaires.

To play safe I would send out double that number, i.e. 2,000 questionnaires.

Because if you don’t receive enough completed questionnaires you might need to send out more later.

You will not be able to receive as many responses through interviewing as you would through sending out questionnaires via the internet or the postal system.

Then again, you will not need to hold as many interviews as you would send out questionnaires.

Your sample must be representative of the population forming your target group.

Representativeness is determined by the size and composition of your sample.

A too-large sample is better than a too-small one.

You should, for example, not send your questionnaires only to people living in an old age home if you are doing research on people of all ages.

A random sample in terms of gender, age group, population group, intellectual capacity, interests and many more are often needed.

I will share 26 different types of sampling with you in this and five future articles.

Obviously, it would not be a good idea to discuss all of them in one post.

Therefore, I will spread them over six posts as shown on the slides.

Here, I will start with boosted sampling and case study sampling.

This article:

Introduction to sampling.
Boosted sampling.
Case study sampling.

Article 2:

Cluster sampling.
Convenience sampling.
Critical case sampling.
Dimensional sampling.

Article 3:

Event sampling.
Extreme case sampling.
Matched sampling.
Multi-purpose sampling.
Non-probability sampling.
Opportunistic sampling.

Article 4:

Probability sampling.
Purposive sampling.
Quota sampling.
Random sampling.
Reputational-case sampling.
Snowball sampling.
Stage sampling.

Article 5:

Stratified sampling.
Systematic sampling.
Theoretical sampling.
Time sampling.

Article 6:

Typical-case sampling.
Unique-case sampling.
Volunteer sampling.
Closing remarks.

Boosted sampling

Boosted sampling is a variant of purposive sampling.

It is a type of sampling where you take specific steps to ensure that certain individuals or types of individuals, who might otherwise have been omitted or underrepresented in the sample, are included.

People with special needs are often not included in samples because there are so few of them.

You can, for example, “boost” the sample by intentionally including people with special needs.

It is important to ensure that all relevant people are represented in the sample if you intend to do statistical analysis and quantitative research.

You might need the responses of specific people even if you are doing qualitative research.

Case study sampling

A case study is an investigation into a specific instance or phenomenon in its real context.

It is used to illustrate a general principle, pattern behavior, etc.

Case studies can establish cause and effect in a real context.

The value of a case study is largely dependent on the size and composition of the sample.

The advantage of a case study is that it serves as a small sample of a large whole, which makes it much more manageable.

This, of course, will only be so if the case being investigated is representative of the entire population being investigated.

It is important to allow case study events to speak for themselves rather than to depend too much on your interpretation, evaluation and judgement.

Data should be collected systematically and rigorously.

This means that you need to prepare well for case study research.

You should never manipulate a case because that will lead to false information and conclusions.

It is, however, not wrong to select a case as your sample to fit the purpose of your research.

For example, it would not be wrong to select tramps in an area frequented by them if you are doing research on factors that make people decide to turn their backs on society.

Summary

Sampling includes data collection and analysis.

It is always used in combination with other research methods.

The size of the sample will be statistically calculated in quantitative research.

Factors like time, funds, data collection methods, the purpose and topic of the research will decide the size of the sample in qualitative research.

The sample must be representative of the population for your research.

In boosted sampling you will take deliberate steps to ensure that certain elements in the population are included in the sample.

A case study sample will focus on a specific group or phenomenon.

News

ARTICLE 32: Research Methods for Ph. D. and Master’s Degree Studies: Literature Study

27th Aug 2020 No Comments

Written by Dr. Hannes Nel

Introduction

People who do not like reading will probably shy away from making use of literature study for a Ph. D. or master’s degree.

I can understand such an attitude, especially if you don’t have any specific questions that you need answers to.

Or an academic problem to solve.

For some, there are many more exciting things to do rather than to read a book.

I guess studying late at night, over weekends, public holidays and university holidays is not something to look forward to.

Perhaps it takes a special kind of person, or some alien external force to motivate “normal” people to spend what they regard as leisure time engrossed in books.

However, if you embark on Ph. D. or master’s degree studies you will need to sacrifice some of your social freedom.

That does not mean that you will need to live like a recluse.

It only means that you will need to read a lot.

The point is this – as an adult, you are the captain of your ship.

Only you can decide what you should do with your life.

You need to be absolutely motivated to study for a Ph. D. or master’s degree.

So, let’s discuss literature study.

Literature study

Literature study will form part of most research work, regardless of what main research methods are used.

With the expansion of online data sources, it is becoming progressively more possible to undertake research without leaving your study or office.

You can even do research while having something to eat and drink in a restaurant or a bar.

Texts and records can be researched in four distinct ways:

The study of content.
The social construction of documents and records.
Documents ‘in the field’
Documents in action and documents in networks.

However, doing all your research by studying documents is deskwork rather than fieldwork.

When documents are included in the dataset, they tend to serve as ‘background material’ for many research projects.

Literature is often used to crosscheck oral accounts.

The empirical opportunities to analyze documents are endless: newspaper articles, advertisements, policy documents, government reports, blogs and web sites, schedules, letters, posters, pamphlets, brochures, campaign material, etc.

Everyday life offers a range of situations that involve forms and documents.

Particularly regarding interpersonal encounters within various institutions.

Even so, researchers interested in identifying practical examples may find it hard to recognize relevant events in a pile of documents.

The majority view of documentary data in qualitative research is that documents are detached from social action.

In contrast to an interview, the document appears more stable and fixed, whereas the content of an interview seems more dynamic and alive.

Documents can be quite lively agents in their own right rather than merely containers of text.

They can tell people what to do.

They can stir up conflicts.

They can evoke emotions such as anger, relief, envy, pride and despair.

Additionally, people do things with documents.

They can use them for various purposes, both intended or unintended.

They exchange documents.

They hide documents.

They write books and make movies about documents.

To recognize the aspects of documentary data, fieldwork rather than deskwork is required.

Summary

All researchers must do some literature study, regardless of which research approach or method they mainly use.

Texts and records are studied for the content, social construction, corroboration of other research methods and to support action research.

Literature study mostly means reading and analyzing books and other documents.

It can also include doing research on the internet.

Documents often trigger chains of interaction far beyond the original piece of paper.

Literature is often used to confirm research through other research methods.

Some researchers would rather do without literature study.

They prefer practical fieldwork or research in a laboratory that is not contaminated by old data.

News

ARTICLE 31: Research Methods for Ph. D. and Master’s Degree Studies: Historical Research

27th Aug 2020 No Comments

Written by Dr. Hannes Nel

Introduction

Historical research can be conducted with the aim of answering a wide range of social research questions.

It is a systematic process of describing; analyzing and interpreting; and comparing the past.

The past can, of course, also be compared to the present.

It is based on information from selected sources as they relate to the topic under study.

Descriptive research

When conducted with a description in mind, historical research attempts to construct a map of the past.

It can also describe a developmental trajectory of events, for example, an educational system, political developments, community growth or deterioration, etc.

It involves locating events in time and place and requires sensitivity towards understanding the context within which an event took place or developed over time. Trajectories of development are plotted, and it may or may not try to explain how or why events occurred.

Historical research used for descriptive purposes also establishes background information about events.

Analytical and critical research

In analytical research, relationships between factors or events are critically interrogated within the context in which they occurred with the aim of exploring the possible causes and impact of the factors or events.

It requires critical, analytical scrutiny of documents and a process of cross-checking records and reports about incidents.

It aims to impact decision-making and policy formulation.

When conducting historical research, you might need to search for successes or mistakes from the past in order for a society or even an individual to avoid repeating mistakes made by others and to capitalize on the successes of the more fortunate or wiser.

The process of analytical research is complicated by the fact that there is seldom just one cause of an event.

You can never be certain if you did not omit the consideration of other important factors or events.

All sources of information should, therefore, be subjected to evaluation through processes known as external and internal criticism.

External criticism refers to the authenticity of the information.

For example, is the document that you are evaluating real or fake?

Internal criticism refers to whether the information is credible.

You need to determine if the information is consistent and accurate.

Rigour and sound analytical critique of sources lend considerable validity and reliability to the findings and conclusions reached.

Inferences about intent, motive and character are common, with the understanding of appropriateness to the context of the period.

Comparative research

Comparative historical research is much wider in scope than other historical research designs because the units of analysis are often whole societies or systems within societies.

It implies systematic research for similarities and differences between the cases under consideration.

Comparative researchers usually base their research on secondary sources, such as policy papers, historical documents, official statistics, etc.

Some degree of interviewing and observation can also be involved.

Historical comparative researchers generally prefer first-hand accounts, written by people who witnessed something personally, rather than documents derived from other secondary sources.

These researchers need to verify two important aspects regarding documents:

The validity of the documents’ content and its authenticity.

The content of a document is valid when it is not distorted, exaggerated or false.

Validity can be checked by making use of triangulation or any other form of corroboration.

Additional sources of information can be other, similar books, alternative documents, certification, living witnesses, demonstrations by the individual who claims authenticity, traces of validating substances, etc.

Authenticity refers to whether a document is a forgery.

Historical research can adopt paradigmatic approaches such as constructivism, critical race theory, empiricism and post-colonialism.

Sources of data

Broadly speaking there are four types of historical evidence that you can use – primary sources, secondary sources, running records and recollections.

Customarily, researchers rely on primary sources.

That is, the original source texts also called archival data.

Archival data are mostly kept in museums, archives, libraries or private collections.

Emphasis is given to the written word on paper, although modern histiography can involve any medium.

Substantial historical data is already captured electronically.

The internet is already a popular source of information, even though authenticity and accuracy are difficult to confirm.

Secondary sources are the work of other researchers writing about the issue being studied.

In its widest sense, a document simply means anything that contains text.

Official reports, records from schools, hospitals and courts of law, films, photographs, reports from journals, magazines, newspapers, letters, diaries, emails, and even graffiti scrawled on a wall are all examples of documents.

Running records are documentaries maintained by organizations.

For example, minutes of meetings, records of events, commentaries about events, war diaries, etc.

Recollections include autobiographies, memoirs, and oral relay of historical information.

Summary

Historical research answers a wide range of social questions.

It can be used to describe, analyse or interpret the past.

People can learn from historical events if it is accurately and objectively communicated.

Both mistakes and successes from the past can provide lessons to learn.

Analytical research attempts to provide the basis for understanding the past by exploring past events and trends and applying these to current events and trends.

Analytical research is complicated by the fact that successes and mistakes are mostly attributable to many variables.

Accuracy, authenticity and validity of data can be achieved through external and internal criticism.

Historical research makes use of four types of evidence – primary sources, secondary sources, running records and recollections.

News

ARTICLE 30: Research Methods for Ph. D. and Master’s Degree Studies: Grounded Theory

26th Aug 2020 No Comments

Written by Dr Hannes Nel

Introduction

Hello, I am Hannes Nel and I will discuss Grounded Theory in this post.

What is grounded theory?

Grounded theory is a type of inductive thematic analysis (ITA).

It was developed by Glaser and Strauss in the 1960s.

Glaser and Strauss supported symbolic interactionism as a philosophical perspective.

How is grounded theory used?

Grounded theory uses inductive reasoning to generate the theoretical understandings, of research by grounding the theory in the data that the researcher collected.

It is a highly systematic method for mostly studying social experiences, interactions and structures.

Grounded theory discovers, develops and provisionally verifies phenomena.

This means that the data originate in the framework for the study and should deliver logical and relevant conclusions.

Integrating grounded theory with other research and data collection methods

It is almost always necessary to use grounded theory in conjunction with one or more other research methods.

Any data collection method may be used in conjunction with grounded theory methods, bearing in mind that data collection should build on a naturalistic, interpretive philosophy.

Grounded theory methods specify analytical strategies, not data collection methods.

Grounded theory:

Is a qualitative research approach.
Requires an open mind, objectivity and ethical and responsible analysis of data.
Is especially popular amongst those who study humanistic sciences.
Can also be used for the study of non-human phenomena.

The purpose of grounded theory

The primary purpose of grounded theory is to generate theory from observations of real life.

Grounded theory aims at the discovery of regularities, the identification of categories or elements and the establishment of their connections.

Theoretical models and new theoretical concepts and arguments should be created and continuously revised as you collect and analyse data.

Grounded theory holds as a basic view that qualitative researchers do not go around testing an existing body of knowledge, but rather that they build new theory by allowing their data collection to steer their thoughts and conclusions into the unknown.

The grounded theory process

Grounded theory research should be done in a specific and well-defined context.

The research should be grounded in social reality and not be just an exercise in theorizing.

It uses a typical research process of data collection, data analysis, coming to conclusions, and formulating findings.

Findings should be transformable into formal theoretical models.

The process of collecting data is a prerequisite for analysis, while theory development should result from the analysis.

Researchers sometimes think that grounded theory is about the research process, especially data collection and analyses.

Although data collection and analysis are important research activities, the essence of grounded theory does not lie in the research process but rather in the attitude of the researcher towards the data and the purpose of the research.

It requires that each piece of the data is systematically compared with other data on the same or related issue or topic.

You should not ignore small units of text.

It just might have the potential to improve current theory and practice.

At the same time, you should not waste time with data that is clearly of no significance, because analysis is a time-consuming activity.

You can compare existing data with other existing data or with new data.

Grounded theory is based on the subjective experiences of humans.

You may also use your own experiences to understand the experiences of others.

Guard against just adopting the ideas, perceptions or models of others.

If you do this, you run the risk of just packaging old, existing knowledge differently.

Verification is a natural element of any scientific research because it strengthens the authenticity and validity of the findings and provides you with a measure of security.

Data collected should not be over-verified, because grounded theory epistemology leans strongly towards the generation of new theory rather than the analysis of existing theory.

Deconstruction can be used to lend a good measure of authenticity to the data.

Do not neglect to acknowledge the work of other researchers that you consulted and quoted.

Computer software

You can use dedicated computer programmes to arrange, compare and analyse the data that you collected.

ATLAS.Ti is an example of software that you can use.

There are a good number of others. I am just mentioning ATLAS.Ti because it is the one that I used and am familiar with.

You can easily find suitable software by just Googling for them.

Most dedicated computer programmes make use of coding.

Coding can be described as a sophisticated form of notecards like the ones that we used many decades ago.

You will create codes for salient data with most of the available relevant software.

You can also write explanatory notes in the form of memorandums.

The programme groups related codes and memorandums together.

This enables you to get a clear and holistic picture of concepts and arguments so that you can more easily come to conclusions and findings.

Your findings should be or lead to new knowledge, theories and models.

From the codes and memorandums, new theory and new theoretical models can be discovered through inductive reasoning.

You will, of course, not be required to develop new knowledge or theories on master’s degree level, but you will need to show that you understand and can apply existing knowledge and theories.

Inductive reasoning entails systematic data collection and analysis which leads to discovery, development and verification.

Most importantly, dedicated programmes substantially simplify the process of writing your research report.

Grounded theory methodology needs not to be limited to computer analysis only.

You can, for example, still use the old notecard system or you can develop your own system on a computer.

The value of grounded theory

Grounded theory enables you:

To step back and critically analyse situations.
To recognise the tendency towards bias.
To think abstractly.
To be flexible and open to helpful criticism.
To be sensitive to the words and actions of respondents.
To adopt a sense of absorption and devotion to the work process.

Utilizing grounded theory for research should enable you to see beyond the ordinary and to arrive at new understandings of social life.

The most important value of grounded theory is that it enables you to generate theory and to ground that theory in data.

Paradigms that can be used with grounded theory

Before we look at paradigms that can be used with grounded theory – don’t be too concerned if at this stage you do not know the paradigms.

I will discuss 29 such paradigms in later posts.

I suggest that you then return to my earlier posts on research methods to get the bigger picture.

Any paradigmatic approach can be used with grounded theory.

Mostly, however, grounded theory displays elements of post-modernism as well as symbolic interactionism.

Post-modernism lends itself to the achievement of formal theory while symbolic interactionism implies that the study is grounded in a specific empirical world.

As already mentioned, grounded theory requires elements of interpretivism as well.

There are two versions of grounded theory, namely objectivist and constructivist grounded theory.

Objectivist grounded theory is rooted in a positivist paradigmatic approach.

The objectivist viewpoint claims that it is possible to discover objective truth.

The data already exists, and you will need to discover theory from them.

Constructivist grounded theory has its roots in an interpretivist paradigmatic approach.

The constructivist viewpoint rejects the objectivist viewpoint, contending that there is no objective truth waiting to be discovered.

Constructivist grounded theory assumes that truth exists only through interaction with the realities of the world.

Meaning is, therefore, constructed rather than discovered.

Summary

The following are the elements of grounded theory:

The purpose of grounded theory is to build new theory.
Current theory or observation can serve as the basis for new theory.
Grounded theory deals with how data and phenomena are interpreted and used rather than how they are collected.
You should systematically review units of data as they become available.
Any research method should utilize the philosophy behind grounded theory, meaning that any researcher should be open-minded and objective.
Building new theory requires analytical induction, meaning that new theory emerges from collected data inductively through a series of steps.
Grounded theory requires the development of five interrelated properties.
- The theory must closely fit the relevant field of study in which the new theory will be used.
- The new theory must be readily understandable to laymen concerned with the field of study.
- The new theory must be relevant to a multitude of diverse daily situations within the focus area of the field of study.
- New knowledge should be generalizable as widely as possible.
- The new knowledge must allow those who use it to have enough trust in the validity and accuracy of the new knowledge, theories and models.
Dedicated computer programmes enable you to discover regularities in data, to identify categories or elements and to establish their connections.

News

ARTICLE 29: Research Methods for Ph. D. and Masters Degree Studies: Field Research

25th Aug 2020 No Comments

Written by Dr. Hannes Nel

Introduction

Once when I sent out questionnaires for a research project, the CEO of a manufacturing plant collected and worked through the questionnaires completed by his employees.

There were probably 100 to 200 completed questionnaires. He removed all the questionnaires that he felt could put him in a bad spot and sent me the rest.

Of course, I did not use any of them.

I was fortunate that one of his employees phoned and told me what he did.

Field Research

For many researchers, the collection of data involves or at least includes fieldwork.

Not all field research deals with people.

Also, fieldwork is not limited to research in nature only.

Field research can require the use of quantitative or qualitative analysis.

Preparing for field research

You should first refine your research project and develop your data collection instruments before you embark on fieldwork.

This is because your topic and the context in which your research will be done will determine if you should do fieldwork or not.

Where field research will take place

Field research, or fieldwork, means conducting empirical research in real-world settings.

You can do fieldwork in a classroom, observing students or lecturers, in factories, on ships, in aircraft and many more.

Doing fieldwork is not a must for all research.

You will use fieldwork if your research topic demands it if you feel that it will enable you to do accurate and valid research, if your study leader expects it from you, if you can afford it, and if it is something that you will enjoy doing.

The research process

In the spirit of grounded theory, you should be open-minded about the realities that you encounter.

Let the data that you collect lead your thinking processes.

Don’t try to bend what you see and experience to fit your preconceived ideas.

Preconceptions that you should get rid of include personal beliefs and initial theoretical propositions.

It might be necessary to divide your observations into different categories.

Don’t categorize your observations and events prematurely, though.

The first days in the field are often seen as the most challenging and emotionally rewarding.

Meeting a new group of people in their environment, about which you might not know much, can be uncomfortable, perhaps even intimidating.

Remember that such people might not trust you in the beginning.

People are suspicious of the unknown and they may resist you and your research.

Your research will be as intimidating to them as their environment to you.

If your doctoral or master’s degree studies are an extension of your previous studies, you might feel more comfortable with the environment in which you will do research.

Even so, you will probably encounter some new experiences and observations.

Doing site visits is a formally recognized way of doing fieldwork.

Site visits can be the only data collection method that you may use.

However, it is mostly necessary to collect other data as well.

You should be able to develop categories, propositions and eventually meaning based on what you experience and observe in the field.

Participant observation takes place during a site visit and a site visit may include other data collection activities.

Can you see how you can integrate different research methods?

Experimental methods, for example, can fit in well with fieldwork.

An advantage of site visits is that they enable data to be collected from many field settings as part of the same study.

The data from any single setting or site may be limited in terms of quantity, quality and variety.

Studying cross-site patterns might be necessary for comparison, to identify trends or patterns in phenomena or behavior and for corroboration of data.

Not just any field setting will be suitable for research and data collection.

Difference In functions, context, time and size can render some sites irrelevant to your research topic.

It might sometimes be necessary to visit the same site at different times and more than once.

An example of this is where you do research on the effect over time of global warming on glaciers, rivers, vegetation, etc.

Site visits are likely to be more rigid than participant observation.

Site visits usually follow a pre-established schedule, as well as an agenda while you are in the field.

Site visits can be time-consuming and require substantial preparation.

It can also require good timing, for example, if you were to do research on the migration of salmon up rivers to spawn.

Interviewing, conversing with participants, and observing them while they do something that you do research on will also require good planning and timing.

Members of a field setting for interviewing may have helped you to arrange the schedule.

They will probably use the opportunity to schedule the interviews so that it will suit them.

The disadvantage of this is that they can also prepare their responses to your questions in advance, thereby making it artificial and probably not valid.

Their responses may be idealized and what they think you would like to hear.

This is an example of reflexivity.

Reflexivity in this context means that the presence of the researcher affects the people being studied.

The same kind of situation can happen in the case of participant observation.

A further complication arises when you are accompanied by your host during the site visit.

The host may wish to monitor you and see and hear what you learn from his employees.

Paradigmatic approaches that fit well with field research include behaviorism, constructivist, critical race theory, critical theory, functionalism, neoliberalism, positivism, pragmatism, radicalism and scientism.

Summary

Field research can deal with people, phenomena in nature or even history.

It may require qualitative and/or quantitative research.

You will need to do a literature study and decide how you will collect further data first before you will know what kind of field research you should do.

Fieldwork can be done almost anywhere.

You need to be open-minded and objective about the data that you collect through fieldwork.

Also, be prepared to go through a process of mental adjustment. You will need time to grow accustomed to strange people and an unknown environment.

The contextual conditions between participant observation and interviewing are not the same, but both can be damaged by reflexivity.

You can integrate fieldwork with most other research methods.

Fieldwork is not always necessary for all research on especially the doctoral level.

News

ARTICLE 28: Research Methods for Ph. D. and Master’s Degree Studies: Experimental Methods

25th Aug 2020 No Comments

Written by Dr Hannes Nel

Introduction

Can lessons from events be gained in a laboratory?

Lessons that will apply to every place and all people?

You be the judge of that after reading this article.

Quantitative experimental methods

The essential feature or experimental methods is that you deliberately control and manipulate the conditions which determine the results in which you are interested.

To achieve this, you will need to introduce an intervention and measure the difference that it makes.

It is mostly used by people doing quantitative research and in natural sciences.

The precondition for research to be experimental is that it should meet the requirements for scientific research.

The requirements for scientific research are that the researcher must have control over the variables, careful measurement, and establishing cause and effect relationships.

An experiment involves making a change in the value of one variable – called the independent variable – and observing the effect of that change on another variable – called the dependent variable.

Using a fixed design, experimental methods can be confirmatory, seeking to prove or disprove a hypothesis, or explanatory, which would be discovering the effect of certain variables.

The variability and validity of experimental methods often depend on the ability to replicate the results from an experiment in similar situations and generalizing the findings to other similar situations.

You can, for example, test the psychological effect of the COVID-19 pandemic on people by asking them relevant questions during an interview.

You can also ask people to complete a questionnaire.

Responses to the questions can be measured in different ways, depending on the type and context of the experiment.

In the experimental methodology, generalization is referred to as “external validity”.

External validity means that the same results will be obtained with different groups and in different places.

You can imagine that external validity will not be easy to achieve.

This is mostly because of the number of different external variables that can influence your experiment.

The less external variables there are, the better will the chances be that you will achieve external validity.

Experiments can be experiments in a laboratory, field experiments or natural experiments.

Regardless of where the experiments are done, they require quite some planning and preparation.

Most, if not all, tests that are done in a laboratory are examples of experimental research.

Experiments in a laboratory

A laboratory experiment would be an experiment that is conducted under highly controlled conditions where accurate measurements are possible.

Laboratory experiments are mostly easy to replicate.

That is because standardized procedures and measurements of variables are used.

There are not many external variables over which you have no or little control.

Laboratory tests, however, do not always replicate real life, with the result that findings will probably not be generalizable outside the laboratory.

Field experiments

Field experiments are done in real-life environments.

The researcher still manipulates the independent variable but in a real-life setting.

Findings in field experiments should be generalizable to real-life settings.

A limitation of field experiments is that there will probably be many external variables that might bias the results.

Consequently, it would be difficult for other researchers to replicate the results.

Natural experiments

Natural experiments are also conducted in real-life environments.

They differ from field experiments in that the researcher cannot manipulate the independent variables – events occur as they would in real life.

Behavior in a natural experiment is more likely to reflect real life because of its natural setting.

It might be necessary for covert observation.

That is, the target group for the research does not know that they are being observed.

The reason for this is that people often behave differently when they know that they are being watched than they would normally behave.

This is even worse if they know that video footage is being taken of them.

Covert research can have ethical consequences which you should consider and remove in advance.

The researcher has no control over external variables that might bias the results.

Experiments in qualitative research

The parallel in qualitative research is when a part or the whole qualitative research study is the subject of attempted replication of a phenomenon or event.

For example, within a single qualitative research study, replication can sometimes be achieved by conducting more than one of the same or at least similar case studies.

The more similar the findings from the cases, the more a replication might be claimed.

However, replication is not always possible because of the dynamic nature of human phenomena.

Paradigmatic approaches that fit well with experimental methods include behaviourism, empiricism, modernism, phenomenology, scientism and symbolic interactionism.

Summary

In experimental research:

You, as the researcher, manipulate the conditions for the research.
You must have control over the variables.
Careful measurement of phenomena or behavior will be necessary.
You will need to establish a cause and effect relationship.
External validity will not be easy to achieve.

Experimental methods can be confirmatory or explanatory.

An independent variable is the input variable, whereas the dependent variable is the outcome variable – the result.

It is not easy to achieve external validity with any of the experimental research types.

That is because of differences in variables between the place where the experiment is done and real-life conditions.

Experiments can be experiments in a laboratory, field experiments or natural experiments.

The conclusions of experimental methods can be achieved through qualitative research.

Close

In closing,

I used one additional source of information for this article. That is McLeod, S.A. (2012, Jan 14) Experimental method. Simply psychology: https://www.simplypsychology/org/experimental-method.html. I accessed the source on 10 March 2020.

The main source of information that I use for all my articles and videos on Ph. D. and master’s degree research is my book entitled Preparing Qualitative Research Reports.

News

ARTICLE 27: Research Methods for Ph. D. and Master’s Degree Studies: Ethnography

24th Aug 2020 No Comments

Written by Dr Hannes Nel

Introduction

Why would academics do research about people that they probably regard as inferior?

In a positive vein, probably because they want to help those who are less fortunate than them.

Did the researchers ask the people if they needed or wanted help?

I would think yes, at least most of the time. But not always.

And if the researchers try to help, will they do the right thing?

Unfortunately, not always.

I think it can be humiliating and frustrating to have outsiders dissect your value system and the way you have lived for centuries.

And then they tell you how you should live and what you should do.

Remember, these people know their culture and way of life and they are watching you while you observe them.

Sometimes it can be funny, but it can also be embarrassing and lead to serious damage and harm to both the people being researched and the researcher.

It is not easy for an outsider to understand the culture and being of another population group.

People with a populist orientation will probably feel that such differences are not important and that it should be eliminated.

That, however, is not the reality for many nations.

For an outsider ethnography as a research method is much more difficult than we think. That is also why many researchers embarrassed themselves when they tried to tell the world how people from other cultures think, live and do things.

What is ethnography?

Some may regard ethnography and ethnomethodology as the same thing.

They are not entirely wrong.

Ethnomethodology is the philosophy behind ethnic research.

Ethnomethodology claims that research deals with everyday life amongst people in a social setting; where currently observed behavior and interaction provides the most accurate and valid data.

For the purposes of this post, ethnomethodology is regarded as a paradigm that can be used with ethnography, which is a research method.

Ethnography, therefore, is the method by which ethnic research is conducted.

I will discuss ethnomethodology as a paradigm in a future post.

How did ethnography begin?

Ethnography mostly has colonialism as its foundation.

It relates to distant cultures that were occupied by the seafaring nations, largely between the fifteenth and seventeenth centuries.

Colonialism came to an end in the nineteenth and twentieth centuries.

The influence of colonization can still be seen in what used to be colonies and is popular as a topic of research.

Unfortunately, some researchers call any kind of quantitative research ethnography.

Ethnography has traditionally been associated with social and cultural anthropology.

Sources consulted in ethnography

Ethnographic research sometimes needs to be done through semi-structured interviews or literature study that will include official documents.

This can include historical data and not just current observations.

The process of ethnographic study

Ethnography encompasses forms of research that are extremely diverse from a methodological point of view.

Ethnography can have a contemporary or historical focus.

It is the aim of ethnography to establish the hidden inferences that distinguish, for example, different eating habits.

Such a phenomenon is then placed in its social and cultural context.

Ethnography, therefore, is oriented toward studying shared meanings and practices, or culture.

As a researcher, you might need to study a community for a long period, perhaps years to gain some understanding of their culture and value system.

Although ethnography is traditionally associated with long-term fieldwork, some aspects are employed in applied settings.

Applied, or current settings require a naturalistic research approach.

Observing individual and group behavior in its natural context and participating in that context can generate insights that might otherwise have eluded you.

Ethnography follows a holistic perspective, based on the premise that human behavior and culture are complicated phenomena and are composed of, and influenced by, a multitude of factors.

These might include historical precedents, the physical context in which people live and work, the social structures in which individuals are embedded and the symbolic environment in which they act.

Ethnography assumes that all human behavior is intentional and observable.

Research should, therefore, be oriented towards understanding the reasoning behind people’s actions.

This means that you, as the researcher, and often not a member of the community, will probably spend time living in the community observing and doing in-depth interviews, reading and researching primary source material and observing the lives of the people that you wish to study.

Eventually, you will compile all this data and analysis to create a full picture of the group you are studying.

The picture includes not only reporting what people do or say but also some analysis that tells us about the social structure and worldview of the community.

You should not disrupt the environment or routine of the target for your research.

This means that you, as the researcher, might need to be “invisible” and involve yourself in unobtrusive observation.

In this instance, an etic approach will probably be more effective than an emic approach.

You will need to establish trust and rapport between you and those whom you do research on.

Also, you should guard against becoming emotionally involved with the target group for your research because it might cloud your judgement.

Research reports are in the form of a narrative, with key evidence, such as detailed descriptions of episodes, being reproduced to illustrate your findings and recommendations.

Paradigmatic approaches that fit well with ethnography include relativism, behaviorism, constructivism, critical race theory, critical theory, ethnomethodology and post-colonialism.

Summary

Ethnography literary means “to write about a group of people”.

Research topics and problem statements or questions require the study of social and cultural processes and shared meanings within a group of people.

Participant observation is the method most used in ethnographic research.

Normally ethnographers will spend lots of time in the field to study the lives of people from within their naturalistic setting.

The “thick” detail of the data that you can obtain through participant observation usually fulfils the key criterion of validity far better than data obtained by other methods.

Ethnographic research can also consult data sources such as interviews and literature studies.

It also offers flexibility and can provide the basis for inductively generating new theoretical explanations.

Ethnographic research can:

follow a qualitative or quantitative approach,
will probably follow an etic approach, although an emic approach would also be possible, and
can be done by one or a group of researchers.

« Previous 1 … 9 10 11 12 13 … 29 Next »