Quarior, Thanks so much for your thoughtful reply! I think I understand your question a bit better now and can (hopefully) see where you're coming from
Source of the post
Ok, I hope you understand more the community and I said this because for my scolarity, is about statistic but I just in my first year but I have little lesson for crossing variables ( X explicative, Y to explain).
With your statistic survey, I think is more complicated with many variable qualitative with open answers.
Have a nice night/day.
Certainly, having lots of open free response questions means large amounts of messy data! I will have to spend some time tidying and processing before analysis, but I think it can be a productive process. Like, I enjoy seeing the categories, words, and frames that people use/provide when they have free-choice, because it kind of tells me a bit about the ways they actually think about things, rather than having to fit their frames/thoughts/experiences into narrower categories that I've already come up with? That's what I like about qualitative (and ethnographic) research more generally! Just hearing and learning with many people's stories can be so generative for learning about issues, practices, social groups, meanings, and often as a researcher you get surprised - like new angles or issues you hadn't even thought of before.
But then is anectodal or personal experiential evidence statistically significant or relevant on a larger scale? I mean I think that depends a bit on the research context, but personally think these kinds of methods are great for exploring new areas, and identifying topics/themes/issues/areas that are important, and then you can zoom in and return to them with methods that might be a bit more rigorous.
Standard practice for processing/analysing things like interview transcripts, text responses also involves iteratively developing codes for core themes or concepts, and then tagging the text for them, so you can actually see/say 'Oh well turns out half the respondents thought this one issue was really important and talked about it/framed it using this one concept that came from x place' or whatever, which can help you make claims about the body of data as a whole. Subject to interpreter bias? Sure, that is definitely an issue or argument to make, though there are other tools out there that you can use to help with the identifying of themes and concepts for coding that (I believe) are based on word co-occurrence Bayesian stat models, so help the researcher get around limitations of subjective interpretation... I'm playing with some of them now and they're pretty interesting and fun!
But some minor examples of interesting things that might emerge from bigger open-ended qual stuff like this could be that actually many people reference the same event as shaping or being important for the community, or that users of SE tend to settle into two camps of ways they learn through or how they experience the application, or respondents use particular kinds of language that are different to that used in other similar communities, etc. etc. So far, the responses to the survey are incredible, many people have clearly spent a lot of time forming really rich and thoughtful responses, which are full of all kinds of important and interesting information. Because it is so rich, though, analysis has to be a kind of involved, iterative process involving coming at it from a few angles, and will be a bit exploratory as well, and there does need to be that coding and processing (by the researcher or some other software) before the data is in any form to be getting basic stats from.
Does that... sort of make sense or address part of your question? I mean this isn't primarily a statistics project, as I'm not a sociologist with strong mathematics background! I'm an anthropologist (with additional sociology major) trained primarily in softer qual methods - ie., doing ethnography, running focus groups, conducting interviews, coding/analysing transcripts, doing content/thematic/discourse analysis, as well as using digital media research tools. So *definitely* not an expert in statistics specifically!
I hope you have a lovely day and would be super happy to discuss these issues further, I'm sure I could learn a lot from/with you!