Signal Detection Experiment

Signal Detection Experiment

Objectives

This study aims to design and build an experiment to measure the concept of signal detection theory. In this experiment, I will collect data on myself using a PEBL SDT task to compare how factors change within sensitivity and bias will affect my performance.

I want to use this experiment to understand how to increase the hit rate and reduce the false-alarm rate in user visual design. It is essential for users to quickly get the information they want in the user interactive design , rather than finding it through multiple searches.

Experimental Design

I use PEBL2.1, which is the Psychology Experiment Building Language. In this assignment, I used the system's own SDT demo experiment.

There are two mode types throughout the experiment, and each mode has three benchmarks. I will experience all methods under the same conditions.

In these six experiments, the first three belong to the 'A-B Response' base rate, which will follow the original experimental settings, A has on average 46 stars, and B has on average 54 stars. So the independent variables will be the number of stars in AB and my experimental performance, and the dependent variables will be hit rate, false alarm rate, and sensitivity.
The last three experiments belong to confidence rating. I changed the number of AB, B to higher (60) and the A to lower (40) to study whether such a change would affect sensitivity under the same conditions.

Method

I was the only participant, and the SDT tests were performed in a quiet working environment. In this case, I reduced the external interference to control and maintain the simulated experiment situation.

Data

According to the experimental results obtained by the two predetermined probabilities, the hit rate and false report rate can be calculated. Then the sensitivity and beta value under different probabilities can be obtained from the hit rate and false alarm rate I got. Here are the six results I received and calculated.

Then I create different ROC curves based on the base rate of'A-B Response' and '50:50'.

The two curves show that my hit rate is not very high in the A-B response base rate. In comparison, the false report rate is higher. In this ROC table, we can find that the overall accuracy is very close. What surprised me was that the 50:50 and 1:3 changes in the two reports did not impact my hit rate but made a big difference in my false alarm report rate. I think this may be the choice effect mentioned in class. The more choices I have, the more likely I am to make a false alarm report.

From the perspective of sensitive value analysis, the different buttons in the confidence rating and the number of AB stars increase the sensitive value. We can see that the experiment's sensitivity is almost three times that of the A-B response base rate. Considering that there is only one participant, we can conduct in-depth research on the experiment again after adding more participants.

For the high hit rate in the confidence rating, I think it may be a practice effect. In the beginning, I may not have a deep memory of the visual impact of signals and AB stars, so the correct answer rate is low. Still, as the experiment progresses, I will gradually deepen my impression by receiving signal and visual stimuli. Lead to an increase in accuracy.

Discussion

I think the whole experiment is meaningful, which makes my understanding of Signal Detection Theory deepened. I believe that more options exist on one page, which is equivalent to giving a lot of signals at the same time. Such a setting is not very beneficial to the flow of user interaction. The preconceived impressions derived from this experiment will produce false perceptions of signal reception. To avoid such internal interference, I think that the overall visual design should be clean and smooth so that users can understand our website process at a glance and easily find their goals. Because of the small number of participants, the data in my experiment may not be enough to support this conclusion.

I hope that there will be opportunities to find more participants to explore the factors that affect sensitivity thoroughly in the future.