cut-e is proud to contribute to the scientific world of psychology at work

Symposium: State of the Art of Game-Based Assessment

General Abstract
Game-Based Assessment (GBA) is becoming increasingly popular in workplace assessment, especially, but not exclusively, in the context of large international companies' graduate and apprenticeship schemes, yet little is known about this fascinating Psychological Assessment approach. This six-paper symposium offers a comprehensive insight on the state of the art in GBA by merging the perspectives of Arctic Shores - UK's GBA pioneer - its recently announced partner cut-e, and Birkbeck College. Hands-on games, scientific evidence, and a snapshot of the "behind the scenes" work that enables GBAs will be combined to answer questions such as "What is GBA? Does it work? Is it fair? Is it science?”, and to stimulate discussion between researcher, practitioners, and the wider audience. 

Paper 1: The elephant in the room: What is Game-Based Assessment? (Lara Montefiori, Arctic Shores)

Paper 2: Fairness in Game-Based Assessment. Adverse impact, perceived stress, and effect of screen size. (Dr Maria Panagiotidi, Arctic Shores)

Paper 3: Gaming the game. Game-Based Assessment and self-presentation bias. (Dr Maria Panagiotidi, Arctic Shores)

Paper 4: Measurement invariance, and dyslexia as a moderator in game-based assessments. (Liam Close, Birkbeck College)

Paper 5: Incremental Validity of Online Psychometric Assessments and GBAs: Some Preliminary Findings. (Dr Katharina Lochner, cut-e)

Paper 6: Arctic Shores’ Psychometric Framework and Infrastructure. (Lara Montefiori, Arctic Shores)

ITC 2016 - Poster session: Pilot Assessment - Above and Beyond Ability

This poster was presented by Dr Achim Preuss and Dr Katharina Lochner at the 10th International Test Commission Conference, July 2016 in Vancouver, Canada

Introduction
When assessing airline pilots there is often a focus on mental ability. However, research shows that in most accidents inadequate team behaviour or a breakdown in communication was observed (Maschke, 2004). Moreover, particularly captains need to show professional aviation knowledge and should be strong in the social competencies needed to practice good Threat and Error Management (TEM) and Crew Resource Management (CRM) (IATA, 2012). Thus, assessing competencies along with cognitive ability in pilot selection is essential.

Objectives
We developed a competency model for airline pilots that comprises the competencies essential for TEM and CRM and validated it with performance data from various airlines. The competencies can be assessed using a self-report questionnaire.

Design/Methodology
First a conceptual model was developed based on the insights from literature (e.g., Goeters, 2004; Le, Oh, Robbins, Ilies, Holland, & Westrick, 2011) and practice (e.g., IATA, 2004). Second the model was validated using data from major airlines from the Middle East and Southeast Asia. As predictor we used the personality questionnaire shapes that assesses job-related competencies. As criteria we used performance on the simulator and during flight (base and line checks).

Results
The competency model comprises competencies like stress resistance, interaction with others, or decision making. The empirical studies revealed dimensions like controlled (subscale of stress resistance), agreeable (subscale of interaction with others), and analytical (subscale of decision making) – to name but a few – to be predictors of performance both on the simulator and during flight. The competencies showed incremental validity above cognitive ability tests.

Conclusions
Competencies predict pilot performance and have incremental validity above and beyond cognitive ability. Thus competencies can and should be assessed during pilot selection. The model will need further refinement for different target groups such as captains, first officers, and captains.

SIOP 2016 - Poster: Detecting who is going to cause problems

This poster was presented by Dr Achim Preuss and Dr Katharina Lochner at the 31st Annual Conference of the Society for Industrial and Organizational Psychology, April 2016 in Anaheim, Florida, US

Abstract
An online instrument predicting counterproductive work behaviors is introduced, based on the notion that behavior is best predicted by considering the person, situation, and interaction between both (Mischel & Shoda, 1995). It shows good psychometric properties (Cronbach’s alpha between .71 and .90, correlations with interview results between .39 and .77).

Press paragraph
When screening job applicants companies usually assess factors that predict job success. However, counterproductive work behaviors such as absence from work, fraud, or dangerous behavior cause a lot of damage and therefore, factors that predict such behaviors should be assessed in employment testing as well. A new online instrument based on the notion that behavior is not only based on a person, but also on the situation is introduced. The items describe behaviors rather than traits, which allows for predicting counterproductive behaviors without stigmatizing those scoring low on the questionnaire.

ATP 2016: Panel discussion - The transition from computer to mobile and tablet based testing: Misinterpretations, implications and pitfalls 

Abstract
The increasing use of smartphones and tablets seems to urge test providers to make their tests accessible via these devices. The question doesn't seem to be whether or not we are going to use these mobile devices for testing. There seems to exist, however, some controversy on the purposes and the conditions of meaningful use of these devices.

In this session, the panel will address some questions on testing via mobile devices.

• Is there a difference in data quality gathered by computers and mobile devices and if so, what are the consequences? If this is true, can we base selection decisions on data gathered via mobile devices? Should we complement them? Should we not use them for selection and rather for other purposes such as realistic job previews?
• Based on what criteria should we decide whether or not to include mobile devices in the selection process? A first criterion might be the constructs we want to measure. Computers might be more appropriate for some constructs, mobile devices for others. A second criterion might be practicability. When we are confronted with a significant number of candidates who have a tablet or a smartphone but no computer, it might be a good idea to provide a test on the mobile device. A third criterion might be the stage of the selection process. Maybe we should use mobile devices solely in the beginning of the process?
• Mobile devices enable us to measure other psychological constructs and skills than regular computers. Or not? What is the difference between mobile devices and computers in relation to the constructs we to measure? For instance, is the fact that it is easier to confront the candidates with more text relevant for the measurement of constructs? Or is this only a minor difference?
• Is the impact of the shift towards mobile devices as fundamental as the shift from paper and pencil testing to computer testing? The shift towards computers has confronted the testing industry with important challenges. The tests had to be adapted to be used on computers, for example. Similarly, we can ask ourselves how our tools have to be adapted to mobile devices. Test taking on the computer has also brought us new, interesting possibilities. Adaptive testing and the use of videos, for instance, were not possible when we were using paper and pencil tests. What opportunities does the shift towards mobile devices offer us?

3 session objectives

• To raise awareness of a significant change that is currently happening in the testing industry.
• To start a discussion on suitable and unsuitable application of mobile devices in testing.
• To give some ideas as to how to deal with this change.

How does this session address innovations in testing or examine this topic in an innovative manner? It deals with a significant innovation that is currently happening in the testing industry that we need to address proactively.

On the panel: David Barrett, COO cut-e Group

EAWOP 2015 - Poster: Concurrent construction designs may not overestimate the predictive validity of situational judgment tests (SJTs)

This poster was presented by Mats Englund from cut-e Nordic at the 17th ongress of the European Association of Work and Organizational Psychology (EAWOP), May 2015 in Oslo, Norway

Screening candidates with situational judgment tests (SJTs) is becoming increasingly popular among companies worldwide. Attractive aspects of using SJTs include effective corporate branding and realistic job previews for candidates while maintaining high predictive validity. A common method to establish which items to use involves a validation study and picking the most predictive items from an item pool. Using this method, capitalization on chance may lead to overestimation of the predictive validity, potentially making concurrent designs inaccurate for estimating true predictive validity.

However, in practical cases, data is often collected using classic concurrent validation study designs, where participants already may have learned on the job what behaviors are desirable. This leads to the risk of restriction of range, attributable to inside knowledge rather than true predictive ability/potential, which leads to underestimation of the predictive validity, opposing the overestimation mentioned above. Here, results are presented from the development and validation of an SJT for service positions in a large Swedish organization. Comparisons of results from the construction of the SJT (n=800, concurrent design) with results from high-stakes testing (n=550, predictive design), and a supplementary student-sample study (n=128), suggest that both capitalization-on-chance overestimation and incumbent-knowledge-related restriction-of-range underestimation of the predictive validity were present.

Mental ability scores predicted SJT scores better for students, suggesting SJT restriction of range for employees was attributable to learned desired behavior, and estimated predictive validity proved close to the true predictive validity. These results suggest one reason using a concurrent designs may not always overestimate SJTpredictive validity.

IPPA 2015 - Presentation: Impact of emotions on test performance

We were delighted to be part of the 4th World Congress of the International Positive Psychology Association (IPPA) in Florida, US, June 2015. Dr Katharina Lochner, Research Director for the cut-e Group, shared her research of the impact of emotions on test performance.

The Congress as a whole sought to explore not just how individuals are able to flourish and reach their potential through positive psychology but also how this can, ultimately, lead to a better society – and this area of psychology seems to be going from strength to strength.

Dr Lochner’s research has looked at ability testing which has for many years been used as a staple of recruitment and development assessment programs. Performance on such tests is important when selection or promotion is impacted by the score achieved – and yet, despite their popularity and use in high stakes situations, there has been little research into the impact that emotions have on test performance. Whilst there have been studies which show the positive impact a positive mood has on test performance, Dr Lochner researched the impact of more specific emotions such as joy, contentment, anger and sadness have on test scores.