Predicting a Player's Performance #2
Remember ages ago when I wrote about predicting a player's performance from one season to the next? Well, I had another look at it.
After mentioning a few improvements at the end of the previous piece, I also remembered ChatGPT exists. I have nothing against linear regression, but with the wealth of models and packages available, there had to be something else to try that might feel more satisfying. Since ChatGPT feels like a better solution than searching, being able to ask for advice on a specific problem and receive ‘personalised feedback, I asked what other models might be worth trying. I decided to try Gradient Boosting Regression (considering one package/library/whatever is called xG Boost, it’s hard to turn it down for a football model), and the results felt interesting enough to write about again.
I also cleaned up a bit of the process. I fixed the problem of players playing for two clubs in one season, so it now follows the sequential order and doesn’t opt for the entry with the most minutes. Then, I introduced a goals threshold. It did reduce the dataset to a tiny number (from ~55k to ~2500), but the hope is it’s more informative for predicting goals. Before, I worried the model had too many non-informative data points that added noise to the model and made it easier to perform well (predicting lots of players who never score to never score = good model).
I’m still not sure how best to perform a widescale test of the model. My initial feeling depended on how well it predicted Erling Haaland to perform at Manchester City. But, across the whole dataset, the RMSE of predicted non-penalty goals per 90 compared to actual non-penalty goals per 90 was lower than the RMSE of actual non-penalty goals per 90 compared to the player’s previous season’s performance. It makes me believe there might be some value in such a model, but it might be possible to achieve a better performance just by predicting outliers won’t be outliers again, so who knows?
Looking back at the previous piece, while I included some tables showing a range of results, the problem with cherry-picking interesting things is it’s easy to find it interesting when the model is right. So, I thought I’d dig into some of the biggest misses of the model.
Who did the model miss?
When reviewing the model, my first step was to see who it predicted to be the most productive in front of goal. But this follows a similar problem to the players who never scored. A model predicting players who score a lot to score a lot doesn’t seem that hard. It’s not a bad first port of call and can offer encouragement, but recognising the likes of Lionel Messi, Cristiano Ronaldo, Kylian Mbappé and Robert Lewandowski will score regularly doesn’t exactly require a boffin to be hunched over a laptop all day.
What about the players who scored a lot, but the model didn’t predict they would? Can the model recognise those who are on the cusp of breaking out? Or, if a player performs way above expectation, should we believe the player has broken free from the constraints of predictability or that they’ve had an anomalous season they may not repeat?
I wanted to look at players aged 23 or younger who didn’t perform in line with the model, either positively or negatively. First, let’s take a look at those who performed worse than the model predicted, with the biggest difference between predicted and actual goals (not relative difference).
Some don’t seem too bad. Alexander Isak’s 2021/22 season seems to be a blip compared to his other seasons, and had he performed in line with his xG, the model wouldn’t look as bad. You can throw Ferrán Torres, Jean-Philippe Mateta and Bryan Mbeumo in a similar bracket, where they underperformed their xG, so you could argue the model predicted how many you’d expect them to score, even if they didn’t deliver. (But I’m also clutching at straws to defend my model here, I won’t lie)
Ola Brynhildsen is interesting. In 2019 he had 0.23 npG per 90 with 8th placed Stabæk. In 2020, his first season with Molde, they finished 2nd and he scored 0.24 npG per 90. The model must have thought he had a strong performance for an average side. Now, at a stronger side, considering his young age, he should kick on and start to explode. But he didn’t. Yeah, he underperformed his xG, but the model still had higher hopes for him. Then, in 2022 he scored 0.89 npG per 90, albeit from a smaller number of minutes, and currently has 0.47 npG per 90 in 9 Eliteserien matches. The model may not have been right, but picking him out as a young player who might be on the cusp of exploding doesn’t seem that wide of the mark.
Across the two seasons before his move, Jadon Sancho averaged 0.50 npG per 90, so it thought he should maintain a similar level for Manchester United. This hasn’t happened, but Sancho was highly rated and had good numbers at Dortmund, so it’s unlikely a model like this would predict anything else.
Ruben Vargas is in a similar bracket to Jadon Sancho, without the big-money move. The model predicted he’d continue at a similar rate, and considering his age and the fact he didn’t move, there doesn’t seem to be anything in the model to suggest he wouldn’t.
Rony Lopes had a big season in the year prior, and the model predicted it to continue. Monaco fared much worse as a team and had a heavy turnover in the previous couple of seasons, but Lopes also hasn’t seemed able to recapture much of his form since.
Then, I know nothing about Filip Ugrinic or Carlo Holse, but looking at their previous scoring record, I’m curious what the model was smoking to be so big on them scoring. My guess with Ugrinic is he had a respectable scoring record with Luzern, so moving to a stronger side in the same league would see those numbers increase. It’s a situation where considering position might help. He’s listed as a midfielder, and the model could fare better if it saw how infrequently midfielders scored 0.45 npG per 90.
And the <= 23-year-old players who performed much better than the model predicted:
Starting with Patrik Schick, looking at his npG per 90 on FBref, he seems a hard player to predict. I don’t really know what to make of this prediction, to be honest. (I’ve also never really known what to make of Schick.)
Erling Haaland’s prediction seems odd. The model predicted 0.68 npG per 90 for his first half-season at Dortmund, and he scored 1.10 npG per 90. To downgrade him after that information seems strange unless the model thinks it’s not often players have those kind of numbers, so faced with a decision of whether he’s part of the best in the world bracket or just had a good run, it seems to have leaned towards the latter. It seems to have learnt from this experience, as it predicted Haaland to score 0.84 npG per 90 for Manchester City.
It’s interesting the model thought Timo Werner would score less. He was 23 at the start of the season, and his shot numbers increased from one season to the next in the two seasons used to make the prediction. However, the model doesn’t know the relevance/importance of the order of the data, which perhaps opens the door to a time-weighted or exponential average approach.
Terem Moffi is a fun example, given the xG figures. Moffi has performed mostly in line with his xG figures during his time in France but from season to season (with the last season split in two after his January move), he’s oscillated between underperformance and overperformance. He’s scored 0.56 → 0.20 → 0.71 → 0.41 npG per 90, from xGs of 0.41 → 0.31 → 0.37 → 0.56 per 90. It’s another situation where the model predicting his xG makes it feel like a win, even if it isn’t.
Moving to Loïs Openda, even though it’s wrong, I don’t hate this prediction. The model predicted a decrease on his previous season’s performance in the Eredivisie. It seems understandable, especially considering few would have predicted Lens to be so strong this season. On FiveThirtyEight, they had a 13% chance of making the Champions League and a 9% chance of relegation. It’s a situation where it would have been nice to pick up on something early, but it’s not a surprise the model didn’t.
The model predicted Adam Armstrong to do well in every season except the season he did well. But, before this season, he scored 0.16 and 0.36 npG per 90, from ~2.5 shots per 90. In 2020/21, he had 4.71 shots per 90. There’s likely nothing in the model that would predict this kind of change from one season to the next.
Andreas Schjelderup is in a similar bracket to Armstrong. It seems something changed that the model wouldn’t have detected based on the data it uses. Using FotMob data for xG, it went from 0.26 per 90 to 0.40 per 90. His shots only increased from 1.72 to 1.95 per 90, but his shots on target went from 0.66 to 1.23 per 90, with his goals per shot going from 0.08 to 0.26 (not excluding penalties). It seems he ran hot, but something also changed to get him better chances.
Premier League Imports #2
Writing about Brighton’s transfers made me want to make a model like this, so seeing how players perform at a new Premier League club was one of the main things I wanted to do. Below are tables for the top twenty players for predicted npG per 90 each season from 2019/20 to 2022/22. I’m not going to talk through them, I’m just going to chuck up the tables. The RMSE of predicted goals vs actual goals for players who joined a Premier League club was 0.129, compared to 0.241 for the RMSE of the previous season's goals vs actual goals.
2022/23 Season
2021/22 Season
2020/21 Season
2019/20 Season
Conclusion
To call the model simplistic would be an understatement, but it still feels like it contains value. While I wouldn't follow it, it’s interesting enough to serve as a data point worth considering.
Looking at someone like Kaoru Mitoma (who is the main reason I made this model), the model predicted he’d be more productive in the Premier League than Leandro Trossard, based on Mitoma’s season with Union SG and his final J League season. Trossard had a more productive half-season but outperformed his xG. There’s an argument Brighton were selling while Trossard’s stock was high (with the overperformance in front of goal, albeit with a dip in value due to his contract) and also knowing Mitoma could step up.
Building a model only using one previous entry, as there’s missing data for two entries, the model predicted Mitoma would score 0.37 npG per 90 in the Belgian League and 0.26 npG per 90 in the Premier League. In reality, he scored 0.38 npG per 90 and 0.27 npG per 90, respectively. How much confidence do you need in the model to believe that investing €3m into a player who could score 0.37 npG per 90 in the Belgian League is worthwhile?
Putting figures in to see how the model predicted Mitoma’s performance from his two J Leagues to Brighton, it predicted a huge 0.40 npG per 90. That seems quite high, but it seems Brighton’s approach of taking the €3m gamble, sending him on loan to Union SG and getting another data point/benchmark for his performance was likely better than throwing him straight into the Premier League.
So, yeah. The model is fun but has a very limited scope and requires lots of consideration. Some brief thoughts about the limitations and future ideas are below.
Some limitations of the model that likely won’t change:
Lack of underlying data. Since the model uses such a wide range of leagues, I made the choice for quantity over quality. Having xG figures would likely benefit the model. After seeing FotMob has xG figures for more leagues, it may be worth seeing how scrapable it is and merging it with the current data. I could also run a test with a smaller model since xG is in this dataset. Maybe another time.
No tactical information. Not surprising, but since the model only uses goals, shots and SPI information, there’s nothing about how a team creates or the area a player operates in. A good example of this limitation is Antoine Griezmann. The model thought moving to Barcelona would see him increase his output significantly, likely because of Barcelona’s more active attack, but with the context of Barcelona’s squad, most people questioned where he was going to operate, given him and Lionel Messi tend to operate in similar areas, perhaps meaning they could get in each other’s way.
Same league moves seem inflated. Tying in with the above point, if a player moves from a smaller team to a bigger team in the same league, the model seems to think they’ll produce more. The logic is understandable but likely not helpful. Knowing it does this is probably helpful at least.
Some possible ideas for changes to the model:
Consider position. I said it last time, I’ll say it again. It’ll be easy to implement, I’ve just been lazy. Filip Ugrinic’s figures were a reminder of why it might be helpful. There may be a chance that it notices midfielders and defenders score less often and will be more cautious about dishing out high predictions.
Time-weighted data. The model has no idea about the time of the data. While it might make connections, perhaps if features have relationships with each other, it’d be interesting to experiment and see how the model changes should the most recent season be weighted so it’s considered more important than the older entry.
Re-evaluate the presentation. It sounds stupid, but when looking at figures per 90, the differences almost always seem small. Considering a difference of 0.05 per 90 will only account for ~2 goals if a player plays every minute, that isn’t a bad rule of thumb to use for closeness. I could look at percentage difference, or a ratio, but that can feel too extreme at times. It’s just when you see two zero point something numbers, it’s easy for them to feel close because they’re tiny numbers in general. If a player scored 16 goals where you predicted they’d score 12, that’d feel like a bigger difference than ~0.1 per 90 (for a player who played every minute of a Premier League season).