Valuing actions by estimating probabilities (VAEP)

Project description

PSV asked us to look into an ML model that is able to evaluate actions on the field. Based on this evaluation then teams' and players' performance could be evaluated. Therefore, it could be used as a scouting tool or to form a strategy against the upcoming opponent.

Context

This is a project that consists of a collaboration between the football club PSV and students from the Applied Data Science minor at Fontys. The client is Ruud van Elk who is Head of Sport Science & Analytics. PSV already applies data science in multiple different aspects of their business. One of the ways that PSV uses data science is to evaluate played games. This can be games played by PSV or games that are played by the different clubs and maybe even different leagues. PSV uses the data of their own games to evaluate how well individual players and the team are playing. PSV uses the data from other teams for scouting purposes.

Right now, PSV uses xG (expected goals) for these analyses. PSV gave as an assignment to research the xT (expected threat) model. The difference between xG and xT is that xG only looks at the chance that the result of an action will result in a goal, and xT is able to look more layers deep. For example: if someone passes the ball to the location where players often create assists then xT will reward that player for that, xG will not because players do not often score from that position. The goal of the improved model should be that valuable passes and dribbles should also get recognition instead of just the goals.

While researching xT the team came across another soccer action evaluation model called VAEP (Valuing Actions by Estimating Probabilities). The team took the initiative to read some literature on both models and presented the findings to the client. After a discussion, the assignment was changed to research on VAEP since it takes more features into account and looked like it had more potential.

Results

During the project, a lot of results were yielded due to the different changes of the model.

Firstly, the team spent time analyzing why the base model was not working. Then, after figuring it out and making the necessary changes, the first results were received. The output was a dataframe with the ratings and there were three categories - offensive, defensive, and VAEP value. The team had to draw and explain the insights to the client. They were presented in the form of a dataframe.

Then, it turned out more of the source code needed changing so there were new results. The new results were drastically different. They were different in a way that the first batch of results resembled the real-life results better, but the summaries about the best/worst actions made more sense in the second batch of results. Using the first model AJAX is rated as the best team, and PSV is 3rd or 4th, whereas according to the second model they are almost at the bottom. However, for the best/worst actions the first model rewarded with the highest defending value to an action that was an own goal, which makes no sense.

With the fully changed source code, this issue was resolved, and the highest offensive values were not goal attempts anymore, but there were fouls and passes. After taking a different approach and changing some of the action names to match the list in the spadl.config file, the obtained results were similar to the previous ones but probably slightly better.

In both scenarios there is a big negative number for the defensive value, however, at least in the latter case, the VAEP value is not a negative number anymore. Therefore, this might be a step in the right direction. It is interesting why all of this is happening, but since the team does not have any more time to figure it out, this task is for the team of professionals from PSV and ASML.

Methodology

The team used the recommended by the Minor CRISP-DM methodology. The first step was business understanding. During this phase, the team took the initiative and started researching the requested by the client model. However, a better and more interesting model was found. After performing some literature study a presentation comparing the two models was presented to the client. The client's view aligned with ours so we changed the objective of the project to the newly found VAEP model.

The next step was data understanding. In this phase, we performed EDA on the data provided by PSV. The goal was to make sure that the data is evenly distributed, there is no corrupted data, a few visualizations were created, that surprised even the client. After the team was confident about its knowledge of the data we proceeded to the next step.

Data preprocessing is the phase of the project that took the most amount of time. This is due to the reason that the model requires a specific data format called SPADL. For this format most of the attributes were missing from our data, so we had to find a way to calculate them from what we have. Around the middle of the project, this phase became one with the next one, which is Modeling, and this was until the end.

The last phase was Modeling and Evaluating. For the modeling part, we again took the initiative and tried to make the open-source code from the library custom to the PSV data. However, after evaluating the results, in the end, this might not have been the best approach. But we believe this is still a valuable insight since this project was more on the research side. Evaluating this model is tricky since we used our client to express his opinion if the results make sense to him, but at the same time, we didn't want to get carried away with making the model produce 1:1 results to the real-life ranking list. The reason for that is the whole point of using ML in our daily lives is to get an alternative, unbiased opinion.

About the project group

All four of us did the ADS specializations, so we had some data science background. Moreover, three of us (Sander, Jochem, Daniel) did an internship in the field. We managed to work well as a team, which led to achieving satisfactory results.

Downloads

• psv_project.pdf