From what I'm seeing, there are several people that had this issue that solved it by uninstalling the game, then reinstalling it to the internal HDD. It might be something to do with drive sizes, as the game is updated and grows larger the game is being spread all over the drive and as time goes on the drawing the game up might be causing issues when loading between events, basically because of the time needed to gather up all the bits, the game doesn't know where to go from there (it's not the best explanation, sorry).
It's a pain in the ass, will take up some time, it's not a guaranteed fix, and I'm assuming based on size your hard drive is an external HDD and not an SSD when making this recommendation. I would try uninstalling the game fully from the large drive it's currently on (your saves should be a separate file and backed up in the cloud), then make space in the internal drive for the game and do a clean install on the internal drive. I'm not sure if the X has a defrag option you can use but if it does I would recommend running it before the re-install. I think the Xbone might do that automatically in standby mode, but I'm not sure.
I run almost everything off SSDs, and have given some games their own drive on my PC because I was having loading issues off my massive external drive I've monikered "The Porn Dumpster" Right now I have a dedicated Forza drive because Forza 7 would hang sometimes for several minutes on the Porn Dumpster. Moving it to a smaller drive erased every issue I was having with the game so it might have a positive effect on your issue.