First, collision. High energy neutrons, the "bowling balls of physics" have interesting effects. As I understand it, the collisions offset atoms in various substances. It is the reason fuel rods need to be replaced. Because of displacement the safety shield around the nuclear core falls apart. It is the reason nuclear power plants have a 40 year lifetime. Displacement, the Wigner effect ( https://en.wikipedia.org/wiki/Wigner_effect ) was the cause of graphite fires in England due to the rapid release of energy.

Collisions will likely cause the walls of the fusion system to fail. Silicon electronics fails in high flux situations requiring special techniques, like silicon on sapphire (https://en.wikipedia.org/wiki/Silicon_on_sapphire).

Absorption will also play a role. The blanket around the fusion core is a neutron multiplier, likely made of beryllium (https://www.osti.gov/biblio/7154770) or lithium. I suspect lithium will win out as it will become a source for lithium-7 using to make hydrogen bomb cores. Either way I suspect this will be a highly radioactive by-product.

Fusion requires a source of tritium, a by-product of nuclear fission. Japan is currently releasing huge stores of tritium (https://www.science.org/content/article/japan-plans-release-fukushima-s-contaminated-water-ocean). However, fusion will compete with nuclear bomb technology because tritium enhances bomb effects, allowing "dial-a-bang" variable sized bomb yield. Because tritium has a short half-life (12ish years) it needs to be made fresh.