The moves by the Commerce Department against the Europe-based firms are the latest effort to rein in a spyware industry that has spiraled out of control in recent years.