Originally the Swordsman was a star performer at various circuses and carnivals, where he did an act demonstrating his extraordinary mastery of knives, swords, and other bladed weapons. At one carnival he met a boy named Clint Barton in whom he saw great talent in archery. The Swordsman trained the boy, who idolized him, to be a secondary performer in his act. (The Swordsman trained Barton in the use of bladed weapons; another performer, Trick Shot, taught Barton archery.) Eventually, the Swordsman came under pressure to repay a large gambling debt. One day the carnival paymaster was robbed, and Barton found the Swordsman in possession of the money. Barton fled in panic onto the high wire, which the Swordsman then slashed by hurling a sword. Barton fell to the ground and the Swordsman left him for dead. But Barton was still alive, and grew up to become the costumed adventurer Hawkeye.

Over the years following his attempted murder of Barton, the Swordsman, in his costumed identity, led a notorious criminal career in Europe, during which he was deported from a dozen nations. Returning to the United States, he offered his services to the Avengers, intending to use his Avengers status to commit even greater crimes. However, the Avengers turned him down. Subsequently he entrapped and unsuccessfully tried to kill Captain America, then the leader of the Avengers. As Captain America and the Avengers surrounded the Swordsman, the Swordsman was teleported away by the Mandarin to the latter's castle in China. The Mandarin intended to use the Swordsman as his double agent inside the Avengers in order to destroy the Mandarin's archenemy, the original Iron Man, from within the organization. To this end, the Mandarin used his technology to convince the Avengers that Iron Man had sent them a message recommending the Swordsman for membership. The Mandarin then teleported the Swordsman back to New York City where the Avengers, suspecting a trap but deciding to play along for now, accepted him as a member. The Swordsman came to admire the Avengers and also to admire the beauty of the Scarlet Witch, one of their members. One night the Mandarin ordered the Swordsman to set off a bomb, which the Swordsman had hidden in one of the Avengers' control panels. The Mandarin wanted the bomb to kill the Avengers so that the absent Iron Man would arrive to investigate and fall into a trap that the Mandarin would set. The Swordsman, however, did not want the Scarlet Witch harmed and did not believe in killing the Avengers without a fair fight. Therefore, he went to disconnect the bomb, but Captain America and Hawkeye saw him and thought he was planting the bomb instead. A fight ensued between the Swordsman and the Avengers, ending with the Swordsman's escape, but not without regrets on his part that his career with the Avengers had come to an end.