Perchlorate (ClO4-) contamination is a widespread concern affecting water utilities. In the present study, functionalized graphene sheets were employed as the scaffold to synthesize a novel graphene-polypyrrole (Ppy) nanocomposite, which served as an excellent electrically switched ion exchanger for perchlorate removal. Scanning electron microscopy and electrochemical measurements showed that the 3D nanostructured graphene-Ppy nanocomposite exhibited a significantly improved uptake capacity for ClO4- compared with Ppy film alone. X-ray photoelectron spectroscopy confirmed the uptake and release process of ClO4- in graphene-Ppy nanocomposite. In addition, the presence of graphene substrate resulted in high stability of graphene-Ppy nanocomposite during potential cycling. The present work provides a promising method for large scale water treatment.