Francisco Victor Newton de Souza was born to Goan Catholic parents in the village of Saligão, Portuguese Goa. After his father and then his elder sister died, he and his mother moved to Bombay, British India, in 1929. Souza's mother remarried, and his half-brother was the painter Lancelot Ribeiro.

Souza attended St. Xavier's College in Bombay, but he was expelled in 1939 for drawing obscene graffiti in the restrooms. He then studied at the Sir J. J. School of Art in Bombay but was also expelled from that school in 1945, because of pulling down the Union Jack flag during a school ceremony and participating in the Quit India Movement. Souza joined the Communist Party of India soon after, and co-founded the Bombay Progressive Artists' Group in 1947.