Is it safe or not? Depends on the country.
If you’re concerned about drinking tap water when traveling in Europe, I’ve added a page to this blog about which countries have reliably safe drinking water and which don’t. Where the water is possibly unsafe, you might want to heed the usual cautions about fresh but unpeeled fruits and vegetables, water for brushing your teeth and even showering. And I’ve cited Condé-Nast Traveler as the source of this list.