The Australian rules I've learned so far...
Must not fly within 3 nautical miles of an airfield.
Must not fly within 30m of a "built up area". This is roughly defined as any building or road.
Must not fly within 30m of another person.
Must fly under 400ft (120m) in controlled airspace. If you're near a city or town, assume you're in controlled airspace.
Must not fly in a park displaying a no model planes sign.
I think there's also a rule saying your plane must be under 1kg AUW to fly in a public place.
All that said, when I first started flying I expected a ranger to jump out from behind a bush as soon as I connected my battery. Since then I've flown in parks with people walking their dogs and things and had many people come and ask me questions about my planes. I've never had anyone act negatively towards my flying, but I've also never hit anyone with my plane either, lol.