Can people see what servers you follow on Discord?

I only really use it when I need to download video game mods and stuff, but just wondering if I should be mindful of that when joining a discord for school or something

Only if they are in the Discord channel as well.

And if that's what you're worried about, don't have your discord account tied to your other social media.

If you're in the same server as someone, you can click their name and see a list of all the servers you share. So only the people who are also in that other server would know you're in there.

You might want to make sure you turn off the feature that autodetects what game you're playing and puts it as your profile status though