Jisho

I think you'll need to use a different word for the "the physical place you go to work, as if you were to label it on a map"

How about these words? I think it's the most commonly used among natives, and works perfectly for the situation.
会社[かいしゃ] - company/workplace
or,
事務所[じむしょ] - office/workplace

If I were to label places I often am on a map I would use stuff like 自宅, 職場, 学校, ジム.
I think 職場 should be a good word to use, I see it used in phrases like 職場の人間関係 or 職場のストレス. It is also alright to use 会社, but it is used more for stuff like 会社を辞める or 会社に行く, where it is more about the concept. 事務所 strikes me as oddly specific, maybe if you're talking about renting office space or the like it's appropriate.

Thank you for all the options!