Tell. How to get to the question near the village of Svorki, judging by the outline, it is somewhere below. (The Witcher 3: Wild Hunt)

Let me guess: you did the quest with Leto? Then this is a bug, you have already cleared this mini-settlement from bandits, but then you went there on the Summer quest - and the question mark returned to its place.
There is only one treatment - load an early save and first go through the Summer quest and only then go to this question mark.

After the quest I went to the question

I was standing in the place where the question is, there is a refugee settlement and the arrow under the question on the minimap points down, which means underground

I have the same, apparently a bug