Firstly, I just want to say thank you for this detailed and easy-to-understand response! I feel like I have a much better idea of where you are coming from regarding these issues
I will try and be as well structered as you are in this response haha
You are exactly right! If we were to build such a line of blocks above the surface, then spawning mechanics under that line would be unaffected, and walking mobs would be able to spawn like normal. I absolutely see what you mean; this would break my own logic. Of course, we are not trying to make a real-life simulator here; we are just balancing the realism with the fantastical elements of Terraria. The 'logic' behind allowing mobs to spawn in such a line is that with a dirt/natural wall placed there, it should somehow be possible for there to be a path that could lead down to the ground in the background; we are imagining that Terraria is 3D for a moment here. Although this wouldn't make much sense for a single dirt wall placed 100 meters in the sky, I think it would still satisfy Terraria players as it would allow for farms and better control over this feature.
There are, of course, many logical inconsistencies here, as you are well aware, allowing for the possibility of walking mobs to spawn anywhere under this line wouldn't make much sense. There wouldn't always be a way to jump from the line down to any point under it, for example. Wasn't implementing logic here the goal in the first place? Yes, kinda. I think there would be more logical consistency in normal playthroughs with this feature, but I also believe that beyond the logical aspect, it can be beneficial for other aspects of gameplay. For example, being able to use our home base as a more effective arena or defensive structures for events.
I think it's important to compare these two scenarios as I believe they are vastly different in concern. What kind of farms over the surface are we talking about here? Could there not be any benefit to having the option of only having flying enemies spawn in such a farm? You might have wondered why I suddenly changed my criteria from natural wall to dirt wall in my response here, and it's simply because you have brought up a very good point that was completely missed by us. There should have been more thought given to specifically what kind of walls can be used here, but this has not been listed yet. For the sake of this response, I'll say that dirt/stone walls would fit the criteria. Back to the farm, if we assume that we only need to place a dirt wall somewhere close to the top of the farm to allow every type of mob to spawn, then this isn't nearly as much effort as the number of walls that would have to be placed to prevent walking enemies from spawning on our roofs or walls. Furthermore, you could place this dirt wall inside blocks, which wouldn't change the visual of the farm at all, while this would necessarily force visual changes to our builds.
There is also an important thing to note here, which is that player-placed walls block the spawning of essentially every mob. Even if we were to go out of our way to place walls everywhere to try and simulate this feature, I don't think that would be immersively satisfying. There wouldn't be these locations where only flying/high jumping mobs pop up, and these clear distinctions to where walking mobs can reach.
I think you are reducing the scope of what this feature would entail if you think that it would mainly affect zombies in towns. Although this is a small problem in our eyes, the bigger problem is posed by the other mobs in the list that would be affected, as well as the buildings and structures that are non-towns. But addressing a problem is only half of the story here; we also believe that there are many other aspects of gameplay that this would enhance, alongside solving these issues.
When looking through how the current spawning works on the wiki, it's clear that the spawning mechanic already goes through many steps, including the step of checking if there is ground below the chosen spawning block. With this already being a working feature, I see no reason to believe that having the same feature with these checkers will be a source of lag. With the simplest implementation that I brought up for these checkers, they wouldn't have to calculate anything other than the jump height versus the height to the surface line. Spawning mobs is also something that happens outside of the screen, so here we can utilize multiple threads in the code to make the game run smoothly while the spawning calculations are happening.
But you are absolutely right that we don't know how the code works and what would cause lag or what wouldn't. I try to emphasize how important this is in the post as well, and thus, saying that this would be too much work or it wouldn't work, I don't think is a valid concern. I also believe there is reason to believe that this could work, as I don't see anything radically new that has to be implemented here that hasn't already been proven to work in Terraria.
This would have to be fact-checked; on the wiki, it says that there is a small risk of enemies spawning in towns in expert mode. I can't recall to 100% if we had this problem in our town when there was no event or some other factor making spawning possible.
I do agree that it makes sense that there are enemies around in the wilderness; however, it doesn't make sense that we have to constantly consider the possibility of walking mobs on the roofs. If we have a protective wall around, we wouldn't expect these walking mobs to get inside our wall perimeter.
Not much more to say here for now. I didn't know that Pirates did that! I do agree that you shouldn't be safe from events.
I think I've already elaborated on this in this response. In short, I believe this feature would be a much more satisfying solution because there are many problems with placing walls, both mechanically for the purpose of blocking only walking mobs and practically for players having to place all of these walls. I will also add that, for some roofs, having walls on top wouldn't be the preferred choice visually. However, this would as you say be a matter of opinion. Echo walls wouldn't solve these problems entirely, as player-placed walls block almost all mob spawns.
Thank you for reading through all this; I do feel like we are making progress here!