Server 2019 unable to route to specific hosts (by IP or hostname.)

Jump to latest Follow Reply
Nevarre

Nevarre

Ars Legatus Legionis
24,110
  • #1
    OK this is an odd one and I'm stumped. TL;DR version is that I'm needing to integrate into a large organizations's networks, and this scenario spans multiple subnets.

    My server 2019 system is virtual on a 10.x/24 network. I'm trying to enumerate printers and one of the IP's held by a printer is never reachable via ping from the command line, can't be found as a printer, the web interface is not found etc. despite the fact that the printer is resolving in DNS. Same symptom trying to reach it by IP instead of hostname. That printer isn't running any firewall, and is visible to any system on the same subnet as well as systems on any other subnet. (The VM is the only system I have on that subnet.)

    Here's the puzzling thing:

    Installing Zenmap on that system, I can sometimes enumerate the system via ping or other methods within nmap, but not reliably. It's sometimes there, sometimes not.

    Other IPs in the same subnet generally work-- some are always reachable, but I'm seeing other odd network behaviors as well-- e.g. sometimes the web interfaces for the printer that do work will time out and come back. Connections to the Internet seem to be unaffected.

    The printer itself is not running any kind of IP filtering or firewall.

    TL;DR: specific host I know 100% is up and pingable is not pingable or reachable in any way from a specific Server 2019 system except sometimes via Zenmap/nmap while other systems on that subnet are reachable. Host (printer) is not firewalled.

    Has anyone seen anything like this before?
     
    O

    Oddabe19

    Ars Praefectus
    4,861
  • #2
    With just basic information, it sounds like an IP conflict to me (sometimes resolving). However, you said everything else can reach it 100% of the time EXCEPT for the VM, correct?

    Is Windows firewall breaking it?

    If other systems can't get to it reliably, what happens if you plug a laptop directly into the printer via ethernet and give the laptop a manual IP on the same subnet as the printer? Does it work then? If so, it probably wouldn't hurt to upgrade the firmware on the printer while in there. Printers are one of the most finicky things ever and sometimes a firmware upgrade can make it go from 10% working to 99.9% working.
     
    Whittey

    Whittey

    Ars Tribunus Militum
    1,849
  • #4
    Does command line ping work for the other devices on the destination network, or is it zenmap's (nmap) sweep that's seeing the other devices? Windows ping uses ICMP echo requests, while nmap can do much more to find machines. If ICMP is allowed to traverse your networks (you can CLI ping other devices on the destination network), tracert the printer and see where the traffic dies (use -d so you don't try to lookup every hop in dns). If a tracert never works on that printer, compare it to another device on that same destination network and talk to the other org's network folk to see if it's the expected output.
     
    Nevarre

    Nevarre

    Ars Legatus Legionis
    24,110
  • #5
    Oddabe19 said:
    With just basic information, it sounds like an IP conflict to me (sometimes resolving). However, you said everything else can reach it 100% of the time EXCEPT for the VM, correct?
    Click to expand...

    Correct. The IP is statically assigned, and no conflicts can be found anywhere. All other hosts can route to this printer and have been able to for a long time (including system on other subnets.)

    Oddabe19 said:
    Is Windows firewall breaking it?
    Click to expand...

    Same behavior without firewall enabled.

    Oddabe19 said:
    If other systems can't get to it reliably, what happens if you plug a laptop directly into the printer via ethernet and give the laptop a manual IP on the same subnet as the printer? Does it work then?
    Click to expand...

    Can't. I can only assign IPs within the subnets I manage within my buildings (which are entirely different ranges) and the IP range that the VM is on is managed by another organization.

    Oddabe19 said:
    If so, it probably wouldn't hurt to upgrade the firmware on the printer while in there. Printers are one of the most finicky things ever and sometimes a firmware upgrade can make it go from 10% working to 99.9% working.
    Click to expand...

    It's on the latest (final) firmware.

    I should note that I've stood up a 2019 server on the same subnet as the printer for testing temporarily and it works.
     
    Nevarre

    Nevarre

    Ars Legatus Legionis
    24,110
  • #6
    Paladin said:
    Sounds like a complex network to start with since you mention routing and a larger organization. The 2019 is not on the same subnet as the printer in question? So what is in between? What is the network layout? Firewalls in between? VPN? Routers? Switches? Wifi? Goat with a pen and scratch paper? ;) :D
    Click to expand...

    There might be IP over carrier pigeon involved.

    Here's the very short version:

    I have control over my Ethernet networks to a degree, although central networking for the organization mandates the hardware we use and has full visibility into our networks down to the switches. I mostly just get to decide how those IPs get assigned and the printer is just one statically assigned IP on the wired Ethernet in the primary building. The building has a nominal router/firewall that's basically permissive to every other network within the larger organization-- so the firewall appears to be a non-issue in this case. After that things get murky. The building backhauls over fiber to a central point in our remote multi-building campus, then that backhauls over some unknown means to the central organization ~10 miles away. At that point out of my hands and not in my 'need to know'...

    We're being mandated to host this server virtually with the central IT department. That was, at one point, run on physical hardware at the main locations's main datacenter. Some of those have been migrated to 'the cloud.' I don't know where my VM actually resides. All IPs presented are within our internal IP space.

    It probably goes without saying but yes-- I've been flinging tickets over the fence in the general direction of the central networking folks and the VM cluster folks, but I'm new here and I'm not getting a response back. IME, in this large org once I know someone at a department, asking them gets around the ticket routing which is handled by monkeys banging away at keyboards who very seldom route tickets correctly. I just don't know any contacts to ask.
     
    Nevarre

    Nevarre

    Ars Legatus Legionis
    24,110
  • #7
    Whittey said:
    Does command line ping work for the other devices on the destination network, or is it zenmap's (nmap) sweep that's seeing the other devices?
    Click to expand...

    Yes, with caveats. I'm finding it to be less than 100% reliable with some hosts that only work sometimes but I'm picking on the one IP that never, ever works (except that nmap can see it).

    Generally speaking ICMP ping for other devices works OK. To try and keep things on a level playing field I'm specifying nmap -sn (network range) so that should use ICMP only and suppress all port scans.


    Whittey said:
    Windows ping uses ICMP echo requests, while nmap can do much more to find machines. If ICMP is allowed to traverse your networks (you can CLI ping other devices on the destination network), tracert the printer and see where the traffic dies (use -d so you don't try to lookup every hop in dns). If a tracert never works on that printer, compare it to another device on that same destination network and talk to the other org's network folk to see if it's the expected output.
    Click to expand...

    First to another identical printer configured identically on the same subnet/same VLAN. Sorry for obfuscating, but ya know...

    C: \windows\system32>tracert -d A.B.200.185

    Tracing route to A.B.200.185 over a maximum of 30 hops

    1 * * * Request timed out.
    2 * * * Request timed out.
    3 1 ms 1 ms 1 ms 10.X.10.242
    4 * * * Request timed out.
    5 1 ms 1 ms 1 ms A.B.200.185


    Then to the one that never, ever works. It takes a different route for reasons uknown:

    C: \windows\system32>tracert -d A.B.200.184

    Tracing route to A.B.200.184 over a maximum of 30 hops

    1 <1 ms <1 ms <1 ms 10.Y.7.3 <---note that this system's default gateway is 10.Y.7.1
    2 * * * Request timed out.
    3 1 ms <1 ms <1 ms 10.X.10.242
    4 1 ms <1 ms <1 ms 10.X.14.14
    5 * * * Request timed out.
    6 * * * Request timed out.
    7 * * * Request timed out.

    -SNIP-

    28 * * * Request timed out.
    29 * * * Request timed out.
    30 * * * Request timed out.

    There's probably not much to do outside of complaining to the right party who might know what's going on, but I'm just afraid that something was misconfigured on the system itself. I was urged to use one of their provisioning profiles that supposedly configures everything correctly, but I can request a blank VM and load my own OS if I make a separate request. I could also thumb my nose at the powers that be and host it in my own datacenter and hope I don't get in trouble later...
     
    Paladin

    Paladin

    Ars Legatus Legionis
    32,552
    Subscriptor
  • #8
    Looks like they gave you an IP that is either misrouted somehow (some old static route) or the network is not as contiguous as you hope. Given your lack of control and access to the network, you are at the mercy of the people running it. I would basically hammer tickets at them and phone calls, etc. and stick the basics. 'I can't ping this thing I should be able to ping, let alone actually print.'
    Show them the differing traceroutes, that is certainly interesting when the IPs are sequential like that.
     
    Nevarre

    Nevarre

    Ars Legatus Legionis
    24,110
  • #12
    To close the loop and probably to nobody's surprise, on the VM cluster my VM somehow was assigned the same IP as another host. The internal virtual switch was doing strange stuff instead of showing consistent random failures like you might expect from a duplicated IP on the network.

    Now with a new IP address all to itself, my Server 2019 VM is working as expected.

    (Note, it was given that IP as a static address by the VM group, and they're trying to figure out why their system did that--not a fat finger on my side of things. I'm just their QA tester apparently.)
     
    • Haha
    Reactions: Paladin
    Paladin

    Paladin

    Ars Legatus Legionis
    32,552
    Subscriptor
  • #13
    That makes some sense. I'm guessing this was not on a run of the mill VMware or Hyper-V system? I would assume something like openstack, or nutanix maybe or at least VMware with NSX or something. Maybe AWS or something with a weird orchestration platform involved. Those are the only ways I can figure getting an IP address assigned by the system that can easily overlap with another VM on the same system (which presumably also had an IP assigned by the same system) and have them fight it out without knowing the other exists.

    In a vanilla hypervisor system the VMs would simply see each other on the same layer 2 broadcast domain (assuming things are coherently setup, even with VLANs they should all converge somewhere) and one would say 'Sorry, my IP is in use, I'm going offline.' With overlay networks you can use the same IPs in multiple 'networks' that are actually software defined and isolated... until they need to talk to stuff outside the platform and things fall apart as the traffic has to cross through the virtual routers on the platform. They could have simply assigned the NAT incorrectly or a floating IP more than once somehow or something like that. The system should not make that easy to do though.
     
    Nevarre

    Nevarre

    Ars Legatus Legionis
    24,110
  • #14
    Paladin said:
    That makes some sense. I'm guessing this was not on a run of the mill VMware or Hyper-V system? I would assume something like openstack, or nutanix maybe or at least VMware with NSX or something. Maybe AWS or something with a weird orchestration platform involved. Those are the only ways I can figure getting an IP address assigned by the system that can easily overlap with another VM on the same system (which presumably also had an IP assigned by the same system) and have them fight it out without knowing the other exists.

    In a vanilla hypervisor system the VMs would simply see each other on the same layer 2 broadcast domain (assuming things are coherently setup, even with VLANs they should all converge somewhere) and one would say 'Sorry, my IP is in use, I'm going offline.' With overlay networks you can use the same IPs in multiple 'networks' that are actually software defined and isolated... until they need to talk to stuff outside the platform and things fall apart as the traffic has to cross through the virtual routers on the platform. They could have simply assigned the NAT incorrectly or a floating IP more than once somehow or something like that. The system should not make that easy to do though.
    Click to expand...

    They're using VMWare Aria, which I'm not all that familiar with (vs. a simple vCenter cluster). It's supposed to orchestrate everything and has robust container support in addition to full VMs, intermingled with one another as discrete units. There's a vCenter interface somewhere, so it may just be a front end-- but I don't have access to any of that. I'm just a user following the self-service provisioning.

    There's a rigorous set of rules on who can use the 10.net with each subnet 'owned' by someone. This was in the range all internally managed by the VM system, and should have been reserved for specific VM guests. There's no reason to overlap just because everyone is supposed to be on the same page about which 10.net addresses you're allowed to use.

    But yeah-- because this was all being handled internally on their virtual switch, it never quite presented the symptoms as cleanly as I would have liked for a simple IP conflict. I miss having visibility into some of this stuff...
     
    • Like
    Reactions: Paladin
    Paladin

    Paladin

    Ars Legatus Legionis
    32,552
    Subscriptor
  • #15
    Yeah Aria has a lot of stuff it can do on top of vsphere and other platforms but ultimately it is just a management layer that provides more control and insight into use and stuff. It should allow self service access too so you could manage your own instance if they would let you. I'm not that familiar, just with marketing stuff so who knows what it really does in real life? ;) You would think it would be able to prevent IP conflicts but there might be something about the deployment that made it possible.

    It is frustrating when you become a consumer of a platform rather than administrator. Well, there are tradeoffs for sure, anyway.
     
    You must log in or register to reply here.
    Jump to latest Follow Reply
    Jump to latest Follow Reply