<div dir="ltr">Hi all, great replies. :) Let me clarify my initial question, and then respond one by one:<div><br></div><div>My intention is to run a web-crawling service on a public cloud. This service is geographically distributed, and therefore will run in multiple regions around the world inside AWS... this means there will be multiple AWS VPCs, each with their own NAT gateway, and traffic destined to websites that we crawl will appear to come from this NAT gateway's IP address.</div><div><br></div><div>The reason I want a predictable IP is to communicate this IP to website owners so they can allow access from these IPs into their networks. I chose IP as an example; it can also be a subnet, but what I don't want to provide is a list of 100 different IP addresses without any predictability.</div><div><br></div><div>I understand that this is not perfect, and would frankly not be my preferred approach to solve the problem.... but we've had requests of this nature from websites to create an allowlist of a limited number of predictable IPs so it doesn't trip their IDSs/other systems they might have... so we're trying to see how well it would work in practice. For the moment, let's set aside the issue as to whether AWS will even let me advertise the same IP on all my VPC NAT gateways, and just look at whether it's technically feasible. My gut feeling is that this wouldn't work well in practice, but I wanted to ask the experts here...</div><div><br></div><div>Also, pointers on what the best practices for solving this issue are most welcome, so I can reference those who ask for IP addresses to this discussion and follow recommendations here.</div><div><br></div><div>Onto the responses:</div><div><br></div><div>@<a href="mailto:owen@delong.com">owen@delong.com</a> and @<a href="mailto:woody@pch.net">woody@pch.net</a> <a href="mailto:athompson@merlin.mb.ca">athompson@merlin.mb.ca</a><br></div><div>> Because there’s no good/reliable way to get the replies back to the correct initiating host. </div><div><br></div><div><div>> When my clients make connections outbound to anycast addresses, the destination is more-or-less stable, and the replies come back to the client's unique IP, so anycast works in that direction. The guarantees are not present in the reverse direction.</div><div><br></div></div><div>Yes, this makes sense as the destination can be anywhere around the world, and that routing is asymmetric as others mentioned. However, if the destination service is "close" (in the routing metric sense) to the initiating host, anycast return IP ought to work well, right? I understand this is a very important caveat and impractical to implement correctly in the real world.</div><div><br></div><div>> We use our IGP (IS-IS) for our Anycast services. We find it to be very</div>basic, and as such, very predictable.<div><br><div>This is interesting... I wonder whether Anycast will still have some failure modes and break TCP connections if routing (configuration) were to change? I checked the PDF linked by Bill Woodcock... while the methodology is the same from 20y ago, would the data still be the same (order of magnitude)? :)<br></div><div><br></div><div><a href="https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf">https://www.pch.net/resources/Tutorials/anycast/Anycast-v10.pdf</a> (p38)<br></div><div>"Limited operational data shows underlying instability to be on<br>the order of one flow per ten thousand per hour of duration."<br></div><div><br></div><div>@<a href="mailto:daniel@corbe.net">daniel@corbe.net</a>, @<a href="mailto:matt@netfire.net">matt@netfire.net</a>, <br></div><div>> Unless you’re twisting knobs, egress traffic should already exit your network at the closest possible egress point to its origin. Is your intention to carry the traffic for longer than that?<br></div><div>No, but I hope my intention is more clear in this email. It's to have a predictable egress IP to simplify firewall rules.</div><div><br></div><div>thanks all!!</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 27, 2021 at 12:25 PM Adam Thompson <<a href="mailto:athompson@merlin.mb.ca">athompson@merlin.mb.ca</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
Without any sarcasm: to make it harder to block.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
If, say, Google, always crawled your site from 8.8.1.2 (random made-up example) then you would see a not-insignificant number of hosts and networks null-routing that IP. I have no idea why someone would do so, but I've seen it done many times. Mostly by people
who don't understand how un-special they are on the internet. Also it would trigger IDS/IPS systems all over the place, having gobs and gobs of connections coming from a single IP.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
That's setting aside the technical issues involved; routing is often asymmetric, i.e. the return packet takes a different path than the inbound packet. So it would, as Owen implied, be nearly impossible to ensure the reply packets got back to the correct TCP
stack. As an example, I'm multi-homed and use path-prepending, so if a packet claiming to be from 8.8.8.8 arrived on one of my commercial links, I would send the reply out the cheapest link, which in my case is a flat-rate R&E network (that has a path to
Google), thus ensuring the reply does <i>not</i> get to the originating anycast node.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
When my clients make connections outbound to anycast addresses, the destination is more-or-less stable, and the replies come back to the client's unique IP, so anycast works in that direction. The guarantees are not present in the reverse direction.<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
The logical extremity of this is that it would be nearly impossible for two anycast addresses to establish a TCP connection to each other. (In general. There will be lots of local cases where it does happen to work, by coincidence.)</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
You'll find that even anycast nodes do not make connections outbound using their anycast address, pretty much for these reasons.</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
-Adam<br>
</div>
<div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<br>
</div>
<div id="gmail-m_-8510111738680190649Signature">
<div>
<div style="font-family:Calibri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)">
<b style="text-align:start;background-color:rgb(255,255,255)"><span style="margin:0px;font-size:10pt;color:rgb(68,84,106)">Adam Thompson</span></b><span style="margin:0px;font-size:9pt;color:rgb(68,84,106);text-align:start;background-color:rgb(255,255,255)"><br>
Consultant, Infrastructure Services<br>
<img alt="1593169877849" style="margin: 0px; max-width: 100%; height: auto;" src="cid:17ae9884f0f607c78dd1"><br>
100 - 135 Innovation Drive<br>
Winnipeg, MB, R3T 6A8<br>
(204) 977-6824 or 1-800-430-6404 (MB only)<br>
<a href="mailto:athompson@merlin.mb.ca" rel="noopener noreferrer" style="margin:0px" target="_blank"><span style="margin:0px">athompson@merlin.mb.ca</span></a><br>
<a href="http://www.merlin.mb.ca/" rel="noopener noreferrer" style="margin:0px" target="_blank"><span style="margin:0px">www.merlin.mb.ca</span></a></span><br>
</div>
</div>
</div>
</div>
<div id="gmail-m_-8510111738680190649appendonsend"></div>
<hr style="display:inline-block;width:98%">
<div id="gmail-m_-8510111738680190649divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> NANOG <nanog-bounces+athompson=<a href="mailto:merlin.mb.ca@nanog.org" target="_blank">merlin.mb.ca@nanog.org</a>> on behalf of Vimal <<a href="mailto:j.vimal@gmail.com" target="_blank">j.vimal@gmail.com</a>><br>
<b>Sent:</b> July 27, 2021 12:54<br>
<b>To:</b> <a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a> <<a href="mailto:nanog@nanog.org" target="_blank">nanog@nanog.org</a>><br>
<b>Subject:</b> Anycast but for egress</font>
<div> </div>
</div>
<div>
<div dir="ltr">(Unsure if this is the right forum to ask this question, but here goes:)
<div><br>
</div>
<div>From what I understand, IP Anycast can be used to steer traffic into a server that's close to the client.</div>
<div><br>
</div>
<div>I am curious if anyone here has/encountered a setup where they use anycast IP on their gateways... to have a predictable egress IP for their traffic, regardless of where they are located?</div>
<div><br>
</div>
<div>For example, a search engine crawler could in principle have the same IP advertised all over the world, but it looks like they don't... I wonder why?<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr">Vimal</div>
</div>
<div dir="ltr"><br>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature">Vimal</div>