<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body dir="auto">
<div dir="ltr">We use Perl to accomplish this kind of thing.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">We blackhole /32s, when we have “enough” of them in the same /24, we remove the /32s after inserting a covering /24. This is a 4 line script, along the same lines of the sed and python suggestions.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Our threshold is pretty low. If we see 4 simultaneous bad actors from the same /24 it’s gone. But we have a very fair process of putting them back into use, think fail2ban.</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Best,</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Deepak </div>
<div dir="ltr"><br>
</div>
<div dir="ltr"><br>
<blockquote type="cite">On Jun 14, 2021, at 3:51 AM, Chris Hartley <hartleyc@gmail.com> wrote:<br>
<br>
</blockquote>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div>I guess something like this... maybe? Surely someone has already done this much better, but I thought it might be a fun puzzle.<br>
</div>
<div><br>
</div>
<div></div>
<div># Let's call it aggregate.py. You should test/validate this and not trust it at all because I don't. It does look like it works, but I can't promise anything like that. This was "for fun." For me in my world, it's not a problem that needs solving,
but if it helps someone, that'd be pretty cool. No follow-up questions, please.<br>
</div>
<div><br>
</div>
<div>./aggregate.py gen 100000 ips.txt # Make up some random IPs for testing<br>
</div>
<div>./aggregate.py aggregate 2 ips.txt # Aggregate... second argument is the "gap", third is the filename...<br>
</div>
<div><br>
</div>
<div>Most are still going to be /32s.</div>
<div>Some might look like this - maybe even bigger:</div>
<div><a href="http://27.151.199.176/29">27.151.199.176/29</a><br>
<a href="http://33.58.49.184/29">33.58.49.184/29</a><br>
<a href="http://40.167.88.192/29">40.167.88.192/29</a><br>
</div>
<div><a href="http://63.81.88.112/28">63.81.88.112/28</a> # This is your example set of IPs with a gap (difference) of 2.<br>
</div>
<div><a href="http://200.42.160.124/30">200.42.160.124/30</a></div>
<div><br>
</div>
<div>"max gap" is the distance between IP addresses that can be clustered... an improvement might include "coverage" - a parameter indicating how many IPs must appear (ratio) in a cluster to create the aggregate (more meaningful with bigger gaps).</div>
<div><br>
</div>
<div>#!/your/path/to/python<br>
</div>
import random<br>
import sys<br>
<br>
def inet_aton(ip_string):<br>
octs = ip_string.split('.')<br>
n = int(int(octs[0]) << 24) + int(int(octs[1]) << 16) + int(int(octs[2]) << 8) + int(octs[3])<br>
return n<br>
<br>
def inet_ntoa(ip):<br>
octs = ( ip >> 24, (ip >> 16 & 255), (ip >> 8) & 255, ip & 255 )<br>
return str(octs[0]) + "." + str(octs[1]) + "." + str(octs[2]) + "." + str(octs[3])<br>
<br>
def gen_ips(num):<br>
ips = []<br>
for x in range(num):<br>
ips.append(inet_ntoa(random.randint(0,pow(2,32)-1)))<br>
# To make sure we have at least SOME nearlyconsecutive IPs...<br>
ips += "63.81.88.116,63.81.88.118,63.81.88.120,63.81.88.122,63.81.88.124,63.81.88.126".split(",") # I added your example IPs.<br>
return ips<br>
<br>
def write_random_ips(num,fname):<br>
ips = gen_ips(int(num))<br>
f = open(fname,'w')<br>
for ip in ips:<br>
f.write(ip+'\n')<br>
f.close()<br>
<br>
def read_ips(fname):<br>
return open(fname,'r').read(99999999).split('\n')<br>
<br>
class Cluster():<br>
def __init__(self):<br>
self.ips = []<br>
def add_ip(self,ip):<br>
self.ips.append(ip)<br>
<br>
def find_common_bits(ipa,ipb):<br>
for bits in range(0,32):<br>
mask = pow(2,32)-1 << bits & (pow(2,32)-1)<br>
<br>
if ipa & mask == ipb & mask:<br>
return 32-bits<br>
else:<br>
pass # print(f"{ipa} & (pow(2,{bits})-1) == {ipa & (pow(2,bits)-1)} ==!=== {ipb} & (pow(2,{bits})-1) == {ipb & (pow(2,bits)-1)}")<br>
<br>
if len(sys.argv) == 4 and sys.argv[1] == "generate":<br>
write_random_ips(sys.argv[2],sys.argv[3])<br>
elif len(sys.argv) == 4 and sys.argv[1] == "aggregate": # TODO: Let's imagine a "coverage" field that augments the max_gap field... does the prefix cover too many IPs?<br>
max_gap = int(sys.argv[2])<br>
fname = sys.argv[3]<br>
<br>
ips = [ inet_aton(ip) for ip in read_ips(fname) if ip!='' ] # ... it'd be a good idea to make sure it looks like an IP. Oh, this only does IPv4 btw.<br>
<br>
ips.sort()<br>
<br>
clusters=[Cluster()] # Add first (empty) cluster.. is this necessary? Who cares, moving on....<br>
last_ip=None<br>
for ip in ips:<br>
if last_ip != None:<br>
#print(f"Gap of {ip-last_ip} between {ip} and {last_ip}... {inet_ntoa(ip)} / {inet_ntoa(last_ip)}")<br>
if ip - last_ip <= max_gap:<br>
#print(f"Gap of {ip-last_ip} between {ip} and {last_ip}...")<br>
clusters[-1].add_ip(ip)<br>
else:<br>
cluster=Cluster()<br>
cluster.add_ip(ip)<br>
clusters.append(cluster)<br>
last_ip = ip<br>
<br>
for cluster in clusters:<br>
if len(cluster.ips) == 0:<br>
continue <br>
if len(cluster.ips) > 1:<br>
first_ip=cluster.ips[0]<br>
last_ip=cluster.ips[-1]<br>
num_bits = find_common_bits(first_ip,last_ip) <br>
mask = pow(2,32)-1 << (32-num_bits) & (pow(2,32)-1)<br>
network = first_ip & mask <br>
print(f"{inet_ntoa(network)}/{num_bits}")<br>
else:<br>
print(f"{inet_ntoa(cluster.ips[0])}/32") <br>
else:<br>
print("Usage:")<br>
print("{0} generate [number of IPs] [file name] # Generate specified number of IPs, save to [file name]")<br>
print("{0} aggregate [max gap] [file name] # Aggregate prefixes based on overlapping subnets/IPs per the max gap permitted...")</div>
</div>
</blockquote>
</body>
</html>