This is another problem from CTFLearn. In this problem, we were given 3 hints, that suggest us to find out about robots.txt. So I decided to just look at ctflearn.com's robots.txt. And found this:

User-agent: *
Disallow: /70r3hnanldfspufdsoifnlds.html

So we can see that URL is disallowed, so maybe we should just check that out?
And when we visit https://ctflearn.com/70r3hnanldfspufdsoifnlds.html we can see that it has the flag.

abctf{r0b0ts_4r3_th3_futur3}

Link to the problem