Navin R Slavin
Fifth line center
Here's what's going on. Mods, please pass it along.
My guess is they recently scaled up to 5 servers to handle the load, but broke replication between them. Dig gives these IPs:
$ dig hfboards.mandatory.com
; <<>> DiG 9.10.6 <<>> hfboards.mandatory.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17976
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hfboards.mandatory.com. IN A
;; ANSWER SECTION:
hfboards.mandatory.com. 300 IN A 104.17.138.150
hfboards.mandatory.com. 300 IN A 104.17.137.150
hfboards.mandatory.com. 300 IN A 104.17.140.150
hfboards.mandatory.com. 300 IN A 104.17.141.150
hfboards.mandatory.com. 300 IN A 104.17.139.150
(Actually, it also responds with many other third octets, all the way up to 150s, which may be Cloudflare's doing. So I don't know how many hosts are ultimately behind the DNS.)
Anyway. For each host, I used curl to spoof a header to get back direct data as follows, substituting the third octet as appopriate, below is the query for 142:
curl --header 'Host: hfboards.mandatory.com' 'http://104.17.142.150/threads/lindholm-hanifin-to-cal-for-hamilton-fox-and-ferland.2507533/page-6' > /tmp/142.html
137 and 142 are in sync, and 138/139/140/141 are in sync. But the two sets of hosts are out of sync with each other. My guess is that in their haste to stand up some new sites, they misconfigured database replication and haven't noticed yet because hey, at least the 500 errors stopped!
Because this is round robin DNS, people will hit different hosts randomly each time, so it looks like posts are "disappearing" when you're hitting a different version of the site.
This is going to be an absolute mess to clean up. But please tell me that someone at mandatory is at least aware of the issue.
My guess is they recently scaled up to 5 servers to handle the load, but broke replication between them. Dig gives these IPs:
$ dig hfboards.mandatory.com
; <<>> DiG 9.10.6 <<>> hfboards.mandatory.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17976
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hfboards.mandatory.com. IN A
;; ANSWER SECTION:
hfboards.mandatory.com. 300 IN A 104.17.138.150
hfboards.mandatory.com. 300 IN A 104.17.137.150
hfboards.mandatory.com. 300 IN A 104.17.140.150
hfboards.mandatory.com. 300 IN A 104.17.141.150
hfboards.mandatory.com. 300 IN A 104.17.139.150
(Actually, it also responds with many other third octets, all the way up to 150s, which may be Cloudflare's doing. So I don't know how many hosts are ultimately behind the DNS.)
Anyway. For each host, I used curl to spoof a header to get back direct data as follows, substituting the third octet as appopriate, below is the query for 142:
curl --header 'Host: hfboards.mandatory.com' 'http://104.17.142.150/threads/lindholm-hanifin-to-cal-for-hamilton-fox-and-ferland.2507533/page-6' > /tmp/142.html
137 and 142 are in sync, and 138/139/140/141 are in sync. But the two sets of hosts are out of sync with each other. My guess is that in their haste to stand up some new sites, they misconfigured database replication and haven't noticed yet because hey, at least the 500 errors stopped!
Because this is round robin DNS, people will hit different hosts randomly each time, so it looks like posts are "disappearing" when you're hitting a different version of the site.
This is going to be an absolute mess to clean up. But please tell me that someone at mandatory is at least aware of the issue.
Last edited: