Speculation: The new servers aren't doing DB replication properly

Navin R Slavin

Fifth line center
Jan 1, 2011
16,175
63,350
Durrm NC
Here's what's going on. Mods, please pass it along.

My guess is they recently scaled up to 5 servers to handle the load, but broke replication between them. Dig gives these IPs:

$ dig hfboards.mandatory.com
; <<>> DiG 9.10.6 <<>> hfboards.mandatory.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17976
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hfboards.mandatory.com. IN A
;; ANSWER SECTION:
hfboards.mandatory.com. 300 IN A 104.17.138.150
hfboards.mandatory.com. 300 IN A 104.17.137.150
hfboards.mandatory.com. 300 IN A 104.17.140.150
hfboards.mandatory.com. 300 IN A 104.17.141.150
hfboards.mandatory.com. 300 IN A 104.17.139.150

(Actually, it also responds with many other third octets, all the way up to 150s, which may be Cloudflare's doing. So I don't know how many hosts are ultimately behind the DNS.)

Anyway. For each host, I used curl to spoof a header to get back direct data as follows, substituting the third octet as appopriate, below is the query for 142:

curl --header 'Host: hfboards.mandatory.com' 'http://104.17.142.150/threads/lindholm-hanifin-to-cal-for-hamilton-fox-and-ferland.2507533/page-6' > /tmp/142.html

137 and 142 are in sync, and 138/139/140/141 are in sync. But the two sets of hosts are out of sync with each other. My guess is that in their haste to stand up some new sites, they misconfigured database replication and haven't noticed yet because hey, at least the 500 errors stopped!

Because this is round robin DNS, people will hit different hosts randomly each time, so it looks like posts are "disappearing" when you're hitting a different version of the site.

This is going to be an absolute mess to clean up. But please tell me that someone at mandatory is at least aware of the issue.
 
Last edited:

SaskCanesFan

Registered User
Feb 27, 2015
2,405
5,970
I don't know anything about computers but everything about the new servers sucks big fat donkey duck. And I plan to say that in every thread I can. Because half my comments already have been lost and didn't post. Which is great help for a discussion forum
 

MinJaBen

Canes Sharks Boy
Sponsor
Dec 14, 2015
20,669
79,605
Durm
These rookie servers are just not developing well. Look at the Canes and Coffee serves that were taken after these servers, they are already doing great. We need to package these servers with the servers from the Cane’s section of HockeyBuzz and trade for CanesCountry servers.
 

Svechhammer

THIS is hockey?
Jun 8, 2017
23,419
85,899
But let's be real here... Trade Deadline HFB went offline from 8am to well past trading closed. The Draft, HFB went offline a half hour before the first pick through the second day.

Maybe it's time to get some legitimate quality IT employees to run this site. This community is too large to have this keep happening. Eventually people are just going to bail in droves.
 

poobags

8) 8) 8( 8)
Jan 27, 2013
590
1,146
Washington
But let's be real here... Trade Deadline HFB went offline from 8am to well past trading closed. The Draft, HFB went offline a half hour before the first pick through the second day.

Maybe it's time to get some legitimate quality IT employees to run this site. This community is too large to have this keep happening. Eventually people are just going to bail in droves.

I'm down to bail. Where to next?
 

Navin R Slavin

Fifth line center
Jan 1, 2011
16,175
63,350
Durrm NC
I don't know anything about computers but everything about the new servers sucks big fat donkey duck. And I plan to say that in every thread I can. Because half my comments already have been lost and didn't post. Which is great help for a discussion forum

They haven't been lost. They're just on random servers that you won't hit every time. Totally different!
 

The Faulker 27

Registered User
Nov 15, 2011
12,819
47,242
Sauna-Aho
Here's what's going on. Mods, please pass it along.

My guess is they recently scaled up to 5 servers to handle the load, but broke replication between them. Dig gives these IPs:

$ dig hfboards.mandatory.com
; <<>> DiG 9.10.6 <<>> hfboards.mandatory.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17976
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hfboards.mandatory.com. IN A
;; ANSWER SECTION:
hfboards.mandatory.com. 300 IN A 104.17.138.150
hfboards.mandatory.com. 300 IN A 104.17.137.150
hfboards.mandatory.com. 300 IN A 104.17.140.150
hfboards.mandatory.com. 300 IN A 104.17.141.150
hfboards.mandatory.com. 300 IN A 104.17.139.150

(Actually, it also responds with many other third octets, all the way up to 150s, which may be Cloudflare's doing. So I don't know how many hosts are ultimately behind the DNS.)

Anyway. For each host, I used curl to spoof a header to get back direct data as follows, substituting the third octet as appopriate, below is the query for 142:

curl --header 'Host: hfboards.mandatory.com' 'http://104.17.142.150/threads/lindholm-hanifin-to-cal-for-hamilton-fox-and-ferland.2507533/page-6' > /tmp/142.html

137 and 142 are in sync, and 138/139/140/141 are in sync. But the two sets of hosts are out of sync with each other. My guess is that in their haste to stand up some new sites, they misconfigured database replication and haven't noticed yet because hey, at least the 500 errors stopped!

Because this is round robin DNS, people will hit different hosts randomly each time, so it looks like posts are "disappearing" when you're hitting a different version of the site.

This is going to be an absolute mess to clean up. But please tell me that someone at mandatory is at least aware of the issue.

Is DB administration your area of expertise?
 

HaNotsri

Regstred User
Dec 29, 2013
8,112
5,969
Here's what's going on. Mods, please pass it along.

My guess is they recently scaled up to 5 servers to handle the load, but broke replication between them. Dig gives these IPs:

$ dig hfboards.mandatory.com
; <<>> DiG 9.10.6 <<>> hfboards.mandatory.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17976
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;hfboards.mandatory.com. IN A
;; ANSWER SECTION:
hfboards.mandatory.com. 300 IN A 104.17.138.150
hfboards.mandatory.com. 300 IN A 104.17.137.150
hfboards.mandatory.com. 300 IN A 104.17.140.150
hfboards.mandatory.com. 300 IN A 104.17.141.150
hfboards.mandatory.com. 300 IN A 104.17.139.150

(Actually, it also responds with many other third octets, all the way up to 150s, which may be Cloudflare's doing. So I don't know how many hosts are ultimately behind the DNS.)

Anyway. For each host, I used curl to spoof a header to get back direct data as follows, substituting the third octet as appopriate, below is the query for 142:

curl --header 'Host: hfboards.mandatory.com' 'http://104.17.142.150/threads/lindholm-hanifin-to-cal-for-hamilton-fox-and-ferland.2507533/page-6' > /tmp/142.html

137 and 142 are in sync, and 138/139/140/141 are in sync. But the two sets of hosts are out of sync with each other. My guess is that in their haste to stand up some new sites, they misconfigured database replication and haven't noticed yet because hey, at least the 500 errors stopped!

Because this is round robin DNS, people will hit different hosts randomly each time, so it looks like posts are "disappearing" when you're hitting a different version of the site.

This is going to be an absolute mess to clean up. But please tell me that someone at mandatory is at least aware of the issue.
So, our posts have become interdimensional? That’s cool.
 

Ad

Upcoming events

Ad

Ad

-->